@User would you mind switching to the thread?

Hey, I just discovered Kedro last night and give it a try this morning with the spacefights tutorial . I'm a bit confused about using kedro in conda env. It seems that

kedro install

or

pip install -r src/requirements.txt

install correctly things in the env, but

kedro ipython

doesn't use it... I was getting errors with loading things in

catalog

and

%reload_kedro

. The errors are gone if I use the main ubuntu env without conda.

I'm wondering if I miss something with conda env, or I got something broken on my workstation ?

Hello ! currently we are building a pipeline with kedro, so we created a custom dataset based on AbstractDataset, and we are wondering how to access to _descrive() method while we are in a node. actually we have access to _load method of Dataset object only. thx

Hi @User the node actually can't access that sort of thing, by design the node shouldn't care how the data is loaded/saved. If you want to expose this information somewhat, perhaps the

before_node_run

hook may be useful

Hello, guys. In my DS pipeline I experiment A LOT and as I understand the focus of Kedro is on good software (not on a lot of experiments), the point is that the nodes itself are really great because I write only once. The question I have is with pipelines and init files where I have to write a lot of new pipelines and declarations to create these different experiments (I am keeping track in mlflow so this part is really ok), the namespace really helps but it's not flexible enough, I have no problems with this current workflow. My question is I am using Kedro in the correct way or am I missing something? Thanks a lot!

Hi @User and welcome! This sounds like you're doing roughly the right thing, but you might be missing a few things: * the best way to clone many nodes/pipelines that are similar but with slightly different parameters (e.g. different model) is through modular pipelines. @User will have some nice examples here to illustrate * note you can nest namespaces, e.g.

linear.experiment_1.train_model

. kedro-viz supports also supports arbitrarily nested hierarchy like this * experiment tracking is now natively supported in kedro. Still work in progress but if you're interested you can see the documentation at https://kedro.readthedocs.io/en/stable/08_logging/02_experiment_tracking.html and a demo at https://demo.kedro.org/ (click the flask icon on the left sidebar)

Example of multiple params: https://github.com/datajoely/modular-spaceflights/blob/main/conf/base/parameters/modelling.yml

I have a question about hooks plugin example, https://kedro.readthedocs.io/en/stable/07_extend_kedro/04_plugins.html#hooks i created hooks example and install it to my venv. if I want to add MyHooks to my project, what i need to do?

You need to add it to

settings.py

. What do you mean by installing the hooks?

I have an example of adding the custom

TimingHooks

here: https://github.com/datajoely/modular-spaceflights/blob/main/src/modular_spaceflights/settings.py

is TimingHooks in plugin?

Apologies! I misread what you wanted, it is not

it's a lifecycle hook

Our plugin documentation could use some work...

what I want to do is defining custom hook in plugin, and if i install plugin, i want to apply it to use the custom hook in my kedro project

Gotcha - @User who will be online tomorrow can help here, @User if he sees this knows that stuff well too

The other thing you can do is read through how

kedro-telemetry

works since it's a really simple plugin https://github.com/quantumblacklabs/kedro-telemetry

thank you, i will read it.

it looks like all I need to do it is installing plugin, after installing the example plugin i could see

2022-01-11 10:27:39,174 - root - INFO - Reached after_catalog_created hook

when i run the pipeline. this is awesome, thanks

one more question is If I already have

after_catalog_created

in a different hook in the current kedro project, which one will have priority between after_catalog_created() in the current kedro project and after_catalog_created() from the plugin?

I don’t know if you can guarantee the order - this is deffo an advanced question!

I’ll wait for the others to be online tomorrow to get back to you on this one!

ok, thanks!

The one from the plugin will be called first; see https://kedro.readthedocs.io/en/latest/07_extend_kedro/02_hooks.html#registering-your-hook-implementations-with-kedro for more info on hook order. 🙂 (I didn't know this until I looked it up just now either lol)

Hi guys hope you are doing good. i'm wondering how to release unused MemoryDataset which i will not use in next pipeline nodes thank you

So I think this is a limitation of the current runner - you could do an explicit

after_node_run

hook since that has the node name, catalog and inputs avaialble https://kedro.readthedocs.io/en/stable/kedro.framework.hooks.specs.NodeSpecs.html#kedro.framework.hooks.specs.NodeSpecs.after_node_run

ok i'll try that thanks !

Is there a way to automatically add a dataset to catalog based on params ? Because the API endpoint is exactly the same but the name

You would have to use a lifecycle hook, but it's very possible