A custom data set might work. You can save K models with the dataset and load 1 later.

I think there is a wider piece about maybe we should have an option to track all parameters automatically. There is also evidence that this should be tied to the Hydra multi-run feature requests + internal multi-runner plugin. @Nero_Okwa something to consider in your research.

This reminds me of a question that I

In this example of connecting the the pipelines (https://kedro.readthedocs.io/en/stable/nodes_and_pipelines/modular_pipelines.html#combining-disconnected-pipelines), is

inputs={"food": "grilled_veg"}

supposed to be

outputs={"grilled_veg": "food"}

instead? Or perhaps it's the input to the

lunch_pipeline

?

prep_pipeline

is the upstream pipeline.

At a quick glance, look like it's a documentation issue. Would you like to start a PR and fix it if you can make it work?

Cool. Thanks for the confirmation. I just did a PR (https://github.com/kedro-org/kedro/pull/1578). I hope I did it correctly.

Is it possible to control the hierarchy in which nodes are visualized via kedro viz? Let me demonstrate with my example. In it, I've had to input a special 'identity node' in order for the viz not be completely messed up...

message has been deleted

There's my pipeline. Let's for the sake of argument define the different rows/levels of this viz as the hierarchy. At the 3rd level, we have the datasets "customer df", "competitor df" and "previous matches df". If you follow the paths, you can see that the only inputs I really need for the nodes at the 4th level (besides the node identity) are "customer df" and "competitor df". The dataset "previous matches df" is really only needed at the second to last level for the node "analysis". For this pipeline, I've included the node "identity" and the sole reason for it is to make the viz look nice. If I were to remove it, the whole viz would look very messy and most importantly, the dataset "previous matches df" (which right now sits at the 3rd level, which is precisely what I want) would instead be moved to the 5th level. So the question is: Is there a way to control at what level datasets are shown in the viz? In my pipeline, "previous matches df" is loaded by the node "load data" and then only used again by the node "analysis". Is there a way for me to choose where to visualize it in the levels between those nodes?

I realize my explanation might've been a little diffuse, so here's a clarification at the end result I'd like to achieve! 🙂

The problem is that I can't. If I were to remove the "identity" node from the previous pipeline to try to achieve the desired result, the dataset "previous matches df" would instead be move to the the 5th level/row of the viz. Is there a way to tell kedro to keep it at the 3rd?

Sorry, I'm quite new to all of this but has anyone compared Kedro against something like "You don't need a bigger boat" (https://github.com/jacopotagliabue/you-dont-need-a-bigger-boat) or ZenML? I'm new to this whole world and, to make a long story short, I'm REALLY lost I'm in the process of evaluating tools / frameworks. I'm currently a single person looking to set up the groundwork for things to come. So far my plans have been to stay local for as long as possible before moving to some distributed computing framework (Dask gave me a lot of trouble in the past). I'm also looking to avoid using tools such as AWS or GCP for as long as possible so ideally the discussion would revolve around local machines. I'd love to hear thoughts and opinions.

https://kedro.readthedocs.io/en/stable/tutorial/visualise_pipeline.html Have you check out this documentation? I think the

layer

can helps if you need to align certain nodes in a specific layer.

Thank you! I will check it out

Ok, a follow-up question on the previous. I managed to set up the layers as you and the documentation suggested. However, won't this create problems if I were to run multiple pipelines in parallell? Essentially, my pipeline takes as it's input two unique parameters (one customer domain, and on competitor domain) and then runs based on that info. I intend to use this pipeline for many combinations of customer domain and competitor domain, but if I do this is parallell then the created datasets will all be saved under the same name in the folder 01_raw. Surely this will be problematic if I run the same pipeline in parallell? This is what I added to my catalog: customer_df: type: pickle.PickleDataSet filepath: data/01_raw/customer_df.pickle layer: raw competitor_df: type: pickle.PickleDataSet filepath: data/01_raw/competitor_df.pickle layer: raw previous_matches_df: type: pickle.PickleDataSet filepath: data/01_raw/previous_matches_df.pickle layer: raw

Not sure if I am following, so the viz problem is solved now and you have a parallel pipeline problem?

I'm going to try a little myself, and come back if I need more help. Thank you!

Hi! Is there a way to log the data version that is being used? I saw this thread for logging the version https://discord.com/channels/778216384475693066/846330075535769601/976559672691662908 but that is only for logging the version requested with --load-version. How can for each node write to log which (latest) version was loaded and written? E.g. something like

2022-05-30 10:01:09,039 - kedro.io.data_catalog - INFO - Loading data from `batch` (PartitionedDataSet) (version: 2022-05-15T05.24.31.017Z)
2022-05-30 10:01:18,467 - kedro.io.data_catalog - INFO - Saving data to `processed_batch` (ParquetDataSet)... (version: 2022-05-30T08.23.54.197Z)

Would be great 🙂

With Kedro 0.18.0+ the command kedro jupyter notebook won't load the catalog anymore. kedro ipython does work. Am a bit puzzled. Any ideas?

So this is often due to a broken catalog, if you try and run kedro run does it work? Or do you get an error?

kedro runs fine, and also in kedro ipython the catalog is loaded just fine.

That's weird! @antony.milne any ideas here?

This is the perfect time to use a kedro hook!

When I do a manual '''%load_ext kedro.extras.extensions.ipython''' in the notebook everything works as normal again. So it seem the extension is not loaded.

That's good you've got a fix, keen for Antony to chime in as he'll be able to work it out

Thx

When you start your notebook, did you choose the same kernel that your notebook is running in?

Hmm.. Any more hints possible here? My guess would be to use the

before_dataset_loaded

hook, but how do I get access the version that will be loaded?

https://kedro.readthedocs.io/en/stable/kedro.framework.hooks.specs.DataCatalogSpecs.html#kedro.framework.hooks.specs.DataCatalogSpecs Does this hook have enough information for you?