Hi all, if I want to pass a parameter to my node I can do so by writing:

node(
    func=foo,
    inputs=['input', 'params:parameter'],
    outputs='output'
),

Is there something similar I can do to pass a credential from the

credentials.yml

to my node?

We don't support this by design, you could do a

parameters.yml

in your

local

folder, but typically your nodes should focus on data flow not IO

thanks

Hey kedroids, looking for a code style opinion here, is there any way to pass inputs directly into pd.concat without an extra function?

python
node(lambda *frames: pd.concat(frames), ["cars", "cars"], "two_cars")

How would you concatenate pandas dataframes?

Cunning pd.concat

Hello, how can I catch a DataSetError? I tried "except kedro.io.data_catalog.DataSetError" but then I get NameError: name 'kedro' is not defined...

I think you just need to do

from kedro.io.data_catalog import DataSetError

before you do this

+ do

except DataSetError

rather than the full classpath

it worked, thank you!

I have a quick question on experiment logging: for example if I want to generate 10 models, I understand I can log all 10 hyperparameters in the JSON. how can I save models as 10 pickle files and appended into the config.yaml file?

Hey so there are a few things here: - To reuse the same pipeline multiple times I would recommend reading our modular pipeline docs https://kedro.readthedocs.io/en/stable/nodes_and_pipelines/modular_pipelines.html - To log parameters you should use the experiment tracking features https://kedro.readthedocs.io/en/stable/tutorial/set_up_experiment_tracking.html - To save pickles you can find example on the catalog docs just search form

pickle.PickleDataSet

https://kedro.readthedocs.io/en/stable/data/data_catalog.html

This reminds me of a question that I used to have. When doing a hyperparameter search, I need to train K (dynamic number, could be 10 or 20) models, then later in the pipeline should pick the best model to continue. What's the best way to handle the K entry in catalog?

I think a hook, but it's not ideal.

A custom data set might work. You can save K models with the dataset and load 1 later.

I think there is a wider piece about maybe we should have an option to track all parameters automatically. There is also evidence that this should be tied to the Hydra multi-run feature requests + internal multi-runner plugin. @Nero_Okwa something to consider in your research.

This reminds me of a question that I

In this example of connecting the the pipelines (https://kedro.readthedocs.io/en/stable/nodes_and_pipelines/modular_pipelines.html#combining-disconnected-pipelines), is

inputs={"food": "grilled_veg"}

supposed to be

outputs={"grilled_veg": "food"}

instead? Or perhaps it's the input to the

lunch_pipeline

?

prep_pipeline

is the upstream pipeline.

At a quick glance, look like it's a documentation issue. Would you like to start a PR and fix it if you can make it work?

Cool. Thanks for the confirmation. I just did a PR (https://github.com/kedro-org/kedro/pull/1578). I hope I did it correctly.

Is it possible to control the hierarchy in which nodes are visualized via kedro viz? Let me demonstrate with my example. In it, I've had to input a special 'identity node' in order for the viz not be completely messed up...

There's my pipeline. Let's for the sake of argument define the different rows/levels of this viz as the hierarchy. At the 3rd level, we have the datasets "customer df", "competitor df" and "previous matches df". If you follow the paths, you can see that the only inputs I really need for the nodes at the 4th level (besides the node identity) are "customer df" and "competitor df". The dataset "previous matches df" is really only needed at the second to last level for the node "analysis". For this pipeline, I've included the node "identity" and the sole reason for it is to make the viz look nice. If I were to remove it, the whole viz would look very messy and most importantly, the dataset "previous matches df" (which right now sits at the 3rd level, which is precisely what I want) would instead be moved to the 5th level. So the question is: Is there a way to control at what level datasets are shown in the viz? In my pipeline, "previous matches df" is loaded by the node "load data" and then only used again by the node "analysis". Is there a way for me to choose where to visualize it in the levels between those nodes?

I realize my explanation might've been a little diffuse, so here's a clarification at the end result I'd like to achieve! 🙂

The problem is that I can't. If I were to remove the "identity" node from the previous pipeline to try to achieve the desired result, the dataset "previous matches df" would instead be move to the the 5th level/row of the viz. Is there a way to tell kedro to keep it at the 3rd?

Sorry, I'm quite new to all of this but has anyone compared Kedro against something like "You don't need a bigger boat" (https://github.com/jacopotagliabue/you-dont-need-a-bigger-boat) or ZenML? I'm new to this whole world and, to make a long story short, I'm REALLY lost I'm in the process of evaluating tools / frameworks. I'm currently a single person looking to set up the groundwork for things to come. So far my plans have been to stay local for as long as possible before moving to some distributed computing framework (Dask gave me a lot of trouble in the past). I'm also looking to avoid using tools such as AWS or GCP for as long as possible so ideally the discussion would revolve around local machines. I'd love to hear thoughts and opinions.

https://kedro.readthedocs.io/en/stable/tutorial/visualise_pipeline.html Have you check out this documentation? I think the

layer

can helps if you need to align certain nodes in a specific layer.

Thank you! I will check it out

Ok, a follow-up question on the previous. I managed to set up the layers as you and the documentation suggested. However, won't this create problems if I were to run multiple pipelines in parallell? Essentially, my pipeline takes as it's input two unique parameters (one customer domain, and on competitor domain) and then runs based on that info. I intend to use this pipeline for many combinations of customer domain and competitor domain, but if I do this is parallell then the created datasets will all be saved under the same name in the folder 01_raw. Surely this will be problematic if I run the same pipeline in parallell? This is what I added to my catalog: customer_df: type: pickle.PickleDataSet filepath: data/01_raw/customer_df.pickle layer: raw competitor_df: type: pickle.PickleDataSet filepath: data/01_raw/competitor_df.pickle layer: raw previous_matches_df: type: pickle.PickleDataSet filepath: data/01_raw/previous_matches_df.pickle layer: raw

Not sure if I am following, so the viz problem is solved now and you have a parallel pipeline problem?

I'm going to try a little myself, and come back if I need more help. Thank you!

Hi! Is there a way to log the data version that is being used? I saw this thread for logging the version https://discord.com/channels/778216384475693066/846330075535769601/976559672691662908 but that is only for logging the version requested with --load-version. How can for each node write to log which (latest) version was loaded and written? E.g. something like

2022-05-30 10:01:09,039 - kedro.io.data_catalog - INFO - Loading data from `batch` (PartitionedDataSet) (version: 2022-05-15T05.24.31.017Z)
2022-05-30 10:01:18,467 - kedro.io.data_catalog - INFO - Saving data to `processed_batch` (ParquetDataSet)... (version: 2022-05-30T08.23.54.197Z)

Would be great 🙂