Any suggestions for handling dynamic pipelines? I have a scenario that I would like to run a model training and model evaluation based on labels on a dataset. So each label would trigger one pipeline.

So dynamic pipelines are a bit of a minefield because they become complex very quickly!

In this situation I'd go for two approaches: - have your labeling process write to a dataset kedro can read - have the output of your labeling process passed in to kedro run --params

So for example we have the labels, A and B, so the labeling pipeline would output a single dataframe with values for A and B, then the training pipeline is coded for handling one label, then I can pass kedro run --param label:A, did I get it right?

Something like that

Hi Kedro community!! I just started using kedro and I got stuck. I'm pretty sure that lots of people had the same issue, but I've tried searching everywhere and I can't find the solution. Could anyone point me to the right answer? So, my pipeline is pretty standard, I assume: * load images from a directory * preprocess * CNN inference * postprocess * save output I've created a partitioned image dataset like in the pokemon example here https://kedro.readthedocs.io/en/latest/extend_kedro/custom_datasets.html. I'd like to apply the inference pipeline sequentially to each image (or each minibatch), but I can't figure out how to pass image-by-image to the pipeline. Does anyone know how to do this?

in catalog.yml we are using runtime arguments like bucket_name below: raw_boat_data: filepath: "s3a://${bucket_name}/${key_prefix}/${folders.raw}/boats.csv" file_format: parquet bucket_name is also defined in globals.yml and is overridden by runtime parameter through commandline I see that sometimes the bucket_name is not correctly picked from the command line, it is still picking the value from globals.yml. what am i missing here?

The command line part is custom, default kedro doesn't do that iirc

See this if you are on 0.18.x https://github.com/kedro-org/kedro/discussions/1782

Understanding data catalog datasets

Using kedro in databricks workflow job

Hey, I wonder whether the Pipeline README markdowns are meant to end up in the automatic sphinx docs. It doesn't seem so? I searched the html files for keywords from my Pipelines' markdown files and didn't have any matches. Or is there an easy way to switch behavior? Thank you!

So we just provide a very thin layer on top of sphinx here and aren't opinionated on how you construct your docs. The readme files here can easily be added to the toc tree in index.rst but aren't automatically added

I see, thank you for helping out a novice 😊

We all were once!

Is there any way to have a pipeline run a bash script as a node?

Via the subprocess module but it's messy

Anyone ever try the

sh

module for this?

Hi guys, Is there any way to continue the pipeline even if a node failed?

Use

on_node_error

hook potentially, but be cautious about the error-handling.

Or a custom runner

thanks. Looking for them !

Other issue, is there a way to reduce the error message from cli?

You can set the level of the log to display via the

logging.yml

, but in general I don't think you want to hide error messages

thanks

how?

You simply update the LEVEL of logging that you desire

There should be a

logging.yml

, it's standard Python Logging module if you need more docs about it.

Even if I change it , the huge error message doesn’t disappear.