🔥

Hey people, would like to know if someone already tried to use Spyder inside a Kedro repo, would be nice to have global variables just like a jupyter lab server.

Spyder should work just fine, but most of our users use Vs code and pycharm which can give you the same sort of experience in the debugger mode

Is there a way from a plain python file to use the catalog.load("..") from kedro? We would like to have a python file for experiments on Scipy but we would like to use Kedro's catalog to reuse the data from the project 🤔

So by design we don't want your nodes to be aware of io, it's much harder to debug if you stop your nodes being pure python functions

You can access the catalog live using kedro hooks

Which may allow you to do what you need

Thanks, I will take a look!

Any suggestions for handling dynamic pipelines? I have a scenario that I would like to run a model training and model evaluation based on labels on a dataset. So each label would trigger one pipeline.

So dynamic pipelines are a bit of a minefield because they become complex very quickly!

In this situation I'd go for two approaches: - have your labeling process write to a dataset kedro can read - have the output of your labeling process passed in to kedro run --params

So for example we have the labels, A and B, so the labeling pipeline would output a single dataframe with values for A and B, then the training pipeline is coded for handling one label, then I can pass kedro run --param label:A, did I get it right?

Something like that

Hi Kedro community!! I just started using kedro and I got stuck. I'm pretty sure that lots of people had the same issue, but I've tried searching everywhere and I can't find the solution. Could anyone point me to the right answer? So, my pipeline is pretty standard, I assume: * load images from a directory * preprocess * CNN inference * postprocess * save output I've created a partitioned image dataset like in the pokemon example here https://kedro.readthedocs.io/en/latest/extend_kedro/custom_datasets.html. I'd like to apply the inference pipeline sequentially to each image (or each minibatch), but I can't figure out how to pass image-by-image to the pipeline. Does anyone know how to do this?

in catalog.yml we are using runtime arguments like bucket_name below: raw_boat_data: filepath: "s3a://${bucket_name}/${key_prefix}/${folders.raw}/boats.csv" file_format: parquet bucket_name is also defined in globals.yml and is overridden by runtime parameter through commandline I see that sometimes the bucket_name is not correctly picked from the command line, it is still picking the value from globals.yml. what am i missing here?

The command line part is custom, default kedro doesn't do that iirc

See this if you are on 0.18.x https://github.com/kedro-org/kedro/discussions/1782

Understanding data catalog datasets

Using kedro in databricks workflow job

Hey, I wonder whether the Pipeline README markdowns are meant to end up in the automatic sphinx docs. It doesn't seem so? I searched the html files for keywords from my Pipelines' markdown files and didn't have any matches. Or is there an easy way to switch behavior? Thank you!

So we just provide a very thin layer on top of sphinx here and aren't opinionated on how you construct your docs. The readme files here can easily be added to the toc tree in index.rst but aren't automatically added

I see, thank you for helping out a novice 😊

We all were once!

Is there any way to have a pipeline run a bash script as a node?

Via the subprocess module but it's messy

Anyone ever try the

sh

module for this?

Hi guys, Is there any way to continue the pipeline even if a node failed?

Use

on_node_error

hook potentially, but be cautious about the error-handling.

Or a custom runner

thanks. Looking for them !