I’m looking for a custom sequential runner. Any tips ?

Hello. Would someone be able to please tell me the best way to debug my pipelines? I use PyCharm. I wish to use the debugger. Which script should I run? Do I need to change anything in the run configuration? Thanks!

There is a tutorial for setting up pycharm run config in the docs! Then you just add breakpoints and use the debugger

Also if you're lazy you can use pdb and the

breakpoint()

statement

https://kedro.readthedocs.io/en/stable/development/set_up_pycharm.html Got you covered!

Hi All. How do I reference credentials from Azure backed key vault so that the data catalog can access the azure storage buckets?

You'll need to use a before catalog created hook to inject the credentials into our catalog

I think

That or tweak how your templated config loader works in settings.py

I added the credentials to my spark configuration and it worked. So I am not passing credentials to data catalog, just making them available at the cluster level.

That's the best way

Trying to adapt the spaceflight tutorial to run on Kubeflow but I get this error:

kedro.io.data_catalog - INFO - Loading data from 'data_science.candidate_modelling_pipeline.metrics' (MemoryDataSet)...

Since this is Kubeflow and MemoryDataset cannot be used, this is not surprising. I observe that

data_science.candidate_modelling_pipeline.metrics

is not mentioned in catalog.yml. Is the fix to simply modify catalog.yml to add in this dataset similar to the way

data_science.active_modelling_pipeline.metrics

is already in there? As in, was this just an oversight? In which case, I would have separate output files for active and cadidate metrics...

If I run

kedro catalog create --pipeline __default__

on the space tutorial, it generates a bunch of datasets not in the catalog:

data_science.active_modelling_pipeline.X_test:
  type: MemoryDataSet
data_science.active_modelling_pipeline.X_train:
  type: MemoryDataSet
data_science.active_modelling_pipeline.y_test:
  type: MemoryDataSet
data_science.active_modelling_pipeline.y_train:
  type: MemoryDataSet
data_science.candidate_modelling_pipeline.X_test:
  type: MemoryDataSet
data_science.candidate_modelling_pipeline.X_train:
  type: MemoryDataSet
data_science.candidate_modelling_pipeline.y_test:
  type: MemoryDataSet
data_science.candidate_modelling_pipeline.y_train:
  type: MemoryDataSet

Why aren't these included in

conf/base/catalog.yml

when their absence causes errors like

ValueError: Pipeline input(s) {'data_science.active_modelling_pipeline.y_train', 'data_science.active_modelling_pipeline.X_train'} not found in the DataCatalog

???

If I run ` kedro catalog create pipeline

Is it possible to access the traceback of a failed kedro pipeline? Like we want to call a rest api, where body of the api will have the error message of the failed kedro pipeline.

Sure, it's just Python program. You can do a try except block and return the traceback message if u want

@rohan_ahire https://github.com/kedro-org/kedro/issues/1846 We are quite interested to understand how people are exposing kedro pipeline via web api and see if there anything we can improve

How do I disable rich in kedro 0.18.3? For my use case, rich backtrace is too verbose and it doesn't integrate well with vscode.

Sometimes rich breaks the line and I can't just ctrl-click the file to get to the code in vscode, and this can become annoying over time

You can switch back to "plain" logging following these instructions: https://kedro.readthedocs.io/en/stable/logging/logging.html#use-plain-console-logging

Didn't work :/

How do I use a different default dataset? Trying to run on Kubeflow where MemoryDataSet won't work. Want to change it to something like PickleDataSet. Seems like DataCatalogWithDefault is no longer in v0.18...

You can define the default dataset with a custom runner

Got it... thanks! https://kedro.readthedocs.io/en/latest/nodes_and_pipelines/run_a_pipeline.html#custom-runners

Oh no, actually it worked partially! The rich logging handler stops logging when I do this change, but the rich backtrace (when something breaks) is still there. This can be tested when you call a pipeline that has wrong inputs or a pipeline that does not exist, for example. So this is only for the console logging, not for the actual backtrace when an exception occurs

This may be the case of an issue

You can see this here: the upper part (the console handler) and below the backtrace still using rich

So you can disable tracebacks this way I think https://github.com/kedro-org/kedro/issues/1712#issuecomment-1189517668

No, this still makes rich the main backtrace. Thanks to this I have found a related issue that is interesting for this discussion: https://github.com/Textualize/rich/issues/2461