Other issue, is there a way to reduce the error message from cli?

You can set the level of the log to display via the

logging.yml

, but in general I don't think you want to hide error messages

thanks

how?

You simply update the LEVEL of logging that you desire

There should be a

logging.yml

, it's standard Python Logging module if you need more docs about it.

Even if I change it , the huge error message doesn’t disappear.

Thanks you anyway

I’m looking for a custom sequential runner. Any tips ?

Hello. Would someone be able to please tell me the best way to debug my pipelines? I use PyCharm. I wish to use the debugger. Which script should I run? Do I need to change anything in the run configuration? Thanks!

There is a tutorial for setting up pycharm run config in the docs! Then you just add breakpoints and use the debugger

Also if you're lazy you can use pdb and the

breakpoint()

statement

https://kedro.readthedocs.io/en/stable/development/set_up_pycharm.html Got you covered!

Hi All. How do I reference credentials from Azure backed key vault so that the data catalog can access the azure storage buckets?

You'll need to use a before catalog created hook to inject the credentials into our catalog

I think

That or tweak how your templated config loader works in settings.py

I added the credentials to my spark configuration and it worked. So I am not passing credentials to data catalog, just making them available at the cluster level.

That's the best way

Trying to adapt the spaceflight tutorial to run on Kubeflow but I get this error:

kedro.io.data_catalog - INFO - Loading data from 'data_science.candidate_modelling_pipeline.metrics' (MemoryDataSet)...

Since this is Kubeflow and MemoryDataset cannot be used, this is not surprising. I observe that

data_science.candidate_modelling_pipeline.metrics

is not mentioned in catalog.yml. Is the fix to simply modify catalog.yml to add in this dataset similar to the way

data_science.active_modelling_pipeline.metrics

is already in there? As in, was this just an oversight? In which case, I would have separate output files for active and cadidate metrics...

If I run

kedro catalog create --pipeline __default__

on the space tutorial, it generates a bunch of datasets not in the catalog:

data_science.active_modelling_pipeline.X_test:
  type: MemoryDataSet
data_science.active_modelling_pipeline.X_train:
  type: MemoryDataSet
data_science.active_modelling_pipeline.y_test:
  type: MemoryDataSet
data_science.active_modelling_pipeline.y_train:
  type: MemoryDataSet
data_science.candidate_modelling_pipeline.X_test:
  type: MemoryDataSet
data_science.candidate_modelling_pipeline.X_train:
  type: MemoryDataSet
data_science.candidate_modelling_pipeline.y_test:
  type: MemoryDataSet
data_science.candidate_modelling_pipeline.y_train:
  type: MemoryDataSet

Why aren't these included in

conf/base/catalog.yml

when their absence causes errors like

ValueError: Pipeline input(s) {'data_science.active_modelling_pipeline.y_train', 'data_science.active_modelling_pipeline.X_train'} not found in the DataCatalog

???

If I run ` kedro catalog create pipeline

Is it possible to access the traceback of a failed kedro pipeline? Like we want to call a rest api, where body of the api will have the error message of the failed kedro pipeline.

Sure, it's just Python program. You can do a try except block and return the traceback message if u want

@rohan_ahire https://github.com/kedro-org/kedro/issues/1846 We are quite interested to understand how people are exposing kedro pipeline via web api and see if there anything we can improve

How do I disable rich in kedro 0.18.3? For my use case, rich backtrace is too verbose and it doesn't integrate well with vscode.

Sometimes rich breaks the line and I can't just ctrl-click the file to get to the code in vscode, and this can become annoying over time

You can switch back to "plain" logging following these instructions: https://kedro.readthedocs.io/en/stable/logging/logging.html#use-plain-console-logging

Didn't work :/

How do I use a different default dataset? Trying to run on Kubeflow where MemoryDataSet won't work. Want to change it to something like PickleDataSet. Seems like DataCatalogWithDefault is no longer in v0.18...