If I run ` kedro catalog create pipeline default ` on the sp

Question

If I run ` kedro catalog create pipeline default ` on the space tutorial it generates a bunch of datasets not in the catalog ``` data science active modelling pipeline X test type MemoryDataSet data science active modelling pipeline X train type MemoryDataSet data science active modelling pipeline y test type MemoryDataSet data science active modelling pipeline y train type MemoryDataSet data science candidate modelling pipeline X test type MemoryDataSet data science candidate modelling pipeline X train type MemoryDataSet data science candidate modelling pipeline y test type MemoryDataSet data science candidate modelling pipeline y train type MemoryDataSet ``` Why aren t these included in `conf base catalog yml` when their absence causes errors like `ValueError Pipeline input s data science active modelling pipeline y train data science active modelling pipeline X train not found in the DataCatalog`

Goss · Answer

Is this happening because intermediate datasets are MemoryDataSet and work when running locally but on a distributed platform like Kubeflow they don t

datajoely · Answer

They won t work if you make each kedro node at kuberflow task

datajoely · Answer

You may also have to change some to persisted if they are shared between pipelines

Goss · Answer

Thanks It sounds like there is a way to control the granularity of how Kedro nodes map to tasks in Kubeflow based on your comment Can you elaborate on that or point me to any docs I couldn t find anything by searching the docs