Do people usually create/edit data catalog YAML file using python in their pipelines? I have a pipeline with known number of output datasets with consistent meaning and naming (

file_A

,

file_B

,

file_C

). I want the folder that this pipeline runs in have its own dynamically generated data catalog so other people can go in and inspect the results from the pipeline easily by just taking the example from https://kedro.readthedocs.io/en/latest/05_data/01_data_catalog.html#configuring-a-data-catalog , is it possible to do this:

python
io = DataCatalog(
    {
        "bikes": CSVDataSet(filepath="../data/01_raw/bikes.csv"),
        "cars": CSVDataSet(filepath="../data/01_raw/cars.csv", load_args=dict(sep=",")),
        "cars_table": SQLTableDataSet(
            table_name="cars", credentials=dict(con="sqlite:///kedro.db")
        ),
        "scooters_query": SQLQueryDataSet(
            sql="select * from cars where gear=4",
            credentials=dict(con="sqlite:///kedro.db"),
        ),
        "ranked": ParquetDataSet(filepath="ranked.parquet"),
    }
)

and then do

io.to_config()

? we have

io.from_config()

but not

io.to_config()

to generate YAML file from the data catalog object

Hi @User - good question

from my understanding, the jinja 2 support is for looping of subsets of datasets, I don't usually do it since I believe the deterministic, written-out datasets are easier to maintain and understand

I don't think we have a mechanism for that - but do you want to do it so that you then re-read that YAML in via Kedro?

what I'm trying to do is, for each set of parameters of the pipeline, create a folder

param1_X_param2_Y

with files

[bikes.csv, cars.csv, etc]

and a data catalog that documents these datasets. Another folder

param1_A_param2_B

with the same set of files but content of the files are different This way, another person can go into a folder and explore these datasets for each parameter combinations and do subsequent analyses without worrying the filepaths

So is this to have a pre-configured project for people to investigate?

initializing

DataCatalog

pointing to

param1_A_param2_B

will load same dataset names but different files

so

it's about making the datasets output from a pipeline easier to navigate

we do have something that make make this easier

what if the number of datasets change between

base

and

local

, for example when you work in cloud you need to output additional files? I thought that the number of datasets and what they are have to be exactly the same in every environments

no they can be diff

also is there a way to export the

DataCatalog

in the example above into YAML?

no

gotcha! thanks so much for the help! Is that something that you guys would accept a PR for?

I think so

I think I get what you mean. I can manually add another dataset to the YAML in "local" and not in base Is there an example of a pipeline that outputs datasets

A, B, C

when run in environment A but outputs dataset

A, B, D, E

when run in environment B?

but it's buried in this part of the CLI https://github.com/quantumblacklabs/kedro/blob/17fbecc9c60c37c2078f7a190ddb0cf5101b65fb/kedro/framework/cli/catalog.py#L139\

Ah! I think that only creates

MemoryDataset

for every missing dataset not in the catalog right?

yes

gotcha, different pipelines with subset of shared nodes.

It's fine here! now discord has these fancy threads I'm not worried. In general dynamic pipelining is an advanced topic, but this is before we even run the thing!