You can use the extension syntax for yaml files to...
# advanced-need-help
You can use the extension syntax for yaml files to define your datasets
Copy code
_dataset_csv: &dataset_csv
  type: pandas.CSVDataSet
  layer: raw
  filepath: ""

  <<: *dataset_csv
  filepath: ${base_location}/raw/x/t/z.csv.gz
Then you can instantiate your pipeline with a node that has as an input a dictionary {n: n for n in your_dataset_names}. You can even use jinja2 to template it. After that you can just dump everything in a parquet file, assuming it fits in ram. If you don't like the fact that this causes duplication (you need to define your dataset names both in your catalog and in your code), then you can instantiate your datasets using the after_catalog_created hook using python. Ofc this assumes that you don't need to use the partitioned features like lazy loading. I did check out the PartitionedDataset class. It's just 300 lines. What you want should be manageable by overwriting
, adding a list param for your datasets and then calling the
method. Then you can use that list in
to return your files instead of a directory listing.