Can you explain why it is fragile?

We don't typically like dynamic pipelines since they make reproducibility difficult.

And by that extension non-determism makes things difficult to debug.

Got it! Our biggest struggle is that there is no clean way to add conditional logic in the pipeline (that is dependent on the

KEDRO_ENV

). And on top of that, we want to be able to more dynamically construct pipelines using logic that generates additional nodes during registry.

@Matthias Roels I had a similar problem. Kedro is written in such a way that both the catalog and the pipelines must be static

I use multiple datasets, multiple algorithms and want to run the same pipeline on them, but they have a different number of tables with different names and the algorithm has a different io

the way I solved it was by carefully namespacing my catalog and creating it dynamically based on the combination of datasets, algs. So, when i load up kedro ipython I can access all of my datasets

and then enumerating between all combinations of datasets, algs in my registry. My pipelines have a name of . or if I want to run part of the pipeline ... Example: cars.yolov5.measure

I have 32 pipelines and kedro launches in around ~2s if that's useful to you

I'd love to showcase a demo of this - this sounds super neat

Hi guys, I'd like to know a way to build a dataset class kind of a PartitionedDataSet but the filenames are fixed. Thanks in advance!

Could you elaborate more on what you mean?

In the sense that instead of PartitionedDataSet searching every file in the directory of a certain suffix, it searches a list of pre-determined filenames. If the load method finds the specified list of filenames, it loads it, just like PartitionedDataSet. If there are files missing, it can return empty or raise an error. I tried overloading the

__init__

method of PartitionedDataSet but I think there should be a better way

Subclassing to do this sort of custom functionality is very much encouraged

It makes sense to use a PartitionedDataSet instance but also maybe it should be easier doing a specific dataset that does this

I want to write something that is easier for someone who reads the code to maintain

The things that I imagine are quite complicated

I think the best way is subclassing too but maybe there was something already done

I will do something and if I have problems I'll ask here again. Thank you @datajoely

Quick question: in mid June, Spark 3.3 was released. I noticed that kedro 0.18.x is not yet compatible with this release because of the strict delta lake 1.x dependency. Will that be resolved soon?

Thanks for raising this - I've just raised a PR to resolve this so it will be in the next version.

We use Kedro for all of use cases, and we are assessing if adding a Feature Store like Feast to manage all the features that we use. Anyone has had any experience integrating the kedro data catalog for getting historical features and online features? using the same dataset in the catalog? or would it take a big development in the Kedro Datasets side?

It's a research prototype for data synthesis. I will keep it in mind when I get close to publishing the source code

You can use the extension syntax for yaml files to define your datasets

_dataset_csv: &dataset_csv
  type: pandas.CSVDataSet
  layer: raw
  filepath: ""

dataset_2021_2:
  <<: *dataset_csv
  filepath: ${base_location}/raw/x/t/z.csv.gz

Then you can instantiate your pipeline with a node that has as an input a dictionary {n: n for n in your_dataset_names}. You can even use jinja2 to template it. After that you can just dump everything in a parquet file, assuming it fits in ram. If you don't like the fact that this causes duplication (you need to define your dataset names both in your catalog and in your code), then you can instantiate your datasets using the after_catalog_created hook using python. Ofc this assumes that you don't need to use the partitioned features like lazy loading. I did check out the PartitionedDataset class. It's just 300 lines. What you want should be manageable by overwriting

__init__

, adding a list param for your datasets and then calling the

super().__init__()

method. Then you can use that list in

_list_partitions()

to return your files instead of a directory listing.

or you could just stick your files in their own directory and rename them so

PartitionedDataset

works without changes 🤷‍♂️

Working on this right now. I need the lazy features too, so I am going for the second approach. Let's see how well it goes.

The thing is that I want to be able to use a PartitionedDataSet of PartitionedDataSets. This is some borderline case that I want do test.

I have a folder structure that has sub-folders and these sub-folders have some specific files each

I want to specify each sub-folder as a single dataset

This feels slightly out of the bounds of what we've tested so far. If you get it working I'd be interested in trying to support this natively