Working on this right now. I need the lazy features too, so I am going for the second approach. Let's see how well it goes.

The thing is that I want to be able to use a PartitionedDataSet of PartitionedDataSets. This is some borderline case that I want do test.

I have a folder structure that has sub-folders and these sub-folders have some specific files each

I want to specify each sub-folder as a single dataset

This feels slightly out of the bounds of what we've tested so far. If you get it working I'd be interested in trying to support this natively

Maybe I'll have to use something like a tar file, I'll see what happens

How many subfolders? Less than 20 I'd use the first approach with N partitioned datasets

I have a node that will receive 1-2 dozen datasets when I'm done with it

It's a lot of folders

It's satellite imagery

I have been successfully using Kedro as a geospatial data pipeline for machine learning

And I am making improvements right now

Oh so it's a directory structure that ends in folders with images

Yes, and images that have regular structure between folders

How many gigs?

about 30gb of images

But distributed in small pieces of about 200kb

Super cool usecase!

I wanted so bad to contribute for this kind of dataset, but I didn't find the time until recently

All I have written is closed source

I've been keen to improve PartitionedDataSet in general so this could feed into a new design

If you want to understand better the case, take a look at this bucket: https://registry.opendata.aws/sentinel-2-l2a-cogs/

It's all based on this folder structure in

s3://sentinel-cogs/sentinel-s2-l2a-cogs/

In the end you have some files that are images that represent the same location in space. However, each image has different data in different resolution

I have built a GDALRasterDataSet that can open geospatial data with some interesting functionalities, like cropping with a specified polygon (directly reading from S3)

Kedro really has a good framework for this

But I'd like to work with all image files like a single one

Because they represent literally the same place

Sounds like partitioned dataset is the proper use case. Provided you can figure out how you'd shard your initial dataset If you use a compressed format afterwards would you get a performance benefit? then I'd be keen to dump the partitions on that format so they're available locally and faster afterwards. Some food for thought