Maybe I'll have to use something like a tar file, I'll see what happens

How many subfolders? Less than 20 I'd use the first approach with N partitioned datasets

I have a node that will receive 1-2 dozen datasets when I'm done with it

It's a lot of folders

It's satellite imagery

I have been successfully using Kedro as a geospatial data pipeline for machine learning

And I am making improvements right now

Oh so it's a directory structure that ends in folders with images

Yes, and images that have regular structure between folders

How many gigs?

about 30gb of images

But distributed in small pieces of about 200kb

Super cool usecase!

I wanted so bad to contribute for this kind of dataset, but I didn't find the time until recently

All I have written is closed source

I've been keen to improve PartitionedDataSet in general so this could feed into a new design

If you want to understand better the case, take a look at this bucket: https://registry.opendata.aws/sentinel-2-l2a-cogs/

It's all based on this folder structure in

s3://sentinel-cogs/sentinel-s2-l2a-cogs/

In the end you have some files that are images that represent the same location in space. However, each image has different data in different resolution

I have built a GDALRasterDataSet that can open geospatial data with some interesting functionalities, like cropping with a specified polygon (directly reading from S3)

Kedro really has a good framework for this

But I'd like to work with all image files like a single one

Because they represent literally the same place

Sounds like partitioned dataset is the proper use case. Provided you can figure out how you'd shard your initial dataset If you use a compressed format afterwards would you get a performance benefit? then I'd be keen to dump the partitions on that format so they're available locally and faster afterwards. Some food for thought

Compression would hurt performance a lot

The thing is that if I use compression I would not be able to use this data source (the s3 bucket), since I cannot change it

About that, you said 30gb. Is it updating?

The source is far more

But when I have a specified dataset it is 30gb

It is not changing