What I do is a spatial join - I have a polygon dataset, and I find the satellite images inside these polygons and download just them

Just what is inside the polygon

And then a dataset that would be many TB is now just 30gb

It's the magic of cloud optimized geotiffs

Oh so you want a general dataset that represents the whole bucket and then you grab specific parts of that bucket And I assume you want to use all the parts after that query and it will fit in ram So maybe using a standard/incremental custom dataset that takes in a query and a bucket and returns the applicable images in memory would be better? The you can optionally dump that in a tar file in an ingest pipeline so it's faster afterwards. Or in a better columnar format. I haven't worked with images much

One of the downsides in kedro is that it assunes you want to load a dataset in ram. No virtual datasets. Works for some use cases, maybe not yours

That would be the ideal case, but I don't think it is possible due to the massive size of the bucket. Instead I catalog all the files in the bucket in the CSV and I get only the files I need from the bucket.

I decided to fork the repo to make a possible implementation for Kedro. I'll do it and show you how it is done

Let's say I have some raw image data. I use a class to manually label objects within these images, which is done in a Jupyter Notebook, and this yields another dataset. This manual process is only performed once, and afterwards the pipeline continues as normal. Is there a way to create a dummy node for Kedro Viz just to demonstrate to others that the labelling process actually occurred?

I think the dummy node is the right call today. but I've wanted to push custom icons for ages- would you like something like that

I suppose I'd need to see an example of the implementation, but if it helps cut out meaningless code and communicate steps clearly, it's a yes from me 👍

I was thinking on the catalog entry providing a font awesome or material icon name, something like that

Could be interesting! I wonder how much demand there would be in general for this

@datajoely @antheas I have added a proof of concept of GDALRasterDataSet, with the expected functionalities for Cloud Optimized GeoTIFF (COG). There is a jupyter notebook showing how it works here: https://github.com/yurigba/kedro/blob/main/kedro/extras/datasets/rasterio/raster_dataset_usage.ipynb

There is a problem though, and that is why I had so many issues: GDAL virtual file systems (https://gdal.org/user/virtual_file_systems.html) kind of conflicts with fsspec

And for COG usage you need to enable virtual file systems

So there is a lot of problems to work around before working properly with geospatial data

But it is really possible if you give up on some stuff like fsspec or make it work around it

Hi, this would be great! could be easy to represent each main node role... stack, clean, join, predict.. etc..

If either of you want this feature pleasure comment on Kedro Viz issue https://github.com/kedro-org/kedro-viz/issues/480

looks good! And intuitive enough. I don't do geospatial though so my understanding stops there

Thanks! There is a lot of work needed to be merge-able but I need to make some decisions about how to implement virtual file systems properly and write a lot of unit tests

This will make it easier to make the other dataset that I proposed (the "partitioned dataset of partitioned datasets")

Is it possible to automate creating readme content using sphinx in kedro? https://stackoverflow.com/questions/73370726/is-it-possible-to-automate-creating-readme-content-using-sphinx-in-kedro

Hi! Is there any way to close the sql engines (created when the catalog is created) after the pipeline runs?

Hi @marioFeynman we made a related change to the SQL dataset last year https://github.com/kedro-org/kedro/pull/1163 This may provide some inspiration on how to think about things. Are the open connections causing issues?

You could probably get somewhere with an after pipeline run hook too

Yes, we are having some problems due a spike in the connections so i wanted to close all the connections that depends on me

I read about the singleton for the engines, so i will need to iterate somehow overall all the datasets objects, and close all the engines after the success/failed pipes?

That was my first idea... is that the recommended way?