Hi ! actually i'm strugling with a resources issues, because i have to process a lot of images with high resolution and the node processing have to store them in RAM and that raises OutOfMemory issue. so i think of processing that sequentially in the node ( the node process image then save it, not like now all images are processed then saved in catalog). i found that it can be doable with custom runner isn't it ? any tips for that ?

So I think you probably could do it with a custom runner

Thank you ! is the catalog released automaticly ? or should i do it manuelly ? But my main issue is that i have a datacatalog that i can't process once with a node so how can i batch the processing

Okay so Kedro will release at the end of the pipeline run (I think)

Perfect ! Yes actually the issue is on loading all datacatalog on memory for processing. so is there any trick to do batch processing of datacatalog or somthing like this ( load_image, process, save in catalog. then repeat for all other images to avoid OOM)

Is it image datasets?

Yes exatly i'm using a custom image dataset with partitionedDataset

Okay have you tired

IncrementalDataSet

it may help here

I think the incrementalDataset is not relevent in this case

today - the partitioned dataset is eager

actually PartitionedDataset provide a loader function to load image isn't it ? so the image is not loaded to memory automatically

so I think the highlighted part actually loads it

i see your point. but don't you think that the loading isn't a problem since we can use the load function to load image within loop, but i think we should consider how to save images gradually to not keep them in RAM. Currently we have to keep it in RAM till the node run ends to save it in datacatalog

ah I'm with you

Yeaaah ! i really apreciate you help 😄

We essentially need ways of running the node on small batches

actually i thought of saving images manually whitout using datacatalog but is not the proper way i think

well that would work - and we don't currently offer a better solution

exatly that what i thought about

so maybe to get you going thats not a bad idea

Yes ! if there is no better way i think i have no choice 😅

Yeah I'm just whiteboarding a hook idea

Cool !

so I've done something a bit crazy but I think it may work

Wow you rock !

let me know if it works

so i can do the processing in the hook part and save directly in cataalog

if it does I'd love to do come up with an offical way of doing this

Of course !

I think there is a lot of merit in this being generic

Ok i will try it , thank you!

there is a core assumption you have the same number of partitions as inputs as you do outputs

yes i think too

🤞

@idriss__ Sorry for the late reply (found this through the issue on GitHub), but what you can do is return a dictionary with `Callable`s as values from your node in order to save lazily using

PartitionedDataSet

.

This a really cool idea - could you provide a snippet?

The example (https://kedro.readthedocs.io/en/stable/data/kedro_io.html#partitioned-dataset-save) of returning a dictionary with

Callable

values doesn't seem to have the expected behaviour. The point is to save each partition one at a time right? It seems that all the files are saved at the same time (see screenshot); each file actually took 4 minutes to process. So it seems like the outputs are stored in RAM until the end of the node. Or am I missing something?

I am having some problems lazyly saving parquet files

If you look at the implementation its definitely running in a for loop https://kedro.readthedocs.io/en/stable/_modules/kedro/io/partitioned_dataset.html#PartitionedDataSet

I just discovered what happenned with the same writing (Exactly the same), if your files are inside another folder it will write all at once, once I removed the subfolder it worked