Any example of a node that lazily loads and lazily saves a p

Question

Any example of a node that lazily loads and lazily saves a partitioned dataset while performing some transformations I ve seen the examples on this page https kedro readthedocs io en stable data kedro io html partitioned dataset save but am having trouble wrapping my head around how this is done in a node that doing a few transformation to a large dataset

datajoely · Answer

So this is at the very front of my mind as several people have asked about this lately

datajoely · Answer

https github com kedro org kedro issues 1413 issuecomment 1093200414

datajoely · Answer

I really want to improve the example but this is the important part ```python return part foo lambda pd DataFrame data 1 2 part bar lambda pd DataFrame data 3 4 ```

datajoely · Answer

the dictionary you return is keyed on the partition name the values are `callable` functions you can use a `def` or like this example use an anonymous lambda The dataset then saves each partition in a loop but it a way much more memory efficient than doing it all within the node

Edak · Answer

So to do this with an input of a partition

Edak · Answer

``` return key lambda preprocess partion load func for key load func in partitioned input items ```

datajoely · Answer

That is my understanding

datajoely · Answer

is it working

Edak · Answer

It seemed like it did

datajoely · Answer

If it does I might include that example straight into the doc using a comprehension is really neat

datajoely · Answer

awesome

Edak · Answer

Cool I commented it on that thread in case it could help anyone else

Zhee · Answer

this is a very valuable example for sure it would be very useful in the docs