I am not sure I understand the problem here?

You can always use a more functional approach like

map

, but for loop is fine too.

I think you mentioned your problem is with a big dataset, the problem with

pandas

is that it is memory hungry, especially during I/O and certain operations. Using the

chunk

args helps to mitigate this problem by only loading & processing small batch of data and stitch them by at the end. If the new dataset already iterate through the entire dataset before you start applying any transformation logic, then it doesn't help your memory problem.

message has been deleted

I can understand the logic, could you repeat what's the question?

Oh you shouldnt do the writing yourself in your node, this is what PartitionedDataSet is for

What type of DataSet can deal with huge dataset that does not fit in memory

So if you return callables to PartitionedDataset it will do in a lazy way

There are examples in the docs

I would probably add that choose format other than csv is better, unless you have to stick with csv.

Which one do you recommend?

Parquet/feather? @datajoely will be the right person to answer that

@noklam @datajoely thanks a LOT!

Team Parquet here!

Just curious if you have any experience with arrow. It's something on my radar but I have never used it😅

Arrow is the modern engine that Parquet can leverage so they're complementary rather than substitutes - this article by the creator of Pandas on the topic is one my favourites https://wesmckinney.com/blog/apache-arrow-pandas-internals/

Yes I have read this article a couple of times. I think you are referring Arrow as a in memory columanr data structure in this context. Feather is kind of like the mapping of this data structuring on disk as a storage, but I haven't seen it used widely.

Hi all, if I want to pass a parameter to my node I can do so by writing:

node(
    func=foo,
    inputs=['input', 'params:parameter'],
    outputs='output'
),

Is there something similar I can do to pass a credential from the

credentials.yml

to my node?

We don't support this by design, you could do a

parameters.yml

in your

local

folder, but typically your nodes should focus on data flow not IO

thanks

Hey kedroids, looking for a code style opinion here, is there any way to pass inputs directly into pd.concat without an extra function?

python
node(lambda *frames: pd.concat(frames), ["cars", "cars"], "two_cars")

How would you concatenate pandas dataframes?

Cunning pd.concat

Hello, how can I catch a DataSetError? I tried "except kedro.io.data_catalog.DataSetError" but then I get NameError: name 'kedro' is not defined...

I think you just need to do

from kedro.io.data_catalog import DataSetError

before you do this

+ do

except DataSetError

rather than the full classpath

it worked, thank you!

I have a quick question on experiment logging: for example if I want to generate 10 models, I understand I can log all 10 hyperparameters in the JSON. how can I save models as 10 pickle files and appended into the config.yaml file?

Hey so there are a few things here: - To reuse the same pipeline multiple times I would recommend reading our modular pipeline docs https://kedro.readthedocs.io/en/stable/nodes_and_pipelines/modular_pipelines.html - To log parameters you should use the experiment tracking features https://kedro.readthedocs.io/en/stable/tutorial/set_up_experiment_tracking.html - To save pickles you can find example on the catalog docs just search form

pickle.PickleDataSet

https://kedro.readthedocs.io/en/stable/data/data_catalog.html

This reminds me of a question that I used to have. When doing a hyperparameter search, I need to train K (dynamic number, could be 10 or 20) models, then later in the pipeline should pick the best model to continue. What's the best way to handle the K entry in catalog?