Hey kedroids, looking for a code style opinion here, is there any way to pass inputs directly into pd.concat without an extra function?

python
node(lambda *frames: pd.concat(frames), ["cars", "cars"], "two_cars")

How would you concatenate pandas dataframes?

This is something I've ponded before and decided that what you're doing here is as good as you can get it. The basic problem is that

pd.concat

takes a single iterable argument rather than

*args

. If you're doing this sort of thing a lot then you could probably a decorator that effectively does that conversion for you, but if it's just for

pd.concat

then what you're doing here is the best you can do really

I also think you can just do:

py
node(pd.concat, ["cars", "cars2"], "two_cars")

Since it a sequence or mapping already https://pandas.pydata.org/docs/reference/api/pandas.concat.html

Tried that before asking. I thought there should be a way @datajoely , but kedro does not return a sequence or mapping.

Ah interesting

could be nice to have a kedro node flag to pass a list of loaded data rather than unpacking the data to your function, but I there probably are not enough use cases to maintain it.

Yeah I've also noted you can't override kwargs with modular pipelines so there's work to be done in this

Is the partial application similar to what I have here?

Yes you have a good doc on it

lol, which one?

awesome, I never actully finished that one.

One request - add anchor tags to the headings so I can link to the right parts 💪

they are there

One step ahead!

added those to all posts a few months back

@waylonwalker here's the decorator I mentioned earlier 🙂

import functools

def inputs_to_sequence(func):
    @functools.wraps(func)
    def wrapper(*args):
        return func(args)
    return wrapper

pd.concat = inputs_to_sequence(pd.concat)

And then you can do directly

node(pd.concat, ["cars", "cars2"], "two_cars")

I'll steal this for the article linked above

If you like! A better version of it would also do the same thing with

**kwargs

I think, since the argument to

pd.concat

could also be a mapping