https://kedro.org/ logo
#beginners-need-help
Title
# beginners-need-help
w

waylonwalker

05/25/2022, 4:06 PM
Hey kedroids, looking for a code style opinion here, is there any way to pass inputs directly into pd.concat without an extra function?
Copy code
python
node(lambda *frames: pd.concat(frames), ["cars", "cars"], "two_cars")
How would you concatenate pandas dataframes?
a

antony.milne

05/25/2022, 4:17 PM
This is something I've ponded before and decided that what you're doing here is as good as you can get it. The basic problem is that
pd.concat
takes a single iterable argument rather than
*args
. If you're doing this sort of thing a lot then you could probably a decorator that effectively does that conversion for you, but if it's just for
pd.concat
then what you're doing here is the best you can do really
d

datajoely

05/25/2022, 4:31 PM
I also think you can just do:
Copy code
py
node(pd.concat, ["cars", "cars2"], "two_cars")
Since it a sequence or mapping already https://pandas.pydata.org/docs/reference/api/pandas.concat.html
w

waylonwalker

05/25/2022, 7:40 PM
Tried that before asking. I thought there should be a way @datajoely , but kedro does not return a sequence or mapping.
d

datajoely

05/25/2022, 7:56 PM
Ah interesting
w

waylonwalker

05/25/2022, 8:17 PM
could be nice to have a kedro node flag to pass a list of loaded data rather than unpacking the data to your function, but I there probably are not enough use cases to maintain it.
d

datajoely

05/25/2022, 8:17 PM
Yeah I've also noted you can't override kwargs with modular pipelines so there's work to be done in this
There is also the age old question of if we should provide a Literal() method so people can plug in python data without partial application
w

waylonwalker

05/25/2022, 8:21 PM
Is the partial application similar to what I have here?
d

datajoely

05/25/2022, 8:21 PM
Yes you have a good doc on it
But we could make it easier
I link to that everytime it comes up
w

waylonwalker

05/25/2022, 8:22 PM
lol, which one?
d

datajoely

05/25/2022, 8:23 PM
w

waylonwalker

05/25/2022, 8:23 PM
awesome, I never actully finished that one.
That's what I thought you were talking about.
I should finish it and get that google juice flowing on it.
d

datajoely

05/25/2022, 8:26 PM
One request - add anchor tags to the headings so I can link to the right parts 💪
w

waylonwalker

05/25/2022, 8:26 PM
they are there
d

datajoely

05/25/2022, 8:27 PM
One step ahead!
w

waylonwalker

05/25/2022, 8:27 PM
added those to all posts a few months back
Do you have other patterns you would add?
a

antony.milne

05/25/2022, 9:54 PM
@waylonwalker here's the decorator I mentioned earlier 🙂
Copy code
import functools

def inputs_to_sequence(func):
    @functools.wraps(func)
    def wrapper(*args):
        return func(args)
    return wrapper

pd.concat = inputs_to_sequence(pd.concat)
And then you can do directly
Copy code
node(pd.concat, ["cars", "cars2"], "two_cars")
w

waylonwalker

05/25/2022, 9:56 PM
I'll steal this for the article linked above
can I tag you in it?
a

antony.milne

05/25/2022, 10:01 PM
If you like! A better version of it would also do the same thing with
**kwargs
I think, since the argument to
pd.concat
could also be a mapping