https://kedro.org/ logo
#beginners-need-help
Title
# beginners-need-help
j

j c h a r l e s

12/30/2021, 9:56 AM
I'm guessing that I am building a modular pipeline using an anti pattern. I generate a base set of entities and entity attributes in
conf/base/parameters/<pipeline-name>.yml
. I use a custom hook that iterates through this yml file and uses
catalog.add_all(new_entries, replace=True)
to generate all the downstream datasets. Then in each pipeline, rather than mapping inputs and outputs, I again iterate through this same set of entities in
conf/base/parameters/<pipeline-name>.yml
and use functools.partial in order to create the node
func
, freezing the input values to be values found in this parameters file.