I'm guessing that I am building a modular pipeline...
# beginners-need-help
I'm guessing that I am building a modular pipeline using an anti pattern. I generate a base set of entities and entity attributes in
. I use a custom hook that iterates through this yml file and uses
catalog.add_all(new_entries, replace=True)
to generate all the downstream datasets. Then in each pipeline, rather than mapping inputs and outputs, I again iterate through this same set of entities in
and use functools.partial in order to create the node
, freezing the input values to be values found in this parameters file.