Title
#beginners-need-help
antony.milne

antony.milne

12/03/2021, 10:39 AM
This is a great question. I actually helped develop exactly something like this a while ago. I'd like to write it up properly in the docs as it's a great example of how you can use modular pipelines. We wanted a forecasting model that predicts something for 2020, and then the output of that is used an input for the same pipeline running for 2021, which in turn feeds the pipeline for 2022, and so on for 10 years… It can be done with something like this in your pipeline registry:
base_pipeline = Pipeline([node(func, "input_data", "output_data")])
# in reality base_pipeline would have many nodes

all_pipelines = {}
for year in range(2020, 2030):
    all_pipelines[f"year_{year}"] = pipeline(
        base_pipeline,
        outputs={"output_data": f"year_{year+1}.input_data"},
        namespace=f"year_{year}"
    )

all_pipelines["all_years"] = sum(all_pipelines.values())