https://kedro.org/ logo
#beginners-need-help
Title
# beginners-need-help
e

Evolute

05/29/2022, 8:53 PM
Ok, a follow-up question on the previous. I managed to set up the layers as you and the documentation suggested. However, won't this create problems if I were to run multiple pipelines in parallell? Essentially, my pipeline takes as it's input two unique parameters (one customer domain, and on competitor domain) and then runs based on that info. I intend to use this pipeline for many combinations of customer domain and competitor domain, but if I do this is parallell then the created datasets will all be saved under the same name in the folder 01_raw. Surely this will be problematic if I run the same pipeline in parallell? This is what I added to my catalog: customer_df: type: pickle.PickleDataSet filepath: data/01_raw/customer_df.pickle layer: raw competitor_df: type: pickle.PickleDataSet filepath: data/01_raw/competitor_df.pickle layer: raw previous_matches_df: type: pickle.PickleDataSet filepath: data/01_raw/previous_matches_df.pickle layer: raw