Ok, a follow-up question on the previous. I managed to set up the layers as you and the documentation suggested.
However, won't this create problems if I were to run multiple pipelines in parallell? Essentially, my pipeline takes as it's input two unique parameters (one customer domain, and on competitor domain) and then runs based on that info. I intend to use this pipeline for many combinations of customer domain and competitor domain, but if I do this is parallell then the created datasets will all be saved under the same name in the folder 01_raw. Surely this will be problematic if I run the same pipeline in parallell?
This is what I added to my catalog:
customer_df:
type: pickle.PickleDataSet
filepath: data/01_raw/customer_df.pickle
layer: raw
competitor_df:
type: pickle.PickleDataSet
filepath: data/01_raw/competitor_df.pickle
layer: raw
previous_matches_df:
type: pickle.PickleDataSet
filepath: data/01_raw/previous_matches_df.pickle
layer: raw