08/29/2022, 7:49 PM
Let's say I have a modular execution pipeline which is supposed to use a trained model to analyse N datasets. I understand how to create N instances of my modular pipeline (one for each dataset), but I was hoping there was some way to auto-generate catalog entries for these datasets at runtime. ie. I would have
dataset_1_result, dataset_2_result, ..., dataset_N_result
in the catalog without having to write them in the catalog manually. I understand it's just yaml input at the end of the day, and I could just write a hacky script to populate the catalog with these entires, but just wondered if there's a more "Kedronic" way to accomplish this. Cheers!