Hi! I'm running a pipeline on an HPC, scheduling j...
# advanced-need-help
i
Hi! I'm running a pipeline on an HPC, scheduling jobs with slurm and I would like to run the same pipeline with different inputs. Until now I always used the TemplateConfigLoader to achieve this, changing the variables in the globals.yml, but now I would like to run the pipelines for all the inputs in parallel taking advantage of slurm. The problem I observed is that using the same catalog and changing the globals for a different pipeline run, if the jobs starts at the same time the catalog used during all runs may be the wrong one. Is there a way to use a catalog stored in a different folder passing the path from the cli to be sure that each pipeline is using a different globals.yml? Or how would be the best practice to avoid using the wrongly rendered catalog or the wrong globals.yml? Thanks!
3 Views