In a similar line to the dynamic catalog, has anyo...
# beginners-need-help
w
In a similar line to the dynamic catalog, has anyone done dynamically generated nodes inside of a pipeline based on params in the catalog? I am utilizing a for loop based on a list in the yaml to generate nodes. However, to do this I created a basic function which acquired the params from the context. However, when pip-installing a project and running a pipeline from the pip-install, this method seems to fail due to the
__main__.py
file attempting to configure the pipelines prior to the session setting the Context with
configure_project(Path(__file__).parent.name)
. Because the data files are not necessarily saved within cwd it is causing the pipeline registration to fail. I tried to create a ConfigLoader with the correct location if a session wasn't present, but this just seems to make the entire pipeline hang. Any idea how to get around
configure_project
?
3 Views