Hi I am very interested in kedro. However I am not interested in the full workflow as described in Spaceflight turtorial. At my work we have an in house built python DAG framework based on dask and I would like to replace it with kedro. What I would like to do is create pipelines dynamically, and use mostly memory datasets in the catalogue. I fear for my use case, I may not be able to use kedro and kedro-viz how I would like. Does the pipelining part of kedro stand alone without the need setting up the kedro template, setting up the data folders and packaging of project?

I have a few follow up questions, when you say create pipelines dynamically, what are you trying to do? Kedro is not only framework, it's also a series of library components. It's possible to use the pipeline abstraction and even data catalog on its own. And we show that in our "Hello Kedro" example: https://kedro.readthedocs.io/en/stable/02_get_started/03_hello_kedro.html However, you don't get Kedro-Viz if you do this 🥴 Why is it that the project template won't work for you?

I would also add that Kedro workflow DAG and Dask exection DAG aren't mutually exclusive

Thank you for the replies. I'd like to think about this some more so I can explain my self adequately. Might be radio silence for a day or two