Hi I am very interested in kedro. However I am not interested in the full workflow as described in Spaceflight turtorial. At my work we have an in house built python DAG framework based on dask and I would like to replace it with kedro. What I would like to do is create pipelines dynamically, and use mostly memory datasets in the catalogue.
I fear for my use case, I may not be able to use kedro and kedro-viz how I would like.
Does the pipelining part of kedro stand alone without the need setting up the kedro template, setting up the data folders and packaging of project?
03/08/2022, 9:14 AM
I have a few follow up questions, when you say create pipelines dynamically, what are you trying to do?
Kedro is not only framework, it's also a series of library components. It's possible to use the pipeline abstraction and even data catalog on its own. And we show that in our "Hello Kedro" example: https://kedro.readthedocs.io/en/stable/02_get_started/03_hello_kedro.html
However, you don't get Kedro-Viz if you do this 🥴 Why is it that the project template won't work for you?
03/08/2022, 10:17 AM
I would also add that Kedro workflow DAG and Dask exection DAG aren't mutually exclusive