Hi I am very interested in kedro. However I am not...
# beginners-need-help
Hi I am very interested in kedro. However I am not interested in the full workflow as described in Spaceflight turtorial. At my work we have an in house built python DAG framework based on dask and I would like to replace it with kedro. What I would like to do is create pipelines dynamically, and use mostly memory datasets in the catalogue. I fear for my use case, I may not be able to use kedro and kedro-viz how I would like. Does the pipelining part of kedro stand alone without the need setting up the kedro template, setting up the data folders and packaging of project?
I have a few follow up questions, when you say create pipelines dynamically, what are you trying to do? Kedro is not only framework, it's also a series of library components. It's possible to use the pipeline abstraction and even data catalog on its own. And we show that in our "Hello Kedro" example: https://kedro.readthedocs.io/en/stable/02_get_started/03_hello_kedro.html However, you don't get Kedro-Viz if you do this 🥴 Why is it that the project template won't work for you?
I would also add that Kedro workflow DAG and Dask exection DAG aren't mutually exclusive
Here is a walkthrough from the dask folks at coiled showing how to use Kedro and Dask together https://www.linkedin.com/posts/gustafrcavanaugh_python-ml-dask-activity-6845770788231106560-hx0v/
here are the latest dask docs on our side https://kedro.readthedocs.io/en/latest/10_deployment/dask.html
Thank you for the replies. I'd like to think about this some more so I can explain my self adequately. Might be radio silence for a day or two