09/01/2022, 11:22 PM
The problem with kedro pipeline being one single task in a larger dag is because it could be a big complex pipeline taking a long time to execute and we would not want to restart the entire kedro pipeline if only certain tasks fail. Also we lose the visualization feature as well when we deploy. But I like your idea about smaller kedro pipelines being part of a larger dag. Will have to think about how to deploy kedro pipelines in production. We need to figure out a way to deploy while making sure we reuse the modularity that kedro provides and the faster development time it provides.