We are currently using Kedro to build our pipelines to build our ML models.
The kedro documentation explains how to deploy a Kedro pipeline on AWS Step Functions where every Kedro node is an AWS Lambda function or running the entire pipeline in Sagemaker.
But our team wanted to deploy the pipeline to AWS and have some nodes run as a lambda ,some nodes(like the node that trains the model) as a sagemaker train job, some nodes that take a long time to run in a ECS/Sagemaker process job.
Our team wrote a plugin which manages to do this. The developer just adds a tag to a node(like "lambda"/"ECS"/"Sagemaker_train"). After that if the dev runs a command thats added by the plugin we parse the pipeline and based on the tags we use CDK to deploy the pipeline. The plugin has the CDK code needed to then deploy the pipeline on AWS Step functions
Before the team puts more effort into optimizing/adding more features to the plugin i wanted to check if a solution already exists to do this.
And if our approach to deploy the Kedro pipeline to AWS is correct.