Hi everyone - *TLDR: I am looking for Best Practices to use Kedro to leverage Azure Machine Learning features and using MLOps techniques. *
I will try to be clear but please let me know if I am not. I have shared what we are trying to build.
Some context
We are developing a model to be used as a prediction service by an existing application on AKS. We want to implement as much as possible "best in class" methods for MLOps using Azure Machine Learning features (Experiment tracking, model registry, maybe Azure ML Pipelines, etc.). Our code will reside in Azure Devops so we are also thinking about how to use Azure pipelines in the mix.
Problematic
Our DS is a very big fan of the Kedro framework. I am looking for the best way to have him adapt the framework to use the features of Azure ML to train, track, register, monitor datasets and models and deploy them to our different workload environments.
I have so many questions I am not even sure where to start.
- I would like to know if there are best practices to launch kedro pipeline runs after a new code push automatically in the Azure Machine Learning Service. Either for running a specific kedro pipeline or for building a whole Azure ML Pipeline exactly like the Kedro one?
- What parts of the infrastructure should be leveraging the Azure SDK, and what parts should be just Azure CLI commands in the CICD pipelines?
If you have any resource or example of someone successfully implementing this kind of thing it would be super helpful!