https://kedro.org/ logo
#advanced-need-help
Title
# advanced-need-help
d

datajoely

08/04/2022, 9:29 AM
It's a good question, I don't think it's wise to just map feature to feature as you'll end up limiting yourself to just the overlapping parts. The path of least resistance from a kedro point of view is to use a Azure VM / cluster and data lake storage. The pipeline, experiment etc primitives azure ml provides aren't great accelerators.
f

Flow

08/04/2022, 10:34 AM
Could you expand on this? I have been playing around with azureml and writing some scripts and plugins similar to kedro kubeflow and airflow to “translate” dags into the azure version of it. However I find a lot of the azureml things good in theory but cumbersome in practice… however it is kind of integrated within itself so in theory one does not need a whole set of different open source solutions. Would be super interesting to hear thoughts
Might just be “tried it didn’t like it” which I would find reassuring 😂
c

chris

08/04/2022, 11:20 AM
the best thing i came up with in our dev environment was to dockerize the kedro pipeline and run it as a single pythonscriptstep node in azureml, all works fine but we kind of lose the scalability
/the visibility the operations team wants to have from using the platform
d

datajoely

08/04/2022, 11:59 AM
I wonder if you can use hooks to expose the run lifecycle in a way that azure can visualize
y

Yetunde

08/04/2022, 12:17 PM
I did come across this plugin; we like the Get In Data team: https://github.com/getindata/kedro-azureml
c

chris

08/04/2022, 1:16 PM
this looks actually perfect, will give it a go
f

Flow

08/05/2022, 10:40 AM
ah nice I guess I can stop doing my kedro-azureml then 😄
thanks @Yetunde for sharing
d

datajoely

08/05/2022, 10:58 AM
Please share you're experiences back to us, we'd love to hear how it goes
y

Yetunde

08/05/2022, 3:25 PM
@Flow Maybe contribute to this project? We love seeing things like this! Let us know if you have feedback you can give to this team too.
Additionally, we're prioritising Databricks and then the AWS ecosystem to see how Kedro could better integrate with them. Down the line, Azure ML would be next, so we'll swing back to you with questions to help inform what we build.
f

Flow

08/05/2022, 3:47 PM
Yeah for sure. There is no space for ego in open source. I’ll check their things out
e

em-pe

08/31/2022, 8:47 AM
If you'd like to see kedro-azureml plugin in action without getting hands dirty @marrrcin made a short tutorial video on the subject

https://www.youtube.com/watch?v=w_9RzYpGplY