https://kedro.org/ logo
Title
j

jaweiss2305

01/16/2022, 3:13 PM
Just curious how are other folks scheduling kedro runs? Cron jobs?
d

datajoely

01/16/2022, 4:13 PM
Hi @User this is one solution - but not really recommended for production deployments
as well as a maintained version of Kedro Airflow https://pypi.org/project/kedro-airflow/
j

jaweiss2305

01/17/2022, 2:38 PM
Thanks @User . This is a helpful guide!
d

datajoely

01/17/2022, 2:39 PM
In the community developed plugins section there are a couple of others like
kedro-kuberflow
we can't really support users who opt for that here (you can post questions in #908346260224872480 ), but they are great options
j

jaweiss2305

01/17/2022, 2:46 PM
Is there a general recommendation on production deployment? What is most common?
d

datajoely

01/17/2022, 2:46 PM
it depends on a couple of factors
most of them operational nor technical
is this something that a team will maintain? is it something that needs clever stuff like retry? will it run in the cloud?
CRON is the lowest effort but doesn't support some of the more complex bits
Airflow can do everything but introduces a bunch of complexity
j

jaweiss2305

01/17/2022, 2:52 PM
That makes sense. FWIW, I work for a start-up and am translating a bunch of jupyter notebooks into kedro pipelines then plan on deploying in a productionalized manner. It seems to me that Airflow (via Astronomer) requires the least amount of effort. However, I wanted to gauge the community sentiment especially in relation to Kubeflow. Thanks for sharing your expertise!
d

datajoely

01/17/2022, 2:54 PM
So this was published over christmas
we can't take any credit for it! but the chap who wrote it is active on twitter if you want to get in touch
Also @User one of our maintainers gave this overview talk a while back at EuroPython

https://www.youtube.com/watch?v=-FedSW2SN7A

He talks about the airflow stage at 26 mins in
j

jaweiss2305

01/17/2022, 3:18 PM
That was well explained. Thanks for sharing.