https://kedro.org/ logo
Join Discord
Powered by
Just curious how are other folks scheduling kedro ...
# beginners-need-help
j
Just curious how are other folks scheduling kedro runs? Cron jobs?
d
Hi @User this is one solution - but not really recommended for production deployments
as well as a maintained version of Kedro Airflow https://pypi.org/project/kedro-airflow/
j
Thanks @User . This is a helpful guide!
d
In the community developed plugins section there are a couple of others like 
kedro-kuberflow
we can't really support users who opt for that here (you can post questions in #908346260224872480 ), but they are great options
j
Is there a general recommendation on production deployment? What is most common?
d
it depends on a couple of factors
most of them operational nor technical
is this something that a team will maintain? is it something that needs clever stuff like retry? will it run in the cloud?
CRON is the lowest effort but doesn't support some of the more complex bits
Airflow can do everything but introduces a bunch of complexity
j
That makes sense. FWIW, I work for a start-up and am translating a bunch of jupyter notebooks into kedro pipelines then plan on deploying in a productionalized manner. It seems to me that Airflow (via Astronomer) requires the least amount of effort. However, I wanted to gauge the community sentiment especially in relation to Kubeflow. Thanks for sharing your expertise!
d
So this was published over christmas
we can't take any credit for it! but the chap who wrote it is active on twitter if you want to get in touch
Also @User one of our maintainers gave this overview talk a while back at EuroPython

https://www.youtube.com/watch?v=-FedSW2SN7A▾

He talks about the airflow stage at 26 mins in
j
That was well explained. Thanks for sharing.
PreviousNext
Powered by