Just curious how are other folks scheduling kedro runs Cron

Question

Just curious how are other folks scheduling kedro runs Cron jobs

datajoely · Answer

Hi < jaweiss2305> this is one solution but not really recommended for production deployments

datajoely · Answer

We have a guide documented here https kedro readthedocs io en stable 10 deployment 01 deployment guide html

datajoely · Answer

as well as a maintained version of Kedro Airflow https pypi org project kedro airflow

jaweiss2305 · Answer

Thanks < datajoely> This is a helpful guide

datajoely · Answer

In the community developed plugins section there are a couple of others like `kedro kuberflow`

datajoely · Answer

https kedro readthedocs io en stable 07 extend kedro 04 plugins html community developed plugins

datajoely · Answer

we can t really support users who opt for that here you can post questions in < 908346260224872480> but they are great options

jaweiss2305 · Answer

Is there a general recommendation on production deployment What is most common

datajoely · Answer

it depends on a couple of factors

datajoely · Answer

most of them operational nor technical

datajoely · Answer

is this something that a team will maintain is it something that needs clever stuff like retry will it run in the cloud

datajoely · Answer

CRON is the lowest effort but doesn t support some of the more complex bits

datajoely · Answer

Airflow can do everything but introduces a bunch of complexity

jaweiss2305 · Answer

That makes sense FWIW I work for a start up and am translating a bunch of jupyter notebooks into kedro pipelines then plan on deploying in a productionalized manner It seems to me that Airflow via Astronomer requires the least amount of effort However I wanted to gauge the community sentiment especially in relation to Kubeflow Thanks for sharing your expertise

datajoely · Answer

So this was published over christmas

datajoely · Answer

https medium com google cloud migrate kedro pipeline on vertex ai fa3f2c6f7aad

datajoely · Answer

we can t take any credit for it but the chap who wrote it is active on twitter if you want to get in touch

datajoely · Answer

Also < 456226577798135808> one of our maintainers gave this overview talk a while back at EuroPython

datajoely · Answer

https www youtube com watch v= FedSW2SN7A

datajoely · Answer

He talks about the airflow stage at 26 mins in

jaweiss2305 · Answer

That was well explained Thanks for sharing