https://kedro.org/ logo
Join Discord
Powered by
Did anyone here setup airflow instance using kedro...
# advanced-need-help
a
Did anyone here setup airflow instance using kedro-package pipeline? unable to configure, spent good amount of time following the docs, would be great if anyone walked through the same path can assist. The issue is summarised here https://github.com/kedro-org/kedro-plugins/issues/46
f
I have got a setup running with airflow and kedro. It was a bit bumpy but it's running well now. I can try to help. Are you running airflow locally or using a docker-compose?
a
good to hear, im trying locally
f
/conf/base
this looks like the path for looking for the config is the root folder is that really from where you are running things? Should this not be 
/path/to/airflow_home/conf/base
?
If I remember right airflow will go to $AIRFLOW_HOME as the default execution folder
I'd also suggest considering using the docker-compose setup for airflow to make things more reproducible in case you want to move the setup somewhere else later
a
airflow is installed at
Copy code
airflow % pwd
/Users/sl/airflow
and i have the conf/ here 
/Users/sl/airflow/conf/base
however when i check for which airflow,
Copy code
airflow % which airflow
/Users/sl/mambaforge-pypy3/envs/airflow_env/bin/airflow
airflow is running from the venv
f
The first is just a folder right? the which command actually checks where it is installed. The first however looks like the airflow home directory
ValueError: Given configuration path either does not exist or is not a valid directory: /conf/base
This path looks like it not looking in the airflow home folder though
did you change the 
settings.py
to provide a different location for the config?
I'd still encourage you to try docker-compose: https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html That way you can directly connect into the worker and see what the execution environment looks like and what files are there.
a
unfortunately, nothing has been changed in settings.py and is still not finding conf/ in the airflow home folder. I will try the docker-compose approach.
@datajoely here is where i stand now
built the docker image for my project and ran the project from the container to see if the pipeline is working using 
python -m epc_fi
and it works
then used the same image for the airflow-webserver and airflow-worker in docker-compose.yaml
still my airflow UI has the kedro import error
for anyone who might be walking through the same path, here is how it worked
2) Build the docker image for kedro pipeline
3) Attach the image from step2 to airflow-worker and airflow-scheduler in docker-compose.yml file
This solved the issue. Thanks to @Flow for taking his time
5 Views
PreviousNext
Powered by