Did anyone here setup airflow instance using kedro package p

Question

Did anyone here setup airflow instance using kedro package pipeline unable to configure spent good amount of time following the docs would be great if anyone walked through the same path can assist The issue is summarised here https github com kedro org kedro plugins issues 46

Flow · Answer

I have got a setup running with airflow and kedro It was a bit bumpy but it s running well now I can try to help Are you running airflow locally or using a docker compose

arum · Answer

good to hear im trying locally

Flow · Answer

` conf base` this looks like the path for looking for the config is the root folder is that really from where you are running things Should this not be ` path to airflow home conf base`

Flow · Answer

If I remember right airflow will go to $AIRFLOW HOME as the default execution folder

Flow · Answer

I d also suggest considering using the docker compose setup for airflow to make things more reproducible in case you want to move the setup somewhere else later

arum · Answer

airflow is installed at `airflow pwd Users sl airflow`

arum · Answer

and i have the conf here ` Users sl airflow conf base`

arum · Answer

however when i check for which airflow `airflow which airflow Users sl mambaforge pypy3 envs airflow env bin airflow`

arum · Answer

airflow is running from the venv

Flow · Answer

The first is just a folder right the which command actually checks where it is installed The first however looks like the airflow home directory

Flow · Answer

`ValueError Given configuration path either does not exist or is not a valid directory conf base`

Flow · Answer

This path looks like it not looking in the airflow home folder though

Flow · Answer

did you change the `settings py` to provide a different location for the config

Flow · Answer

I d still encourage you to try docker compose https airflow apache org docs apache airflow stable start docker html That way you can directly connect into the worker and see what the execution environment looks like and what files are there

arum · Answer

unfortunately nothing has been changed in settings py and is still not finding conf in the airflow home folder I will try the docker compose approach

arum · Answer

< datajoely> here is where i stand now

arum · Answer

built the docker image for my project and ran the project from the container to see if the pipeline is working using `python m epc fi` and it works

arum · Answer

then used the same image for the airflow webserver and airflow worker in docker compose yaml

arum · Answer

still my airflow UI has the kedro import error

arum · Answer

for anyone who might be walking through the same path here is how it worked

arum · Answer

1 Running airflow from docker https airflow apache org docs apache airflow stable start docker html

arum · Answer

2 Build the docker image for kedro pipeline

arum · Answer

3 Attach the image from step2 to airflow worker and airflow scheduler in docker compose yml file

arum · Answer

This solved the issue Thanks to < Flow> for taking his time