So we have two out of the box plugins which may help We have

kedro-docker

which helps you dockerise your pipeline, but in my opinion is mostly useful for people unfamiliar with docker and need help getting started. We also have

kedro-airflow

which helps convert your kedro pipeline into an Airflow DAG. This was co-developed by the folks at Astronomer. Have you used either of these? The second one feels more useful for your purposes

So I know I have to install my kedro project package inside the airflow venv. The thing is that for doing this I need to modify the airflow docker image or docker compose file everytiem I want to install a new kedro project package in my airflow environment

In truth this feels like an airflow problem / nature of python problem rather than a Kedro one. I'm still waiting on some answers though

It's not a particularly long post but this SO thread my be useful https://stackoverflow.com/questions/56891035/how-to-manage-python-dependencies-in-airflow

It seems a problem with the conf/base directory. Any tips?

Is that the actual name of the folder?

because I think we use the * symbol as a glob wildcard?

The stars will cause issues with Kedro

Is there anywhere you specify that?

seeing you docker-compose/dockerfile may be helpful too

inside projects folder I've the following structure: project_name >> conf, data, logs folder related to kedro project

and I've installed the kedro project inside all of the containers of airflow

always in the root project_name

Airflow recognize my DAG and all of the python packages needed for my pipeline

but when I execute the workflow I get the error I've shared before

let me check wit the team

If you post your Dockerfile, I can help you amend it to make it work

I dont have any Dockerfile related to my kedro project. I've just accessed to the airflow containers and installed the kedro wheel file in a root where conf/, data/ and logs/ directories live.

PROBLEM SOLVED. The thing was the project_path variable of the DAG file generated by kedro using the kedro-airflow plugin. It was set by default as project_path = Path().cwd, and the project folder is not the same as the dag file, so changing this to the correct project path makes everything work 🙂

Well done @User 🥳

So in an ideal world - would it be something that we could make configurable in kedro-airflow?

or would it be something that would just be worth putting in docs?

If you have a second - it would be great if you could look at this PR to the readme? https://github.com/quantumblacklabs/kedro-airflow/pull/83/files?short_path=b335630#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5 Would this have helped if it was there a few days ago! 🤦

this is something we're thinking about

This may be helpful https://stackoverflow.com/questions/26059111/build-a-wheel-egg-and-all-dependencies-for-a-python-project

Hi guys. Here again (sorry😅 ). This is my DockerOperator in the airflow dag:

I'm getting the following error in airflow:

any tips about what's going on?

In production, it will be a docker cluster somewhere with a proper url so this permission problem will not exist.

Regarding the shape of the DAG: you can certainly retain the shape of the DAG with DockerOperator. In the

command

section of the operator, simply change

kedro run

to

kedro run --node=...

. Essentially change

KedroOperator

with

DockerOperator

and retain the same DAG by parameterising the command.

> I'm not using Astronomer but I guess we would face the same issue, so yes, I think this is a good place. I'd also modify the kedro-airflow github repository With Astronomer CLI you won't face the same problem because their Dockerfile gobbles up the entire project directory, so

conf/

will be included. An example on how to do that by yourself would be: * Package the Kedro project as a wheel and install it in the dockerfile with

RUN pip install <the wheel>

(this will install all of the project dependencies into the container, including `kedro-mlflow`in your case) * Copy the whole project directory into the current working directory in the container with:

COPY . /home/kedro-project

* Set the working directory in the container to

WORKDIR /home/kedro-project

* Now

KedroOperator

or

kedro run

should work because the `conf/`is present at the current working directory

Let me see if I have time today. I will cook up a sample repository for you to demonstrate this.

Sorry I caused more confusion the other day: I was suggesting `DockerOperator`as another option, not that you really need it to make things work.

@User hey I got this one up as a demo for you: https://github.com/limdauto/demo-kedro-airflow-docker-compose. The idea is: * Install the wheel file into your container with this https://github.com/limdauto/demo-kedro-airflow-docker-compose/blob/main/Dockerfile. It should install all dependencies of your project, including kedro-mlflow etc. * Mount the conf and data dir: https://github.com/limdauto/demo-kedro-airflow-docker-compose/blob/main/docker-compose.yaml#L64-L65 * Tell docker compose to build the image: https://github.com/limdauto/demo-kedro-airflow-docker-compose/blob/main/docker-compose.yaml#L47-L48 (or you can do what you did and build an image yourself and push it somewhere) The rest is normal airflow and kedro: create the dags with kedro airflow create, etc.

You shouldn't need more than this to run Kedro on Airflow with docker-compose

lmk if it helps

BTW thank you very much for your support and your demo 🙂

also what os is your local machine?

1s

is this docker running on WSL?

and could you do a

docker network ls

and paste the result here please?

since this is windows, instead of running the chmod command, another approach is to mount this under `x-airflow-common: volumes`:

# note the 2 // here for windows
- //var/run/docker.sock:/var/run/docker.sock

If that's still not working, try adding

network_mode: airflow-docker_default

in

x-airflow-common

and my DAG:

'

I think the problem is in my kedro project Dockerfile, where I have to add the airflow user I guess:

UPDATE: I've set this in Docker Desktop

and set Ubunto as the default WSL distro

via PS terminal I've accessed the wsl -d Ubuntu, and /var/run/docker.sock exist

so I run chmod 777 /var/run/docker.sock

ok with that

but still getting the same error 😦

and I have changed the kedro project Dockerfile also:

Still facing the same issue 😦 I bet my problem is with my kedro project DockerFile:

all I can suggest is digging around the logs for clues

I'm trying to figure out if it is possible for tasks to share the same container in order to share state between tasks

Sorry I lost a bit of context. I will need to re-read this thread again but you are right in theory: installing the wheel file should install all of the project dependencies.

All data stored as MemoryDataset in my kedro project would be lost, forcing me to persist every dataset, artifact, etc somewhere and I think that's not a good solution/best practice

we are working on a way to deploy things as a modular pipelines which take advantage of MemoryDataSet

but it's not released yet!

but this is the answer to the dependencies one

Maybe It'd be good to know how does the variable 'project_path' work inisde the dag.py, because there is something strange in that behaviour

Hi guys, one important thing to know that I have already discovered: using KedroOperator MemoryDataset are not shared between task, so we need to persist every data of our project in a shared folder (volume) with Airflow as using the DockerOperator. I think that is an important thing to remark in kedro-airflow and kedro-docker docs.

I guess you are in touch with the kedro-airflow plugin developers, so I think in a future release they could fix that problem using the XComs functionality of Airflow, telling Airflow that every data stored as a MemoryDataset in kedro should be transferred between task using XComs.

we can update the docs - but as I said a bit earlier, we will revisit deployment in general early 2022 once our current work on modular pipeliens enables this

so it's front of mind - just need to sequence things right