Hello guys! I got a question related to the kedro-airflow plugin but I'm not sure if this is the place to ask for...my company has Airflow installed in an internal server using Docker. Kedro docs says that we should install our kedro packaged project inside the airflow environment, but that means we should change the Airflow Docker Image every time we want to install a new package, module, etc. Is there a way to manage that scenario without modifying the Airflow DockerImage/Compose file??

Let me get back to you on what our view of best practice is

Yes, I've used kedro-airflow, so I already have my kedro pipeline as an Airflow DAG. The thing is when I open the Airflow UI an error 'X module not found' arises, where X could be kedro, pandas or whatever package Airflow doesn't install when it is deployed in the internal server

This isn't an area that I'm super knowledgeable - let me ask about

I'm running with the following error when trying to execute a kedro pipeline with airflow:

Yeah the * directory is causing issues I think?

nope. thats not the name of the folder

any idea where the * is coming from?

* matches any number of directories (3 in this case) between /opt/ and /conf/base

is that a docker or AirFlow thing?

I don't know... I guess it is an Airflow thing but I don't know how to deal with the stars 😦

hmm let me do some googling

that's my docker-compose file

Thank you for this

Hi @jcasanuevam the problem is that

kedro package

doesn't package your

conf/

directory. In our guide where we use Astronomer, their Dockerfile copies the entire project directoy into the container. I suspect that isn't happening in your case?

If you post your Dockerfile, I can help you amend it to make it work

That is. kedro package doesn't package my conf/ directory and that's why I copied conf/ data/ and logs/ directories as kedro docs says

Oh wow!

Putting in docs would make it much easier. As a reminder to modify the value of that variable if the project path is not the same as where the dag file is located (It shouldn't be the same as a best practice)

Understood - I'll raise a documentation PR

super!

Would it make sense to put it under step 3? https://kedro.readthedocs.io/en/stable/10_deployment/11_airflow_astronomer.html#step-3-convert-the-kedro-pipeline-into-an-airflow-dag-with-kedro-airflow

I'm not using Astronomer but I guess we would face the same issue, so yes, I think this is a good place. I'd also modify the kedro-airflow github repository

Understood - thank you!

Perfect! I think that's enought. Thank you!

Thank you!

One question. If my project uses the kedro-mlflow plugin, once I packaged my project, does it package the kedro-mlflow plugin? I'm geting the following error:

Yes we only package the kedro project

mmm so maybe a better solution than package the project is to use the kedro-docker plugin in order to get the Dockerfile and run it inside the Airflow environment in the desired project root?

I think that sounds simpler to me

I will give it a try and tell you, thanks!

@User if you have a docker container, you can also run it in Airflow with the DockerOperator

But with DockerOperator I lose the option of having each node of my kedro pipeline defined as a task in airflow (kedro-airflow plugin allows us to do that, 'translating' our kedro pipeline as an airflow DAG). My DAG using DockerOperator would be summarised to a single task built with DockerOperator and i like the fact of having my kedro pipeline visually represented in airflow. Is there a way to have it using DockerOperator?

Yea, in this setup, you need to give the user inside airflow permission to access the

docker_url

so it can launch the operator. Try running

sudo chmod 777 /var/run/docker.sock

on your host machine and see if it helps?

In production, it will be a docker cluster somewhere with a proper url so this permission problem will not exist.

Regarding the shape of the DAG: you can certainly retain the shape of the DAG with DockerOperator. In the

command

section of the operator, simply change

kedro run

to

kedro run --node=...

. Essentially change

KedroOperator

with

DockerOperator

and retain the same DAG by parameterising the command.

> I'm not using Astronomer but I guess we would face the same issue, so yes, I think this is a good place. I'd also modify the kedro-airflow github repository With Astronomer CLI you won't face the same problem because their Dockerfile gobbles up the entire project directory, so

conf/

will be included. An example on how to do that by yourself would be: * Package the Kedro project as a wheel and install it in the dockerfile with

RUN pip install <the wheel>

(this will install all of the project dependencies into the container, including `kedro-mlflow`in your case) * Copy the whole project directory into the current working directory in the container with:

COPY . /home/kedro-project

* Set the working directory in the container to

WORKDIR /home/kedro-project

* Now

KedroOperator

or

kedro run

should work because the `conf/`is present at the current working directory

Let me see if I have time today. I will cook up a sample repository for you to demonstrate this.

Sorry I caused more confusion the other day: I was suggesting `DockerOperator`as another option, not that you really need it to make things work.

@User hey I got this one up as a demo for you: https://github.com/limdauto/demo-kedro-airflow-docker-compose. The idea is: * Install the wheel file into your container with this https://github.com/limdauto/demo-kedro-airflow-docker-compose/blob/main/Dockerfile. It should install all dependencies of your project, including kedro-mlflow etc. * Mount the conf and data dir: https://github.com/limdauto/demo-kedro-airflow-docker-compose/blob/main/docker-compose.yaml#L64-L65 * Tell docker compose to build the image: https://github.com/limdauto/demo-kedro-airflow-docker-compose/blob/main/docker-compose.yaml#L47-L48 (or you can do what you did and build an image yourself and push it somewhere) The rest is normal airflow and kedro: create the dags with kedro airflow create, etc.

You shouldn't need more than this to run Kedro on Airflow with docker-compose

lmk if it helps

With that solution we need to do it everytime we want to add a new kedro project to our airflow env and we could have problems with python dependencies. I think I prefer having a docker container of my kedro project and 'call it' from airflow in order to execute it. Now I'm trying running sudo chmod 777 /var/run/docker.sock but I don't find the docker.sock file 😦

which environment is your docker host machine? Is the docker cluster running on that machine?

I have everything running in my local machine

and you are running on your local machine as root?

also what os is your local machine?

windows

aha! sorry I assume you use linux because you use unix socket

1s

is this docker running on WSL?

and could you do a

docker network ls

and paste the result here please?

since this is windows, instead of running the chmod command, another approach is to mount this under `x-airflow-common: volumes`:

# note the 2 // here for windows
- //var/run/docker.sock:/var/run/docker.sock

If that's still not working, try adding

network_mode: airflow-docker_default

in

x-airflow-common

Not working 😦 This is my docker-compose file:

What's error? Could you remove

docker-socke-proxy

as well? Also remove

network_mode

and

docker_url

in the

DockerOperator

constructor. They are overriding values from docker compose.

'

I'm still facing the same error:

Sorry you are struggling with this 😦 This all seems like a problem with docker and WSL... Let me try this out tonight on my windows machine when I get home

Hi @User ! Did you try it out?

@User unfortunately he's out this week. Do you have any updates? I can do my best to help

Hi @User , sorry I've been on holidays. Today I'll give it a try again

what error are you seeing?

That's the problem, I don't see any hahaha. I'm a newbie with Docker but I think the issue is with the user/userid and group/groupid and their permissions

ah it's a bit outside of my knowledge too

https://discord.com/channels/778216384475693066/870590055754899536/872778121110249495 Hi @User I was able to run my kedro project Dockerfile with airflow using the DockerOperator, but what you say in that message I think is not correct. If we just change the 'command' variable across the different DockerOperator it won't work because each DockerOperator creates a new container of your Docker Image so once the first task is finished the container of the next task doesn't have the information of the previous task, so the chain is broken. We just can run all of our kedro project in one DockerOperator task, loosing the tree diagram of airflow.

At this point you've reached my Airflow knowledge too!

Hi guys! I think I'll be back with the KedroOperator and kedro-airflow plugin solution, but I have a question about the wheel file. Why doing it in the way @User says I'll have all dependencies of my project and doing it installing the wheel file using pip command inside my airflow container and copying there the /conf, /data and /logs directories I won't have all dependencies? For me seems to be the same process but one using a Dockerfile of your project and other installing the wheel file directly in the airflow container and copying the desired directories

Hello @User regarding the first question: > it won't work because each DockerOperator creates a new container of your Docker Image so once the first task is finished the container of the next task doesn't have the information of the previous task, so the chain is broken Could you elaborate what do you mean by "the chain is broken"? Which information do you want to retain between tasks? If it's data, since airflow is a distributed scheduler, you will need to persist your data between tasks to a shared location.

Sorry I lost a bit of context. I will need to re-read this thread again but you are right in theory: installing the wheel file should install all of the project dependencies.

Picture from kedro docs. My question to @User is: Does the resulting package also contain the dependencies of the project? (for example, kedro-mlflow plugin if I've got it in my prooject)

Yes - you need to persist at every point in an orchestrator

Thanks @User , could you bring us some light about above question?

let me check with the team

Another issue (this is an old one that seemed to be resolved, but I think not). Regarding the kedro-airflow plugin. Even if we have th same directory for our dag and kedro project we're still having the same issue:

hmm I'm not sure on the second one

Thanks! I manage dependencies with conda, not pip, so requirements.txt file I don't know if fit me

The airflow plugin is managed by the Kedro team