jcasanuevam
07/30/2021, 8:54 AMdatajoely
07/30/2021, 11:18 AMkedro-docker
which helps you dockerise your pipeline, but in my opinion is mostly useful for people unfamiliar with docker and need help getting started.
We also have kedro-airflow
which helps convert your kedro pipeline into an Airflow DAG. This was co-developed by the folks at Astronomer.
Have you used either of these? The second one feels more useful for your purposesjcasanuevam
07/30/2021, 12:32 PMdatajoely
07/30/2021, 12:34 PMjcasanuevam
08/02/2021, 11:06 AMdatajoely
08/02/2021, 11:07 AMjcasanuevam
08/02/2021, 11:08 AMdatajoely
08/02/2021, 11:08 AMjcasanuevam
08/02/2021, 11:10 AMdatajoely
08/02/2021, 11:11 AMjcasanuevam
08/02/2021, 11:18 AMdatajoely
08/02/2021, 11:18 AMjcasanuevam
08/02/2021, 12:27 PMdatajoely
08/02/2021, 12:51 PMuser
08/02/2021, 1:07 PMkedro package
doesn't package your conf/
directory. In our guide where we use Astronomer, their Dockerfile copies the entire project directoy into the container. I suspect that isn't happening in your case?user
08/02/2021, 1:08 PMjcasanuevam
08/03/2021, 6:10 AMdatajoely
08/03/2021, 9:47 AMjcasanuevam
08/03/2021, 9:52 AMdatajoely
08/03/2021, 9:53 AMjcasanuevam
08/03/2021, 9:55 AMdatajoely
08/03/2021, 9:56 AMjcasanuevam
08/03/2021, 10:08 AMdatajoely
08/03/2021, 10:13 AMjcasanuevam
08/03/2021, 10:48 AMdatajoely
08/03/2021, 10:48 AMjcasanuevam
08/03/2021, 11:41 AMdatajoely
08/03/2021, 11:42 AMjcasanuevam
08/03/2021, 11:45 AMdatajoely
08/03/2021, 11:46 AMjcasanuevam
08/03/2021, 11:47 AMuser
08/03/2021, 1:41 PMjcasanuevam
08/03/2021, 5:29 PMuser
08/05/2021, 9:47 AMdocker_url
so it can launch the operator. Try running sudo chmod 777 /var/run/docker.sock
on your host machine and see if it helps?user
08/05/2021, 9:47 AMuser
08/05/2021, 9:48 AMcommand
section of the operator, simply change kedro run
to kedro run --node=...
. Essentially change KedroOperator
with DockerOperator
and retain the same DAG by parameterising the command.user
08/05/2021, 9:53 AMconf/
will be included. An example on how to do that by yourself would be:
* Package the Kedro project as a wheel and install it in the dockerfile with RUN pip install <the wheel>
(this will install all of the project dependencies into the container, including `kedro-mlflow`in your case)
* Copy the whole project directory into the current working directory in the container with: COPY . /home/kedro-project
* Set the working directory in the container to WORKDIR /home/kedro-project
* Now KedroOperator
or kedro run
should work because the `conf/`is present at the current working directoryuser
08/05/2021, 9:54 AMuser
08/05/2021, 9:54 AMuser
08/05/2021, 11:17 AMuser
08/05/2021, 11:18 AMuser
08/05/2021, 11:18 AMjcasanuevam
08/05/2021, 12:12 PMuser
08/05/2021, 12:24 PMjcasanuevam
08/05/2021, 12:25 PMuser
08/05/2021, 12:26 PMuser
08/05/2021, 12:26 PMjcasanuevam
08/05/2021, 12:27 PMuser
08/05/2021, 12:27 PMuser
08/05/2021, 12:27 PMuser
08/05/2021, 12:27 PMuser
08/05/2021, 12:29 PMdocker network ls
and paste the result here please?user
08/05/2021, 12:32 PM# note the 2 // here for windows
- //var/run/docker.sock:/var/run/docker.sock
If that's still not working, try adding network_mode: airflow-docker_default
in x-airflow-common
jcasanuevam
08/05/2021, 4:41 PMuser
08/05/2021, 5:33 PMdocker-socke-proxy
as well? Also remove network_mode
and docker_url
in the DockerOperator
constructor. They are overriding values from docker compose.user
08/05/2021, 5:35 PMjcasanuevam
08/06/2021, 4:44 AMuser
08/09/2021, 1:52 PMjcasanuevam
08/17/2021, 8:25 AMdatajoely
08/17/2021, 9:00 AMjcasanuevam
08/23/2021, 5:57 AMdatajoely
08/23/2021, 12:22 PMjcasanuevam
08/23/2021, 12:24 PMdatajoely
08/23/2021, 12:25 PMjcasanuevam
08/25/2021, 12:20 PMdatajoely
08/25/2021, 1:12 PMjcasanuevam
08/26/2021, 6:02 AMuser
08/26/2021, 8:35 AMuser
08/26/2021, 8:37 AMjcasanuevam
08/26/2021, 8:43 AMdatajoely
08/26/2021, 8:53 AMjcasanuevam
08/26/2021, 10:37 AMdatajoely
08/26/2021, 10:38 AMjcasanuevam
08/26/2021, 11:59 AMdatajoely
08/26/2021, 12:00 PMjcasanuevam
08/26/2021, 12:04 PMdatajoely
08/27/2021, 9:36 AM