Hello guys !
I finally got it to work. The problem was on my end... In fact, as I was also tracking the git hash with MLflow, I had to add the git folder in the Docker container which of course I didn't do ...
Anyway, execution works now !
Just had one question :
When I run the pipeline locally without Airflow, I have the same run ID for each of versioned datasets. However, when I run it with Airflow, it creates different run IDs which makes it difficult to track and reproduce the outputs. Can you please help me get the same run ID with Airflow ?
You can reproduce this behaviour with this repo :
Here are the commands I execute for Airflow run :