The other option, which is probably preferable, is to do the logic in the hooks themselves which do have the env name as a parameter

This isn't possible if you're using 3rd party hooks

Some similar requests a few days ago

Nice one @noklam !

the KEDRO_ENV does not seem to be set by the time the session creation does the "validate_settings()"

Sorry it doesn't get set automatically (it may in the future) but you can use it to set the env pre run

So it's instead of the CLI arg

ok thanks!

Hey, I was wondering if there was an elegant solution to overwrite parameters dynamically? I am instanciating a pipeline 12 times, but they all need to run with a different parameter called date_max, e.g. "07/01/22" for the first one, and the other ones are decrementing one month, e.g. "06/01/22"... etc.. The pipelines are generated of a template dynamically and ideally I would just pass the adjusted parameter.

parameters.yml
t_-0:
  filters:
    date_max: 2022/07/01
t_-1:
  filters:
    date_max: 2022/06/01

So I want to avoid doing this, as I would need to pass 12 or more variables on each induction. Whereas they are actually all are dependent on the first one.

Hooks!

Although we've talked a lot about the dangers of dynamic pipelines in terms of complexity and maintenance overhead

Could you elaborate a little the hooks solution?

Same concept https://discord.com/channels/778216384475693066/846330075535769601/931130020762648576

Quick Q, we can create our custom starters, which is very useful - we use it to set company-wide standards. Is there a way to create custom pipeline layouts? So when we do

kedro pipeline create

it will use a custom cookiecutter layout created by us? Technically it seems possible as I see in the

kedro

repo there is a dedicated cookiecutter for pipelines

How can I run

catalog.load

in a non-IPython context? https://stackoverflow.com/questions/74127089/how-can-i-run-catalog-load-in-a-non-ipython-context

Hi, I am trying to run my kedro project in Apache Airflow. I follow the instructions and am able to launch the Airflow app when I do not attach my kedro image in docker-compose.yaml. However, when I do attach the image, by running docker-compose up, I get: Starting airflow_redis_1 ... done Starting airflow_postgres_1 ... done Starting airflow_airflow-init_1 ... done Starting airflow_airflow-webserver_1 ... done Starting airflow_airflow-triggerer_1 ... done Recreating airflow_airflow-worker_1 ... error Recreating airflow_airflow-scheduler_1 ... Recreating airflow_airflow-scheduler_1 ... error ERROR: for airflow_airflow-scheduler_1 Cannot start service airflow-scheduler: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "scheduler": executable file not found in $PATH: unknown ERROR: for airflow-worker Cannot start service airflow-worker: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "celery": executable file not found in $PATH: unknown ERROR: for airflow-scheduler Cannot start service airflow-scheduler: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "scheduler": executable file not found in $PATH: unknown ERROR: Encountered errors while bringing up the project.

I don't think any of this is kedro specific - it looks like docker/airflow configuration

yes, it does seem like that. However, I was wondering if someone has encountered this before and could help me with this. Or if kedro documentation could be improved to avoid such configuration issue?

It's not something I've seen before unfortunately

Trying to create a Kedro starter integrated with Poetry. I'm following this issue (https://github.com/kedro-org/kedro/issues/1722) to get it done. Looking at the docs in this link (https://kedro.readthedocs.io/en/stable/extend_kedro/plugins.html#extend-starter-aliases) I should edit

setup.py

to get it done. Now... (1) I'm creating a starter, so there isn't any

setup.py

yet as I have not executed Kedro new. So how exactly am I supposed to edit it if it is not yet created? Second (2), I don't want to use

setup.py

, as Poetry does not need it. So how should I go about supporting new kedro starters?

Running Kedro on Airflow

How to show plotly chart in kedro https://stackoverflow.com/questions/74169483/how-to-show-plotly-chart-in-kedro

Quite confused about code management in a modular pipelines Kedro project. On one side, to support micro-packaging I should put all my pipeline code within that pipeline's folder. On the other side, if I have nodes/functions/classes that I want to re-use across pipelines, I'd want to place their code in the

<my_project>/src/<my_project>

level - but that will prevent me from using micro-packaging (as the docs states). So what is the recommended way to work with modular pipelines that share nodes/functions/classes?

Hello I have one question regarding using kedro to train a tf/keras CNN model: on the keras documentation( https://keras.io/api/data_loading/image/#imagedatasetfromdirectory-function ), it is written to use tf.keras.utils.image_dataset_from_directory to load the images. As I am not using 'inferred' labels (I did not manage to sort image by label in my pipeline in sub folder with a partitioned dataset) I am trying to use os.walk to generate the proper label list. However I am failing with os.walk to get the proper list of files. Would someone have an example how to perform this please?

How to generate kedro pipelines automatically (like DataEngineerOne does)? https://stackoverflow.com/questions/74178198/how-to-generate-kedro-pipelines-automatically-like-dataengineerone-does

Just curious, is there any good example/best practices on how to test the kedro pipeline code with mock data, specifically integration tests ?

@mrjpz99 this is a previous thread on the topic https://discord.com/channels/778216384475693066/931533715291648041