@noklam yes that’s exactly what I meant. I get that it adds a significant layer of complexity, however I’m not sure that I see a better way of enabling users to construct a pipeline from a pre-defined set of building blocks without writing code. Thanks for the pointer towards the hooks, I’ll give that a look.

Include Quarto rendering in kedro pipeline and pass it inputs/outputs https://stackoverflow.com/questions/74099238/include-quarto-rendering-in-kedro-pipeline-and-pass-it-inputs-outputs

Hi, is there a way to have the configurations in settings.py configurable based on the kedro environment. For example, I want the hooks for kedro-viz and kedro-neptune disabled in production environment. Thanks

Interesting usecase - best solution off the top of my head is to use the KEDRO_ENV environment variable and just include an if statement in settings.py

I was also thinking of a using an ENV var, glad to hear that there is one already defined for the KEDRO_ENV! Thanks

The other option, which is probably preferable, is to do the logic in the hooks themselves which do have the env name as a parameter

This isn't possible if you're using 3rd party hooks

Some similar requests a few days ago

Nice one @noklam !

the KEDRO_ENV does not seem to be set by the time the session creation does the "validate_settings()"

Sorry it doesn't get set automatically (it may in the future) but you can use it to set the env pre run

So it's instead of the CLI arg

ok thanks!

Hey, I was wondering if there was an elegant solution to overwrite parameters dynamically? I am instanciating a pipeline 12 times, but they all need to run with a different parameter called date_max, e.g. "07/01/22" for the first one, and the other ones are decrementing one month, e.g. "06/01/22"... etc.. The pipelines are generated of a template dynamically and ideally I would just pass the adjusted parameter.

parameters.yml
t_-0:
  filters:
    date_max: 2022/07/01
t_-1:
  filters:
    date_max: 2022/06/01

So I want to avoid doing this, as I would need to pass 12 or more variables on each induction. Whereas they are actually all are dependent on the first one.

Hooks!

Although we've talked a lot about the dangers of dynamic pipelines in terms of complexity and maintenance overhead

Could you elaborate a little the hooks solution?

Same concept https://discord.com/channels/778216384475693066/846330075535769601/931130020762648576

Quick Q, we can create our custom starters, which is very useful - we use it to set company-wide standards. Is there a way to create custom pipeline layouts? So when we do

kedro pipeline create

it will use a custom cookiecutter layout created by us? Technically it seems possible as I see in the

kedro

repo there is a dedicated cookiecutter for pipelines

How can I run

catalog.load

in a non-IPython context? https://stackoverflow.com/questions/74127089/how-can-i-run-catalog-load-in-a-non-ipython-context

Hi, I am trying to run my kedro project in Apache Airflow. I follow the instructions and am able to launch the Airflow app when I do not attach my kedro image in docker-compose.yaml. However, when I do attach the image, by running docker-compose up, I get: Starting airflow_redis_1 ... done Starting airflow_postgres_1 ... done Starting airflow_airflow-init_1 ... done Starting airflow_airflow-webserver_1 ... done Starting airflow_airflow-triggerer_1 ... done Recreating airflow_airflow-worker_1 ... error Recreating airflow_airflow-scheduler_1 ... Recreating airflow_airflow-scheduler_1 ... error ERROR: for airflow_airflow-scheduler_1 Cannot start service airflow-scheduler: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "scheduler": executable file not found in $PATH: unknown ERROR: for airflow-worker Cannot start service airflow-worker: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "celery": executable file not found in $PATH: unknown ERROR: for airflow-scheduler Cannot start service airflow-scheduler: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "scheduler": executable file not found in $PATH: unknown ERROR: Encountered errors while bringing up the project.

I don't think any of this is kedro specific - it looks like docker/airflow configuration

yes, it does seem like that. However, I was wondering if someone has encountered this before and could help me with this. Or if kedro documentation could be improved to avoid such configuration issue?

It's not something I've seen before unfortunately

Trying to create a Kedro starter integrated with Poetry. I'm following this issue (https://github.com/kedro-org/kedro/issues/1722) to get it done. Looking at the docs in this link (https://kedro.readthedocs.io/en/stable/extend_kedro/plugins.html#extend-starter-aliases) I should edit

setup.py

to get it done. Now... (1) I'm creating a starter, so there isn't any

setup.py

yet as I have not executed Kedro new. So how exactly am I supposed to edit it if it is not yet created? Second (2), I don't want to use

setup.py

, as Poetry does not need it. So how should I go about supporting new kedro starters?

Running Kedro on Airflow