Hopefully this suits your use case nicely

Is it common to manage environments for different projects that use the same pipeline? Even when using an orchestrator like Kubeflow ? Or is it a better alternative to create a pipeline class and then register different pipelines that would inherit it and change the datasets, keeping the environments in the canonical form.

That’s a good question- I think we typically need a mix depending on the variability between projects

If your different use cases are very different then environments have very little use

If they are mostly the same with a couple of differences then environments are your way to go

The other construct that may be applicable here is modular pipelines a newer feature of kedro, but one that we’re working on making it better and better

Expanding the modular_pipeline configuration

conf/base/parameters/<pipeline_name>.yml

to it's catalog counterpart is something Kedro thinks about? like

conf/base/catalog/<pipeline_name>.yml

... or it doesn't make sense?

It seems like it does recognize this ... as it has a note with

conf/<env>/catalog/<pipeline_name>.yml

saying it will not be packed with

kedro pipeline package

command. Thanks.

Yeah but you can use

kedro catalog generate

to generate the catalog structure after pull

Hello once again, I've tried to take advantage of the modular_pipeline configuration files but probably used it wrongly. In order to apply this to the problem I've described, I registered my "generic pipeline" twice with different names (

app1

,

app2

) and tried to set two parameter files with following the template

conf/<env>/parameters/<pipeline_name>.yml

. But it seems that upon running the pipeline

kedro run --pipeline app1

, kedro reads everything in

conf/<env>/parameters/

folder, and not only

parameters/app1.yml

.. this results in a conflict as the

parameters/app2.yml

has the same entries. Same error is raised for catalogs. So I guess it is not designed to use like this ... Could this be requested as a feature or it is fundamentally wrong.

What error are you getting?

As it is the same pipeline, registered twice, it has the same entries in configuration files, the error is:

ValueError: Duplicate keys found in conf/local/parameters/app1.yml and: conf/local/parameters/app2.yml ...

Ah got you

The question is if rather modular pipelines should read configurations exactly following the template said in the documentation

conf/<env>/parameters/<pipeline_name>.yml

or load as it is doing now

parameters*/**

reading all

<pipeline_name>.yml

files at once.

Let me think about this

Ah I think what you are looking for is called

namespacing

https://kedro.readthedocs.io/en/stable/06_nodes_and_pipelines/03_modular_pipelines.html#how-to-use-a-modular-pipeline-with-different-parameters

does this help?

Yep, I've tried it, it works ...but the drawback was that the parameters will have to be app1.paramName .. and that will cause several duplications as I can't have a single default for the parameters in base env for example. But maybe it can't be avoided

for the shared parameters you can have them in the base environment

but the deltas can be namespaced

I think

Once I've added a namespace in the pipeline everything gets the

app1

prefix ... What could be done is slicing the pipeline and adding the namespace only in specific nodes ... this will keep parts with the prefix and others not ... but if I ever needed to change a "shared" parameter it will be a headache ... and conflicts will raise again.

As far as I understand - this is currently the recommended best practice, I'll check with the team here to make sure I'm not missing anything

It's Ok .. I appreciate it ... I'm almost convinced that I'll have to duplicate all default parameters adding the prefixes already ... 😩 ... at least this can be automated. It may not sound bad as I'm presenting a situation with 2 pipelines but we will have more than 20.

Yes it's a very good thing to think about up front

in general configuration at scale is something I'm passionate about simplifying . we recently finished a big research piece that focused on how we simplify the catalog (first priority) https://github.com/quantumblacklabs/kedro/issues/891, but the parameters will be the second priority. We have some changes to this namespacing thing coming in 0.18.0 (expected end of year) so I'll let the team mention them when they catch up with this thread

Just to be fair .. @User's first solution does solve the problem I set. The team here just found that it could be dangerous in the future because it extrapolates the environment beyond it's original usage. But I have nothing concrete against it.

Well modular pipelines and configuration scalable pattern for managing large projects, but you should also be wary of premature opimisation