datajoely
09/20/2021, 6:34 PMdatajoely
09/20/2021, 6:34 PMdatajoely
09/20/2021, 6:34 PMWaldrill
09/20/2021, 7:18 PMdatajoely
09/20/2021, 7:24 PMdatajoely
09/20/2021, 7:24 PMdatajoely
09/20/2021, 7:25 PMdatajoely
09/20/2021, 7:25 PMdatajoely
09/20/2021, 7:26 PMWaldrill
09/20/2021, 7:40 PMconf/base/parameters/<pipeline_name>.yml
to it's catalog counterpart is something Kedro thinks about?
like conf/base/catalog/<pipeline_name>.yml
... or it doesn't make sense?Waldrill
09/20/2021, 8:06 PMconf/<env>/catalog/<pipeline_name>.yml
saying it will not be packed with kedro pipeline package
command.
Thanks.datajoely
09/21/2021, 3:20 PMkedro catalog generate
to generate the catalog structure after pullWaldrill
09/21/2021, 10:28 PMapp1
, app2
) and tried to set two parameter files with following the template conf/<env>/parameters/<pipeline_name>.yml
.
But it seems that upon running the pipeline kedro run --pipeline app1
, kedro reads everything in conf/<env>/parameters/
folder, and not only parameters/app1.yml
.. this results in a conflict as the parameters/app2.yml
has the same entries.
Same error is raised for catalogs.
So I guess it is not designed to use like this ... Could this be requested as a feature or it is fundamentally wrong.datajoely
09/22/2021, 7:45 AMWaldrill
09/22/2021, 11:09 AMValueError: Duplicate keys found in conf/local/parameters/app1.yml and: conf/local/parameters/app2.yml ...
datajoely
09/22/2021, 11:23 AMWaldrill
09/22/2021, 11:23 AMconf/<env>/parameters/<pipeline_name>.yml
or load as it is doing now parameters*/**
reading all <pipeline_name>.yml
files at once.datajoely
09/22/2021, 11:23 AMdatajoely
09/22/2021, 11:33 AMnamespacing
https://kedro.readthedocs.io/en/stable/06_nodes_and_pipelines/03_modular_pipelines.html#how-to-use-a-modular-pipeline-with-different-parametersdatajoely
09/22/2021, 11:33 AMWaldrill
09/22/2021, 11:36 AMdatajoely
09/22/2021, 11:36 AMdatajoely
09/22/2021, 11:36 AMdatajoely
09/22/2021, 11:36 AMWaldrill
09/22/2021, 11:44 AMapp1
prefix ... What could be done is slicing the pipeline and adding the namespace only in specific nodes ... this will keep parts with the prefix and others not ... but if I ever needed to change a "shared" parameter it will be a headache ... and conflicts will raise again.datajoely
09/22/2021, 11:45 AMWaldrill
09/22/2021, 11:49 AMdatajoely
09/22/2021, 11:51 AMdatajoely
09/22/2021, 11:53 AMWaldrill
09/22/2021, 12:04 PMWaldrill
09/22/2021, 12:04 PM