Without cli or envs

As in - have two parameters with ability to override one?

Yes

Well I'm not sure you can pick a winner if they live at the same level

You could use the pure python way of having a default parameter

So in your function

def myfunc(a,b=5)

Right

What about a custom parameters loader ?

Like when you call in your node ‘parameters’

I am trying to run the Space Tutorial (starter version 0.18.3) but I keep getting

DataSetError: Save path '.../spacetut/data/06_models/regressor.pickle/2022-10-24T19.26.35.315Z/regressor.pickle' for PickleDataSet(backend=pickle, 
filepath=.../spacetut/data/06_models/regressor.pickle, load_args={}, protocol=file, save_args={}, version=Version(load=None, 
save='2022-10-24T19.26.35.315Z')) must not exist if versioning is enabled.

...even if I clean out the file it is complaining about. And it also happened on the very first run. This is with:

kedro                    0.18.3
kedro-docker             0.3.0
kedro-kubeflow           0.7.3
kedro-telemetry          0.2.1
kedro-viz                4.7.2

What would cause this?

Delete the folder it's in?

The data that is

I tried deleting up to/including 06_models, but still get the error.

And if you say versioning false in the catalog?

Then I get

[Errno 21] Is a directory: '.../spacetut/data/06_models/regressor.pickle'

So delete that directory

Enable versioning and then run it

Back to the first error...

So im out of ideas, I'd be surprised if the tutorial is fundamentally broken but I can check in the morning (it's quite late here in London!)

Not urgent, please let me know if you find an answer tomorrow and thanks for engaging!

Be aware this is with the catalog changes mentioned in https://kedro-kubeflow.readthedocs.io/en/latest/source/03_getting_started/01_quickstart.html. And for version 0.18.3 of the starter (not the 0.17.7 version mentioned in that link which I abandoned after it gave API conflicts)

I'm thinking of writing a plugin for kedro for google cloud workflow and google cloud run jobs, any reason why this won't work?

@datajoely I believe it is because the catalog.yaml in the Quickstart specifies the same file path for two different datasets:

data_science.active_modelling_pipeline.regressor:
  type: pickle.PickleDataSet
  filepath: data/06_models/regressor.pickle
  versioned: true
  layer: models
...
data_science.candidate_modelling_pipeline.regressor:
  type: pickle.PickleDataSet
  filepath: data/06_models/regressor.pickle
  versioned: true
  layer: models

That cannot work... right?

FWIW, I just changed them to

regressor.active.pickle

and

regressor.candidate.pickle

and it ran fine...

Hey guys, can I write to parameters.yml in a kedro node?

@datajoely sorry to bother, is there any way to make a node to trigger multiple other pipelines, each with their individuals nodes?

Ah nice, namespacing issue will get easier in the future

There are ways you could do this but I'd recommend you don't because it will break reproducibility

As don't typically recommend this because the combinatorial complexity introduced makes things v hard to manage/maintain/debug