Hello could anyone help me finding good example for YAML fil Kedro #beginners-need-help

Hello, could anyone help me finding good example f...

Rafał

04/26/2022, 10:08 AM

Hello, could anyone help me finding good example for YAML file being used

kedro run -c config.yml

. I see the official documentation says nothing about

--from-nodes

. I am afraid I have a case that

kedo 0.18.0

ignores my option provided in

run.from-nodes

noklam

04/26/2022, 11:03 AM

Starting a thread here.

noklam

04/26/2022, 11:04 AM

What is the

lv

structure?

Rafał

04/26/2022, 12:26 PM

`load_version' or 'load-version' ?

noklam

04/26/2022, 12:35 PM

I open an issue related to this thread with some additional proposal. https://github.com/kedro-org/kedro/issues/1485

noklam

04/26/2022, 12:39 PM

Copy code

load_version: 
    - dataset_name:version_name

Rafał

04/26/2022, 3:02 PM

Would be also graet to be consistent with the following code

Copy code

session = KedroSession.create(
            metadata, env=config["run"].get("env"), extra_params=params
        )
        session.run(
            pipeline_name=config["run"]["pipeline"],
            node_names=config["run"].get("node-names"),
            from_nodes=config["run"].get("from-nodes"),
            to_nodes=config["run"].get("to-nodes"),
            from_inputs=config["run"].get("from-inputs"),
            to_outputs=config["run"].get("to-outputs"),
        )
        session.close()

Rafał

04/26/2022, 3:03 PM

possibly use uderscores as well But assume that someone creates session using

KedroSession.create

or runs it by

run -c config.yaml

Now the code expect

Iterable

but

run -c config.yml

expects strings in the config file.

noklam

04/26/2022, 3:42 PM

Which field are you referring to?

Rafał

04/26/2022, 6:40 PM

Almost all of

session.run( **kwargs )

are expected to be

Iterable

. I just wanted to mention that providing strings like

node1,node2

is Iterable, too. But

kedro

session will try to iterate it through single characters. Thus, one should remember to split strings with

(comma).

noklam

04/26/2022, 6:41 PM

Why do you need to involve session while parsing run time arguments?

noklam

04/26/2022, 6:42 PM

Isn't that the exact same thing kedro run does?

noklam

04/26/2022, 6:43 PM

I am interested to understand the use case.

Rafał

04/26/2022, 6:47 PM

Ach, I see. You may notice in my code

extra_params=params

I actually iterate

params

from some iterable taken from

config.extra_params_list

This is my own modification of

kedro run -c config.yml

in order to run the single pipeline (given by

config.run

) multiple times 😉 with different

params

, namely taken from

extra_params_list

😉 I know it is tricky, but I didn't figure out better way to run the single pipeline with list of params.

noklam

04/26/2022, 6:51 PM

Extra_param_list is just a custom field that you created for multirun right?

noklam

04/26/2022, 6:57 PM

https://github.com/kedro-org/kedro/blob/main/kedro/framework/cli/project.py Hopefully I didn't misunderstood your use case. Kedro cli is base on Click and it will parse user input from the command line. These comma delimited configuration are then further parse into list/dict whatever they need to be, and then they are passed to session. So what you really want is using the same method how kedro cli is reading config and just iteratively creating session and fun

noklam

04/26/2022, 7:00 PM

_config_file_callback is the function that does the parsing

Rafał

04/26/2022, 8:03 PM

Thank you. I will definitely take a look into click.

Previous Next