https://kedro.org/ logo
Powered by
Title
r

Rafał

04/26/2022, 10:08 AM
Hello, could anyone help me finding good example for YAML file being used 
kedro run -c config.yml
. I see the official documentation says nothing about 
--from-nodes
. I am afraid I have a case that 
kedo 0.18.0
ignores my option provided in 
run.from-nodes
n

noklam

04/26/2022, 11:03 AM
Starting a thread here.
What is the 
lv
structure?
r

Rafał

04/26/2022, 12:26 PM
`load_version' or 'load-version' ?
n

noklam

04/26/2022, 12:35 PM
I open an issue related to this thread with some additional proposal. https://github.com/kedro-org/kedro/issues/1485
load_version: 
    - dataset_name:version_name
r

Rafał

04/26/2022, 3:02 PM
Would be also graet to be consistent with the following code 
session = KedroSession.create(
            metadata, env=config["run"].get("env"), extra_params=params
        )
        session.run(
            pipeline_name=config["run"]["pipeline"],
            node_names=config["run"].get("node-names"),
            from_nodes=config["run"].get("from-nodes"),
            to_nodes=config["run"].get("to-nodes"),
            from_inputs=config["run"].get("from-inputs"),
            to_outputs=config["run"].get("to-outputs"),
        )
        session.close()
possibly use uderscores as well But assume that someone creates session using 
KedroSession.create
or runs it by 
run -c config.yaml
Now the code expect 
Iterable
but 
run -c config.yml
expects strings in the config file.
n

noklam

04/26/2022, 3:42 PM
Which field are you referring to?
r

Rafał

04/26/2022, 6:40 PM
Almost all of 
session.run( **kwargs )
are expected to be 
Iterable
. I just wanted to mention that providing strings like 
node1,node2
is Iterable, too. But 
kedro
session will try to iterate it through single characters. Thus, one should remember to split strings with 
,
(comma).
n

noklam

04/26/2022, 6:41 PM
Why do you need to involve session while parsing run time arguments?
Isn't that the exact same thing kedro run does?
I am interested to understand the use case.
r

Rafał

04/26/2022, 6:47 PM
Ach, I see. You may notice in my code 
extra_params=params
I actually iterate 
params
from some iterable taken from 
config.extra_params_list
This is my own modification of 
kedro run -c config.yml
in order to run the single pipeline (given by 
config.run
) multiple times 😉 with different 
params
, namely taken from 
extra_params_list
😉 I know it is tricky, but I didn't figure out better way to run the single pipeline with list of params.
n

noklam

04/26/2022, 6:51 PM
Extra_param_list is just a custom field that you created for multirun right?
https://github.com/kedro-org/kedro/blob/main/kedro/framework/cli/project.py Hopefully I didn't misunderstood your use case. Kedro cli is base on Click and it will parse user input from the command line. These comma delimited configuration are then further parse into list/dict whatever they need to be, and then they are passed to session. So what you really want is using the same method how kedro cli is reading config and just iteratively creating session and fun
_config_file_callback is the function that does the parsing
r

Rafał

04/26/2022, 8:03 PM
Thank you. I will definitely take a look into click.
#beginners-need-helpJoin Discord
Powered by