Isaac89
11/09/2021, 10:07 AMdatajoely
11/09/2021, 10:08 AMIsaac89
11/09/2021, 10:16 AMantony.milne
11/09/2021, 10:17 AMCOHORT=value1 kedro run
- see last code example here for how you inject this into `TemplatedConfigLoader`: https://kedrozerotohero.com/programming-patterns/how-to-inject-secrets-into-your-kedro-configuration. Not quite sure how slurm would work with setting the same environment variable multiple times in parallel but might be an easy way to do this πIsaac89
11/09/2021, 10:19 AMdatajoely
11/09/2021, 1:50 PMIsaac89
11/09/2021, 2:25 PMdatajoely
11/09/2021, 2:26 PMIsaac89
11/09/2021, 2:29 PMdatajoely
11/09/2021, 2:33 PMkedro run --env="cohort_a"
should give you the right configuration if you set it up correctlyIsaac89
11/09/2021, 4:03 PMdatajoely
11/09/2021, 4:04 PMIsaac89
11/09/2021, 4:09 PMdatajoely
11/09/2021, 4:24 PMIsaac89
11/09/2021, 4:46 PMdatajoely
11/09/2021, 4:48 PMIsaac89
11/10/2021, 11:40 AMdatajoely
11/10/2021, 11:43 AMIsaac89
11/10/2021, 11:46 AMantony.milne
11/10/2021, 2:50 PM--params
is actually already available as a variable in register_config_loader
- it's the extra_params
dictionary described here: https://kedro.readthedocs.io/en/0.17.5/kedro.framework.hooks.specs.RegistrationSpecs.html#kedro.framework.hooks.specs.RegistrationSpecs.register_config_loader. So no need to delve into click to create globals_dict_params
- it's already there for you!--params
. If you want to load globals_dict
from a yaml file then that's exactly what environments are for. I understand you don't want like 1000 different environments, but in your version you still need to create all the 1000 different yaml files? π€globals_dict
through --params
(this is very similar to doing it through environment variables as in the example I linked to earlier)Isaac89
11/10/2021, 11:59 PMdatajoely
11/11/2021, 8:46 AMIsaac89
11/11/2021, 9:15 AMdatajoely
11/11/2021, 10:01 AMextra_params
https://kedro.readthedocs.io/en/latest/04_kedro_project_setup/03_session.html?highlight=extra_params#create-a-session
There is also globals_dict
kwarg in the TemplatedConfigLoader constructor
https://kedro.readthedocs.io/en/stable/_modules/kedro/config/templated_config.html#TemplatedConfigLoaderIsaac89
11/11/2021, 10:37 AMdatajoely
11/11/2021, 10:38 AMantony.milne
11/11/2021, 11:06 AMIsaac89
11/11/2021, 11:55 AMdatajoely
11/11/2021, 11:55 AMIsaac89
11/11/2021, 2:55 PMdatajoely
11/11/2021, 3:07 PM