avan-sh
03/16/2022, 12:54 PMregister_config_loader
hook (https://kedro.readthedocs.io/en/latest/kedro.framework.hooks.specs.RegistrationSpecs.html#kedro.framework.hooks.specs.RegistrationSpecs.register_config_loader) also gets all the extra_params you'll be passing from command line. You can use that to add file_path to globals_dict. And also your filepath placeholder will be ${filepath}
. LMK if I'm not clear, I can help get a quick snippet later if you want.datajoely
03/16/2022, 12:57 PMvivecalindahl
03/16/2022, 1:15 PMavan-sh
03/16/2022, 1:25 PMclass ProjectHooks:
@hook_impl
def register_config_loader(conf_paths, env, extra_params) -> ConfigLoader:
globals_dict = {}
globals_dict["filepath"] = extra_params.get("filepath", "default_value")
return TemplatedConfigLoader(
conf_paths,
globals_dict=globals_dict,
)
vivecalindahl
03/16/2022, 1:25 PMlbonini
03/16/2022, 1:53 PMyaml
# catalog.yml
example_data:
type: PartitionedDataSet
dataset: pandas.ParquetDataSet
credentials: dev_s3
path: s3://bucket/path/to/folder
filename_suffix: "_part.parquet"
yaml
# credentials.yml
dev_s3:
client_kwargs:
aws_access_key_id: xxxx
aws_secret_access_key: xxx
datajoely
03/16/2022, 1:53 PMlbonini
03/16/2022, 2:07 PMgui42
03/17/2022, 2:23 AMgui42
03/17/2022, 2:23 AMdatajoely
03/17/2022, 9:17 AMSchoolmeister
03/17/2022, 9:37 AMparameters.yml
with the following contents:
yaml
folds:
timeseries1:
- start: 2021-08-24 15:00:00+00:00
end: 2021-10-22 03:05:00+00:00
- start: 2021-10-22 03:10:00+00:00
end: 2021-12-28 05:00:00+00:00
- start: 2021-12-28 05:05:00+00:00
end: 2022-01-28 12:00:00+00:00
Can I reference the second [start, end]? I've tried using something like params:folds.timeseries1.1
or params:folds.timeseries1[1]
, but that doesn't work.datajoely
03/17/2022, 9:38 AMSchoolmeister
03/17/2022, 9:40 AMlbonini
03/17/2022, 3:52 PMkedro run
datajoely
03/17/2022, 3:57 PMlbonini
03/17/2022, 3:57 PMdatajoely
03/17/2022, 4:04 PMlbonini
03/17/2022, 4:05 PMgui42
03/17/2022, 5:06 PMdatajoely
03/17/2022, 5:09 PMgui42
03/17/2022, 5:12 PMdatajoely
03/17/2022, 5:13 PMDaehyun Kim
03/17/2022, 6:54 PMdatajoely
03/17/2022, 6:55 PMDaehyun Kim
03/17/2022, 7:00 PMdatajoely
03/17/2022, 7:01 PMDaehyun Kim
03/17/2022, 7:01 PMDaehyun Kim
03/17/2022, 7:02 PMtest_model_plot.png
under the same version(run_id)datajoely
03/17/2022, 7:02 PM