Binky
08/04/2022, 3:48 AMraw_dataset:
type: PartitionedDataSet
dataset:
type: pandas.CSVDataSet
load_args:
index_col: False
path: data/01_raw/${asset}
filename_suffix: ".csv"
layer: raw
and my settings.py reads
CONFIG_LOADER_CLASS = TemplatedConfigLoader
CONFIG_LOADER_ARGS = {
"globals_pattern": "*parameters.yml",
}
When setup this way my pipeline works as expected, pulling the asset variable from parameters.yml. However now I want to be able to override this variable from the command line e.g.
kedro run --params=asset:1234
Any advice on how I can do that? I went on a wild goose hunt on Google and the Kedro github but still can't quite figure it out. It's pretty similar to this issue https://github.com/kedro-org/kedro/issues/1527#issuecomment-1127697871 but i couldn't get the suggested solution to work.kedro run --params:xxxx
that override entries in catalog.yml
etcBinky
08/04/2022, 12:48 PMnoklam
08/04/2022, 1:03 PMBinky
08/05/2022, 2:05 AMantony.milne
08/05/2022, 6:34 PMkedro jupyter lab/notebook
then the variables should be available there. Or if you're on a managed Jupyter instance then you can load them through %load_ext kedro.extras.extensions.ipython.
See https://kedro.readthedocs.io/en/stable/tools_integration/ipython.htmlBinky
09/01/2022, 3:08 AMkedro jupyter notebook
. I usually switch between datasets when I'm running the pipeline, so for example when I want to inspect the output of one run, I will execute the %reload_kedro
magic in my notebook so the context and catalog is reloaded. In a standard Kedro project, this always works well for me. However, when I add the abovementioned modification to the TemplatedConfigLoader, I can no longer refresh the context with the`%reload_kedro` magic.noklam
09/01/2022, 7:40 AMBinky
09/01/2022, 12:49 PMnoklam
09/01/2022, 12:52 PMMyTemplatedConfigLoader
?Binky
09/02/2022, 12:55 AMnoklam
09/02/2022, 7:18 AM