https://kedro.org/ logo
#beginners-need-help
Title
# beginners-need-help
b

Binky

08/04/2022, 3:48 AM
Hi all. I have a beginner question, on Kedro 0.18.2. I have a TemplatedConfigLoader that loads a partitioned dataset from a specific path
Copy code
raw_dataset: 
  type: PartitionedDataSet
  dataset:
    type: pandas.CSVDataSet
    load_args:
      index_col: False
  path: data/01_raw/${asset}
  filename_suffix: ".csv"
  layer: raw
and my settings.py reads
Copy code
CONFIG_LOADER_CLASS = TemplatedConfigLoader
CONFIG_LOADER_ARGS = {
    "globals_pattern": "*parameters.yml",
}
When setup this way my pipeline works as expected, pulling the asset variable from parameters.yml. However now I want to be able to override this variable from the command line e.g.
Copy code
kedro run --params=asset:1234
Any advice on how I can do that? I went on a wild goose hunt on Google and the Kedro github but still can't quite figure it out. It's pretty similar to this issue https://github.com/kedro-org/kedro/issues/1527#issuecomment-1127697871 but i couldn't get the suggested solution to work.
Is this what you are looking for?
kedro run --params:xxxx
that override entries in
catalog.yml
etc
b

Binky

08/04/2022, 12:48 PM
Yes, it is! Thank you so much, @noklam! It works.
n

noklam

08/04/2022, 1:03 PM
Nice!
b

Binky

08/05/2022, 2:05 AM
Hi @noklam, I just noticed that while the code in this repo works, context/session etc are no longer automatically loaded in Jupyter notebooks. I can live with it though, as I can just switch to the usual TemplatedConfigLoader when I need to use the notebook.
a

antony.milne

08/05/2022, 6:34 PM
Hello @Binky, this should definitely work so if it doesn't then there might be some sort of bug. How are you opening your Jupyter notebook?
If you do
kedro jupyter lab/notebook
then the variables should be available there. Or if you're on a managed Jupyter instance then you can load them through
%load_ext kedro.extras.extensions.ipython.
See https://kedro.readthedocs.io/en/stable/tools_integration/ipython.html
b

Binky

09/01/2022, 3:08 AM
Hi @antony.milne sorry for the late reply. I open the Jupyter notebook with
kedro jupyter notebook
. I usually switch between datasets when I'm running the pipeline, so for example when I want to inspect the output of one run, I will execute the
%reload_kedro
magic in my notebook so the context and catalog is reloaded. In a standard Kedro project, this always works well for me. However, when I add the abovementioned modification to the TemplatedConfigLoader, I can no longer refresh the context with the`%reload_kedro` magic.
n

noklam

09/01/2022, 7:40 AM
What happened when u run reload_kedro?
b

Binky

09/01/2022, 12:49 PM
Hi @noklam. With the standard TemplatedConfigLoader, I'm always able to reload the context every time I run reload_kedro. With the modified one I get the error shown in the second screenshot. I've attached the entire traceback as well.
n

noklam

09/01/2022, 12:52 PM
Can you share the implementation of
MyTemplatedConfigLoader
?
@Binky Sorry, just going through the thread and the implementation is actually from meπŸ˜‚
https://github.com/kedro-org/kedro/discussions/1782 Can you try this? It's almost identical, the previous implementation missed a condition to handle some edge case
b

Binky

09/02/2022, 12:55 AM
@noklam, yes, I can confirm it works now. Thanks!
n

noklam

09/02/2022, 7:18 AM
Perfect!
4 Views