Hi all. I have a beginner question, on Kedro 0.18....
# beginners-need-help
Hi all. I have a beginner question, on Kedro 0.18.2. I have a TemplatedConfigLoader that loads a partitioned dataset from a specific path
Copy code
  type: PartitionedDataSet
    type: pandas.CSVDataSet
      index_col: False
  path: data/01_raw/${asset}
  filename_suffix: ".csv"
  layer: raw
and my settings.py reads
Copy code
CONFIG_LOADER_CLASS = TemplatedConfigLoader
    "globals_pattern": "*parameters.yml",
When setup this way my pipeline works as expected, pulling the asset variable from parameters.yml. However now I want to be able to override this variable from the command line e.g.
Copy code
kedro run --params=asset:1234
Any advice on how I can do that? I went on a wild goose hunt on Google and the Kedro github but still can't quite figure it out. It's pretty similar to this issue https://github.com/kedro-org/kedro/issues/1527#issuecomment-1127697871 but i couldn't get the suggested solution to work.
Is this what you are looking for?
kedro run --params:xxxx
that override entries in
Yes, it is! Thank you so much, @noklam! It works.
Hi @noklam, I just noticed that while the code in this repo works, context/session etc are no longer automatically loaded in Jupyter notebooks. I can live with it though, as I can just switch to the usual TemplatedConfigLoader when I need to use the notebook.
Hello @Binky, this should definitely work so if it doesn't then there might be some sort of bug. How are you opening your Jupyter notebook?
If you do
kedro jupyter lab/notebook
then the variables should be available there. Or if you're on a managed Jupyter instance then you can load them through
%load_ext kedro.extras.extensions.ipython.
See https://kedro.readthedocs.io/en/stable/tools_integration/ipython.html
Hi @antony.milne sorry for the late reply. I open the Jupyter notebook with
kedro jupyter notebook
. I usually switch between datasets when I'm running the pipeline, so for example when I want to inspect the output of one run, I will execute the
magic in my notebook so the context and catalog is reloaded. In a standard Kedro project, this always works well for me. However, when I add the abovementioned modification to the TemplatedConfigLoader, I can no longer refresh the context with the`%reload_kedro` magic.
What happened when u run reload_kedro?
Hi @noklam. With the standard TemplatedConfigLoader, I'm always able to reload the context every time I run reload_kedro. With the modified one I get the error shown in the second screenshot. I've attached the entire traceback as well.
Can you share the implementation of
@Binky Sorry, just going through the thread and the implementation is actually from me😂
https://github.com/kedro-org/kedro/discussions/1782 Can you try this? It's almost identical, the previous implementation missed a condition to handle some edge case
@noklam, yes, I can confirm it works now. Thanks!