Thank you @noklam

Is there a recommended way to reuse nodes/utility functions between different pipelines? micropackaging feels a little overkill for just 3 or so functions

also 2nd question: is there a way to directly call a function that defines a node from a jupyter lab session?

lol for the 2nd question I just did

from {{project_name}}.pipelines.{{pipeline_name}}.nodes import *

and it seemed to work well

What stop you from reusing a node between different pipelines?

namespaces?

Do u want to reuse the function? Or the node

function

Then it makes sense to just factor it out from the pipeline

It could be just a separate python module

Same for utils

Project.common.xxx

Then in your pipeline or notebook u just import from the module instead

is the upper level project accessible from the nodes.py file?

It's just python code, nothing kedro specific

The only specific thing is the namespace structure, but it's not relevant to this

very cool

didn't know that was a python thing

thought you could only see the files in the local folder, unless subdirectories had

__init__.py

This is still valid, by default when you create a new kedro project, let's say the project is called

demo

, then you will have

demo/__init__.py

,

demo/pipelines/__init__.py

, this is why you can import from there. If you want to factor out some code into a separate module. You will create a new folder and file like

demo/common/__init__.py

etc.

Hi all. I have a beginner question, on Kedro 0.18.2. I have a TemplatedConfigLoader that loads a partitioned dataset from a specific path

raw_dataset: 
  type: PartitionedDataSet
  dataset:
    type: pandas.CSVDataSet
    load_args:
      index_col: False
  path: data/01_raw/${asset}
  filename_suffix: ".csv"
  layer: raw

and my settings.py reads

CONFIG_LOADER_CLASS = TemplatedConfigLoader
CONFIG_LOADER_ARGS = {
    "globals_pattern": "*parameters.yml",
}

When setup this way my pipeline works as expected, pulling the asset variable from parameters.yml. However now I want to be able to override this variable from the command line e.g.

kedro run --params=asset:1234

Any advice on how I can do that? I went on a wild goose hunt on Google and the Kedro github but still can't quite figure it out. It's pretty similar to this issue https://github.com/kedro-org/kedro/issues/1527#issuecomment-1127697871 but i couldn't get the suggested solution to work.

You need to implement the

MyTemplatedConfigLoader

example half way down the page. The CLI arguments will be in

extra_params

and you need to ensure that these are passed to

globals_dict

HI @datajoely, I did see and attempt to implement that example, but am still stuck. I access

extra_params

by loading the context. I loaded it this way:

with KedroSession.create("elevator-eventmasker") as session:
    context = session.load_context()

context.params

gives the variables in parameters.yml as expected. However

context._extra_params

just gives

None

rather than the CLI arguments.

They're an argument to the create method

Runtime Config to override Catalog

Hey guys, is there any way to show all features that could be used on Data Catalog, like that autocompletion from docstrings and such when you like, type a method and shows all things you can define within, but for data catalog?

message has been deleted

like these ones...

hey @datajoely how you doing?