This is still valid, by default when you create a new kedro project, let's say the project is called

demo

, then you will have

demo/__init__.py

,

demo/pipelines/__init__.py

, this is why you can import from there. If you want to factor out some code into a separate module. You will create a new folder and file like

demo/common/__init__.py

etc.

Hi all. I have a beginner question, on Kedro 0.18.2. I have a TemplatedConfigLoader that loads a partitioned dataset from a specific path

raw_dataset: 
  type: PartitionedDataSet
  dataset:
    type: pandas.CSVDataSet
    load_args:
      index_col: False
  path: data/01_raw/${asset}
  filename_suffix: ".csv"
  layer: raw

and my settings.py reads

CONFIG_LOADER_CLASS = TemplatedConfigLoader
CONFIG_LOADER_ARGS = {
    "globals_pattern": "*parameters.yml",
}

When setup this way my pipeline works as expected, pulling the asset variable from parameters.yml. However now I want to be able to override this variable from the command line e.g.

kedro run --params=asset:1234

Any advice on how I can do that? I went on a wild goose hunt on Google and the Kedro github but still can't quite figure it out. It's pretty similar to this issue https://github.com/kedro-org/kedro/issues/1527#issuecomment-1127697871 but i couldn't get the suggested solution to work.

You need to implement the

MyTemplatedConfigLoader

example half way down the page. The CLI arguments will be in

extra_params

and you need to ensure that these are passed to

globals_dict

HI @datajoely, I did see and attempt to implement that example, but am still stuck. I access

extra_params

by loading the context. I loaded it this way:

with KedroSession.create("elevator-eventmasker") as session:
    context = session.load_context()

context.params

gives the variables in parameters.yml as expected. However

context._extra_params

just gives

None

rather than the CLI arguments.

They're an argument to the create method

Runtime Config to override Catalog

Hey guys, is there any way to show all features that could be used on Data Catalog, like that autocompletion from docstrings and such when you like, type a method and shows all things you can define within, but for data catalog?

like these ones...

hey @datajoely how you doing?

In your yaml editor?

vscode

Right but for editing catalog.yml?

yes

So if you look at the VS Code page of kedro docs you can configure your editor to use jsonschema for autocomplete

ohh, that is awesome

I'll check it right away

Thank you btw

New to Kedro and beginning a first project to test the waters. How are most of you getting data in/out of Kedro other than files? My environment is Kafka queue where I subscribe to a few topics and perform inference on those topics; results are placed onto another topic. Thx

Hi @bushbo so Kedro is a fundamentally batch based tool. The best way to work with Kafka is to sink it into a datbase/filestore to read from.

It wouldn't make sense to run Kedro on each event

Hi Kedro team! Google is releasing a new managed service called Batch for google cloud. (seems like aws batch... ). I think kedro would match quite well with it. Do you plan to evaluate it in your roadmap?

Hi there, is there any way to change the data type from

MemoryDataSet

when creating a

Catalog

from the command line

kedro catalog create --pipeline <pipeline_name>

, like to set another

flag

to change its type or any other config that you might want?

Unfortunately it doesn't accept additional arguments, the main purpose of it is to create Catalog entry quickly, which is by default

MemoryDataSet

. It is hard to pre-determined these data types therefore we assume users will edit them per entry and work with the generated file directly.

I see, well thank you for answering

Thanks for replying to my question; I am currently toying with the concept of putting my data into InfluxDB (I am working with time series data) and then using in Kedro

Very cool stack! Kedro will naturally work better with aggregated outputs rather than events

Yes, batch ML algorithms are more common and somewhat easy to nail down than event based ones.

I'm running into an issue. When I run

kedro run --from-node nodeX

the process hangs and I have to kill that process. After that when I just run a standard

kedro run

when it gets to the node that I attempted to

nodeX

in the pipeline it hangs until I reboot my server.

Kedro is hanging after the first node when attempting to save data. It finishes the function associated with the node; however, the process never ends and no data is saved.

What sort of function are you running and what dataset is the target dataset? In general jumping into a debugger should tell you why things are failing