Then you can use it like a normal variable, the string literal is just alias for the variable. ["mapped_df", "altitude", "disease"]

and if id_vars be a dictionary?

The node is not aware about the type, it just treats it as a variable. You can also pass in named argument. You can also use a dictionary of string literal as node input/output For example https://github.com/quantumblacklabs/kedro-starters/blob/main/pandas-iris/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipelines/data_engineering/pipeline.py

visualising namespaces

@User is there a way to set any kind of lifecycle on versioned datasets? I'm not seeing anything in docs about that... Or is Kedro's position that we should use an underlying filesystem's capabilities here (like ZFS snapshots or S3 lifecycle policies)? @User fyi

what do you mean lifecycle? Some sort of expiry?

ya ya exactly - like if I want to keep the last 5 versions, and then versions at the beginning of the month for the past 12 months, and then annual versions for the last X years or whatever...

Hello there! Is there a way to get the input name inside a node function? For example,

node(func=melt_data, inputs="fcl_elevation")
...

def melt_data(df):
    # how to get "fcl_elevation" inside func?

is there a way to flag a dataset to not run asynchronously and to wait until other nodes are complete, if asnyc is enabled?

The best way to do that is to break the pipeline into pieces and execute from the CLI like

kedro run --pipeline a && kedro run --pipeline b --async && kedro run --pipeline c

&& wil wait for the previous statement to complete, single & will do both simultaneously

Hello guys! I would like to know if it is possible to "extract" the file path of a partitioned dateset catalog used as an input in a node. I tried to look in the internet for alternatives but I couldn't find a solution for it.

Hi all I'm trying to get the TemplatedConfigLoader to work. I'm getting this error

It must be a subclass of kedro.config.config.ConfigLoader

. It appears the TemplatedConfigLoader is inheriting from AbstractConfigLoader in 0.18.0 any idea how to fix this? Tried to change to inheriting from ConfigLoader itself, passes the assert but doesn't use my globals.yml. Thank you

using individual partitions

ConfigLoader issue with 0.18.x

Hey guys! Quick question. How could I handle in the catalog a directory that can have an unknown number of files, but conveniently named?

PartitionedDataset!

This seems nice. My use case is more ML driven. Think of it as train test sets but generated by another team/application.

So I think it should work - but we do have an assumption that things are reproducible so be careful!

yep. Tehy should have the same structure and just be named properly. Ill look into partitioned dataset!

These seem cool. Can the incremental dataset be used to run a pipeline per partition?

Incremental checkpoints the last partition seen

Let's say I have thousands of operations to perform which are computationally expensive. Each iteration yields a set of parameters which I'd like to write one at a time to the same file -- whether this be as rows of a csv, SQL table or whatever -- so that already written data is preserved should the script fail on a particular iteration. What is the "Kedro approved" factory pattern for a use case like this? Any advice would be much appreciated, cheers.

Hey guys. I was following the tutorial on experiment tracking, but all of my data is being saved on memory dataset, regardless of what the catalog says, and the folder 09 is not created, nor are the results saved. Any idea why that would happen? Tutorial: https://kedro.readthedocs.io/en/stable/tutorial/set_up_experiment_tracking.html Python version: 3.8.5 Kedro version: 0.18.0

2022-04-07 06:29:25,232 - kedro.io.data_catalog - INFO - Saving data to `data_processing.preprocessed_companies` (MemoryDataSet)...

preprocessed_companies:
  type: pandas.ParquetDataSet
  filepath: data/02_intermediate/preprocessed_companies.pq
  layer: intermediate

Would be great if you can share a gist or repo of your working directory

factory pattern

Ok, created a temporary repo for that https://github.com/Zemeio/kedro_experiment/tree/main/new-kedro-project

spaceflights tutorial is not saving dataset correctly to be shown in kedro-viz

Hi everyone, I got this error as well (https://github.com/kedro-org/kedro/issues/1409), I would like to know if anybody else got the same error?

(.venv) sebaxtian@Lenovo:~/Workspaces/Sebaxtian/kedro-hello-world$ python hello_kedro.py
Traceback (most recent call last):
  File "hello_kedro.py", line 39, in <module>
    print(runner.run(greeting_pipeline, data_catalog))
TypeError: run() missing 1 required positional argument: 'hook_manager'

kedro version: v0.18.0

The hello kedro example is currently broken and we will be fixing it I'm the next few days. For now the spaceflights tutorial is working as expected!