So we err on the side of explicit rather than implicit so we reserve that type of thing for the human input parts. What you could do is declare a

YAMLDataSet

for the parameters you generate at runtime for safe keeping, then either an automatic or manual process outside of Kedro could mirror those in you actual

paramerters.yml

Ahh, great idea, thank you! 😀

Yeah the balance between readability and dynamism is super hard to find

but I think that works okay

Definitely!

This seems like it addresses my concerns almost fully, thank you for sharing

I ran

kedro viz

but “there is no active Kedro session”. Does anyone know how to make it work? I had managed to run it successfully before, just not sure what has changed since then.

Viz session issue

Is there a way to output a tuple in a node, without having to split them to separate variables?

Of course - just in your regular function:

python

my_node_func() -> Tuple[int]:
   return tuple([1,2])

But in the pipeline I have to specify two output variables, e.g., in your example:

python
Pipeline([
  node(func=my_node_func, input=None, output=["a","b"])
])

Or am I wrong?

ah you just provide one output!

Like

output=["ab"]

?

So in this case you would just need to return

output="a"

and

"a"

would store a tuple

hmm, it seems like it works for other datatypes than a pandas dataframe, e.g, if I return two dataframes in a tuple, it will like to split them

are you getting an error if you try that with Pandas?

One thing you can do if it's giving you and error In

catalog.yml

yaml
a:
  type: MemoryDataSet
  copy_mode: copy

No, it was actually my own fault. I forgot to remove the [] around the output as you said. Thank you!

Hi everyone, kinda new to kedro. I was looking for some examples where in I can pass different datasets to the same pipeline(reusing same pipeline code for different datasets) to process information but unable to find anything. Can anyone help?

This is exactly what modular pipelines are for - I have a work in progress example project here https://github.com/datajoely/modular-spaceflights

Thanks for the fast response. I'll go through it 😃

I suppose I could output a dummy dataset.

Is there a built in dataset that saves to a database? Or do we have to create our own class for that?

Yes there are pandas and spark database connectors

kedro.extras.datasets.pandas.SQLQueryDataSet

kedro.extras.datasets.pandas.SQLTableDataSet

kedro.extras.datasets.spark.SparkJDBCDataSet

kedro.extras.datasets.spark.SparkHiveDataSet

I tried doing a modification to the data ingestion pipeline in https://github.com/datajoely/modular-spaceflights by adding a node to save to a db; in

data_ingestion/pipeline.py

node(
   name="upload_to_db",
   func=lambda x: x,
   input="shuttles",
   output="shuttles_table",
    ),

in

catalog_01_raw.yml

shuttles_table:
  type: pandas.SQLTableDataSet
  table_name: shuttles
  credentials: postgres
  save_args:
    if_exists: replace

but the log shows that

shuttles_table

is a

MemoryDataSet

2021-12-18 08:24:36,993 - kedro.pipeline.node - INFO - Running node: <lambda>([shuttles]) -> [data_ingestion.shuttles_table]
2021-12-18 08:24:36,993 - kedro.io.data_catalog - INFO - Saving data to `data_ingestion.shuttles_table` (MemoryDataSet)...

And the table is not created in the database.

✔The solution was to add the output in the

new_ingestion_pipeline

Pipeline

. I didn't realise that creating another function just to add a namespace to an existing

Pipeline

is done.

If a node (node

B

) is dependent on a previous node (node

A

) having uploaded to a database (e.g.

some_table

as

pandas.SQLTableDataSet

) and I use

some_table

as the input for

B

, does

B

automatically try to download

some_table

to memory (if not already in memory)? I would not like the data downloaded if: - the data is large, hence most of the pipeline time is spent on downloading - `B`'s code is to run SQL queries without ever requiring the data locally

So we don’t support SQL as a remote execution engine - today we bring things into python world. If you use PySpark it will expose sql as data frames and do some of this lazily

We use SQL as a storage layer not an execution engine