https://kedro.org/ logo
Title
r

RRoger

12/17/2021, 9:25 PM
I tried doing a modification to the data ingestion pipeline in https://github.com/datajoely/modular-spaceflights by adding a node to save to a db; in
data_ingestion/pipeline.py
node(
   name="upload_to_db",
   func=lambda x: x,
   input="shuttles",
   output="shuttles_table",
    ),
in
catalog_01_raw.yml
shuttles_table:
  type: pandas.SQLTableDataSet
  table_name: shuttles
  credentials: postgres
  save_args:
    if_exists: replace
but the log shows that
shuttles_table
is a
MemoryDataSet
2021-12-18 08:24:36,993 - kedro.pipeline.node - INFO - Running node: <lambda>([shuttles]) -> [data_ingestion.shuttles_table]
2021-12-18 08:24:36,993 - kedro.io.data_catalog - INFO - Saving data to `data_ingestion.shuttles_table` (MemoryDataSet)...
And the table is not created in the database.