RRoger
12/17/2021, 9:25 PMdata_ingestion/pipeline.py
node(
name="upload_to_db",
func=lambda x: x,
input="shuttles",
output="shuttles_table",
),
in catalog_01_raw.yml
shuttles_table:
type: pandas.SQLTableDataSet
table_name: shuttles
credentials: postgres
save_args:
if_exists: replace
but the log shows that shuttles_table
is a MemoryDataSet
2021-12-18 08:24:36,993 - kedro.pipeline.node - INFO - Running node: <lambda>([shuttles]) -> [data_ingestion.shuttles_table]
2021-12-18 08:24:36,993 - kedro.io.data_catalog - INFO - Saving data to `data_ingestion.shuttles_table` (MemoryDataSet)...
And the table is not created in the database.