RRoger
12/17/2021, 9:53 PMB
) is dependent on a previous node (node A
) having uploaded to a database (e.g. some_table
as pandas.SQLTableDataSet
) and I use some_table
as the input for B
, does B
automatically try to download some_table
to memory (if not already in memory)? I would not like the data downloaded if:
- the data is large, hence most of the pipeline time is spent on downloading
- `B`'s code is to run SQL queries without ever requiring the data locally