Burn1n9m4n
04/26/2022, 7:47 PMIncrementalDataSet
(2) Run logic within function. If dataset is empty, it will just return an empty DataFrame
.
(3) That DataFrame
gets saved as a parquet within S3.
(4) That parquet gets loaded within the next node.
Its (4) that I'm not sure about. That would load a an empty parquet which would require some subsequent handling, I suspect.