https://kedro.org/ logo
b

Burn1n9m4n

04/26/2022, 7:47 PM
So...it would go something like: (1) Setup
IncrementalDataSet
(2) Run logic within function. If dataset is empty, it will just return an empty
DataFrame
. (3) That
DataFrame
gets saved as a parquet within S3. (4) That parquet gets loaded within the next node. Its (4) that I'm not sure about. That would load a an empty parquet which would require some subsequent handling, I suspect.