mlemainque
10/11/2021, 9:43 AMyaml
incremental_sql_dataset:
type: SQLQueryDataSet
sql: SELECT * FROM table WHERE id > %(checkpoint)s
checkpoint:
column: id # Which column to use to update the checkpoint based on the loaded content
filepath: ... # Where to store the checkpoint (same as for partitioned incremental datasets)
But you're right it could easily be done with a custom implementation