mlemainque
10/11/2021, 9:43 AMyaml
incremental_sql_dataset:
  type: SQLQueryDataSet
  sql: SELECT * FROM table WHERE id > %(checkpoint)s
  checkpoint:
    column: id  # Which column to use to update the checkpoint based on the loaded content
    filepath: ... # Where to store the checkpoint (same as for partitioned incremental datasets)
But you're right it could easily be done with a custom implementation