https://kedro.org/ logo
#advanced-need-help
Title
# advanced-need-help
m

mlemainque

10/11/2021, 9:43 AM
For the first point, I was thinking of something like this:
Copy code
yaml
incremental_sql_dataset:
  type: SQLQueryDataSet
  sql: SELECT * FROM table WHERE id > %(checkpoint)s
  checkpoint:
    column: id  # Which column to use to update the checkpoint based on the loaded content
    filepath: ... # Where to store the checkpoint (same as for partitioned incremental datasets)
But you're right it could easily be done with a custom implementation