Hello Kedro team!
I am new to Kedro and I am trying to assess in which ways my team could use this framework in their daily work to replace other heavy tools. For now it is almost exactly the framework we were looking for so long, great work!
One of my concern is about incremental datasets as we often work with huge partitioned datasets fed on a regular basis. I have two questions:
1- Is it planned to integrate the incremental behavior to other datasets than fsspec-based ones? (such as SQL). The checkpoint could be somehow based on one datetime or incremental-id column in the table...
2- Is there any way to somehow load one node's output content? A typical use case is when we want to transform a non-incremental dataset to an incremental one, we read the input data and do an anti join with the output's previous content. But I saw it is not currently possible to have one dataset both as input and output of one node (even though it should not be a problem to solve the DAG order)
Thanks for your time,