Hello folks! I have a question regarding IncrementalDataSet catalog entry. In the documentation, it is specified that a checkpoint file will be created /at the location/ of the dataset to remember which entries have already been processed or not. My question is: if multiple data scientists are running the pipeline from different computer, will the checkpoint file remember which computer has already process which entries? Or will one user have missing points if another has already processed them on its side?

So the checkpoint is technically a whole

kedro.extras.datasets.text.TextDataSet

of its own so it will be persisted at the storage location, which if accessible team wide will be shared

Thank you for your, as always, prompt and helpful answer 🙂

The ID generated is a timestamp at I think millisecond precision which is very unlikely to be non-uniqiue, but possible