Hello, I have a question regarding PartitionedData...
# advanced-need-help
r
Hello, I have a question regarding PartitionedDataSet. Assume that some node A returns the dictionary which is saved as PartitionedDataset B. I have observed that if I call pipeline several time and I return different dictionaries in node A (assuming some parameters have changes) then the dataset is B is beeing incremented (I know this is not IncrementalDataSet). So the old key files exists. Actually this is ok for me. So here is my question. Is it possible to tell the node A to compute only keys that do not exists in dataset B yet? I am wondering if it is possible to use
before_node_run
check the existing keys of B, and what next? How to provide such info to the node?