Hello folks, quick question on PartionedDatasets. I see that at load time I can access partitions at will by using the partition_id (basically the rest of the path where it is saved).
Is there a way I can achieve the same incremental effect at writing time? Meaning at write time write partitions incrementally in such a way that I can relieve memory when the node is running?
I think this can only be done by multiple nodes but I'm not sure.