08/11/2022, 7:52 PM
Hello folks, quick question on PartionedDatasets. I see that at load time I can access partitions at will by using the partition_id (basically the rest of the path where it is saved). Is there a way I can achieve the same incremental effect at writing time? Meaning at write time write partitions incrementally in such a way that I can relieve memory when the node is running? I think this can only be done by multiple nodes but I'm not sure.
huuum acctually, having a dict of callables can maybe help me.