idriss__
03/24/2022, 9:32 AMdatajoely
03/24/2022, 9:52 AMafter_node_run
hook is probably the bestcatalog.releases(dataset_name)
methodidriss__
03/24/2022, 1:32 PMdatajoely
03/24/2022, 1:36 PMidriss__
03/24/2022, 2:50 PMdatajoely
03/24/2022, 3:01 PMPartitionedDataSet
or IncrementalDataSet
to make life easier?idriss__
03/24/2022, 3:04 PMdatajoely
03/24/2022, 3:05 PMIncrementalDataSet
it may help hereidriss__
03/24/2022, 3:40 PMdatajoely
03/24/2022, 3:44 PMpartitions
dictionary already contains the data already loaded into memory.load()
that has been highlighted it will return a lazy object reference that you could load()
yourself in the nodePartitionedDataSet
just like you have for your custom image datasetidriss__
03/24/2022, 3:51 PMdatajoely
03/24/2022, 3:51 PMidriss__
03/24/2022, 3:56 PMdatajoely
03/24/2022, 3:57 PMidriss__
03/24/2022, 3:58 PMdatajoely
03/24/2022, 3:58 PMidriss__
03/24/2022, 3:59 PMdatajoely
03/24/2022, 3:59 PMidriss__
03/24/2022, 3:59 PMdatajoely
03/24/2022, 3:59 PMidriss__
03/24/2022, 4:01 PMdatajoely
03/24/2022, 4:05 PMidriss__
03/24/2022, 4:11 PMdatajoely
03/24/2022, 4:40 PMidriss__
03/24/2022, 4:52 PMdatajoely
03/24/2022, 4:53 PMidriss__
03/24/2022, 4:53 PMdatajoely
03/24/2022, 4:53 PMidriss__
03/24/2022, 4:53 PMdatajoely
03/24/2022, 4:54 PMidriss__
03/24/2022, 4:54 PMdatajoely
03/24/2022, 4:54 PMidriss__
03/24/2022, 4:54 PMdatajoely
03/24/2022, 4:57 PMdeepyaman
04/07/2022, 1:28 AMPartitionedDataSet
.PartitionedDataSet
loop.datajoely
04/07/2022, 8:56 AMRRoger
06/04/2022, 12:34 PMCallable
values doesn't seem to have the expected behaviour. The point is to save each partition one at a time right? It seems that all the files are saved at the same time (see screenshot); each file actually took 4 minutes to process. So it seems like the outputs are stored in RAM until the end of the node. Or am I missing something?Mackson
06/07/2022, 2:16 PMdatajoely
06/07/2022, 2:43 PMPartitionedDataSet
to prove that?Mackson
06/07/2022, 4:24 PM