https://kedro.org/ logo
e

edhenry

07/28/2022, 5:05 PM
Has anyone played with the idea of a Kedro Dataset that honors the PyTorch IterableDatasets* interface? I ask because it would we have rather large datasets that don't fit in memory on a single machine and we would like to maintain consistency with our use of Kedro's catalog functionality, if possible. * https://pytorch.org/docs/stable/data.html#iterable-style-datasets