datajoely
06/22/2021, 1:38 PMuser
06/22/2021, 1:44 PMdatajoely
06/22/2021, 1:44 PMdatajoely
06/22/2021, 1:45 PMhttps://youtu.be/fULOrO-QpsE▾
datajoely
06/22/2021, 1:52 PMwaylonwalker
06/22/2021, 2:05 PMuser
06/22/2021, 3:33 PMuser
06/22/2021, 3:34 PMuser
06/22/2021, 3:34 PMuser
06/22/2021, 3:35 PMuser
06/22/2021, 3:36 PMuser
06/22/2021, 3:36 PMuser
06/22/2021, 3:37 PMImageDataset
supports images but not annotationsdatajoely
06/22/2021, 3:37 PMuser
06/22/2021, 3:43 PMJSONDataset
, but I'm not sure where to start with defining a COCODataset
. Extending the JSONDataset
class to deal with the COCO syntax doesn't seem immediate. There's a Python module pycocotools
which can be used to parse COCO annotation files, but having a COCODataset type in Kedro would be useful to, for example, examine the dataset using kedro ipython
and then `config.load()‘datajoely
06/22/2021, 3:51 PMAbstractDataSet
and implement the __init__()
, load()
and save()
methods.
Looking at this example from pycocotools I think it is super easy for you to just need to import from pycocotools.coco import COCO
and do COCO(your_path)
within the load()
method and you're good to go.
https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocoDemo.ipynb
The only thing worth noting is that this will only work for local files - if you want the dataset to accept either local or cloud files (s3 etc) you may want to extend / steal the basic implementation from any of the existing Kedro datasetsuser
06/22/2021, 4:12 PMnoklam
06/24/2021, 5:19 AMdatajoely
06/24/2021, 6:56 AMwaylonwalker
06/25/2021, 6:45 PMnoklam
06/28/2021, 2:22 AMwaylonwalker
06/30/2021, 2:28 PMafter_node_run
hook that checks all CachedDataSets
to see if all dependents have been satisfied, and runs release()
only after all dependents of that dataset have been ran automatically?waylonwalker
06/30/2021, 2:30 PMdatajoely
06/30/2021, 2:30 PMdatajoely
06/30/2021, 2:33 PMdatajoely
06/30/2021, 2:34 PMdatajoely
06/30/2021, 2:34 PMwaylonwalker
07/01/2021, 12:34 AMwaylonwalker
07/01/2021, 12:37 AMrelease()
after you have moved completely passed all groups using a certain dataset.Yetunde
07/01/2021, 8:58 AMYetunde
07/01/2021, 8:58 AM