Title
r
Rafał
04/26/2022, 3:12 PMHello, I am wondering if there is any possibility to list all available versions of kero's catalog versioned dataset ?
a
avan-sh
04/26/2022, 4:12 PMOne simple way to get the list of all available versions would be to get a list of all folders in the high level folder.
Another approach is possibly to leave version as empty(not pass version), load the object in ipython session and run the following (code from VersionedDataset Implementation)
dataset_name = catalog.load("<your_versioned_dataset_name>")
pattern = str(dataset_name._get_versioned_path("*"))
version_paths = sorted(dataset_name._glob_function(pattern), reverse=True)d
datajoely
04/26/2022, 4:14 PMI did mean this, but didn't realise it was so complicated
Perhaps it makes sense to expose this as a property
The session store will eventually cover more of this
r
Rafał
04/26/2022, 5:56 PMThe best would be to expose catalog item before loading it. I would then call ds=catalog.get(„datasetname”)
after that I could load any version I want and of course ask for list of available versions.
For exampl if I had raw DataSet then I could take filepath in order to get top level root directory of VersionedDataSet
d
datajoely
04/26/2022, 6:02 PMso if you do
catalog.datasets.{dataset_name}so you can do everything @avan-sh suggests in an interactive session
r
Rafał
04/26/2022, 6:03 PMOh that’s great news for me.
d
datajoely
04/26/2022, 6:03 PMbut I think it makes sense for me to make this much easier for users
we do for partitioned dataset, no reason not to here
r
Rafał
04/26/2022, 6:05 PMI see and agree it is quite complicated. Especially to load the single item in order to resolve all others versions.
d
datajoely
04/26/2022, 6:15 PMyes
it's trivial to make a little convenience function
but not accessible to newbies
r
Rafał
04/26/2022, 6:36 PMI thought that one has to load the single item.
Actually it works without loading any item.
dataset = catalog.datasets.__your_dataset_name___
pattern = dataset._get_versioned_path('*')
version_paths = sorted(dataset._glob_function(pattern), reverse=True)Would be great to parse the paths and expose only versions, but I think I would manage to do that.
Thank you guys.
Actually, this method works only for local filepath.
In the case of
s3sorted(dataset._glob_function(pattern), reverse=True)
File "/opt/miniconda3/envs/deep-identifier/lib/python3.8/site-packages/fsspec/asyn.py", line 91, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/opt/miniconda3/envs/deep-identifier/lib/python3.8/site-packages/fsspec/asyn.py", line 71, in sync
raise return_result
File "/opt/miniconda3/envs/deep-identifier/lib/python3.8/site-packages/fsspec/asyn.py", line 25, in _runner
result[0] = await coro
File "/opt/miniconda3/envs/deep-identifier/lib/python3.8/site-packages/s3fs/core.py", line 624, in _glob
if path.startswith("*"):
AttributeError: 'PurePosixPath' object has no attribute 'startswith'
`This helped
dataset = catalog.datasets.__your_dataset_name___
pattern = dataset._get_versioned_path('*')
version_paths = sorted(dataset._glob_function(str(pattern)), reverse=True)d
datajoely
04/27/2022, 10:02 AMYes that makes sense