Rafał
04/26/2022, 3:12 PMavan-sh
04/26/2022, 4:12 PMdataset_name = catalog.load("<your_versioned_dataset_name>")
pattern = str(dataset_name._get_versioned_path("*"))
version_paths = sorted(dataset_name._glob_function(pattern), reverse=True)
I'm guessing @datajoely means the second approachdatajoely
04/26/2022, 4:14 PMRafał
04/26/2022, 5:56 PMdatajoely
04/26/2022, 6:02 PMcatalog.datasets.{dataset_name}
you get access to the lazy object not the dataRafał
04/26/2022, 6:03 PMdatajoely
04/26/2022, 6:03 PMRafał
04/26/2022, 6:05 PMdatajoely
04/26/2022, 6:15 PMRafał
04/26/2022, 6:36 PMdataset = catalog.datasets.__your_dataset_name___
pattern = dataset._get_versioned_path('*')
version_paths = sorted(dataset._glob_function(pattern), reverse=True)
s3
protocol, I got the follwoing error:
sorted(dataset._glob_function(pattern), reverse=True)
File "/opt/miniconda3/envs/deep-identifier/lib/python3.8/site-packages/fsspec/asyn.py", line 91, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/opt/miniconda3/envs/deep-identifier/lib/python3.8/site-packages/fsspec/asyn.py", line 71, in sync
raise return_result
File "/opt/miniconda3/envs/deep-identifier/lib/python3.8/site-packages/fsspec/asyn.py", line 25, in _runner
result[0] = await coro
File "/opt/miniconda3/envs/deep-identifier/lib/python3.8/site-packages/s3fs/core.py", line 624, in _glob
if path.startswith("*"):
AttributeError: 'PurePosixPath' object has no attribute 'startswith'
`
dataset = catalog.datasets.__your_dataset_name___
pattern = dataset._get_versioned_path('*')
version_paths = sorted(dataset._glob_function(str(pattern)), reverse=True)
datajoely
04/27/2022, 10:02 AM