Hello I am wondering if there is any possibility to list all

Question

Hello I am wondering if there is any possibility to list all available versions of kero s catalog versioned dataset

avan-sh · Answer

One simple way to get the list of all available versions would be to get a list of all folders in the high level folder Another approach is possibly to leave version as empty not pass version load the object in ipython session and run the following code from VersionedDataset Implementation ``` dataset name = catalog load pattern = str dataset name get versioned path version paths = sorted dataset name glob function pattern reverse=True ``` I m guessing < datajoely> means the second approach

datajoely · Answer

I did mean this but didn t realise it was so complicated

datajoely · Answer

Perhaps it makes sense to expose this as a property

datajoely · Answer

The session store will eventually cover more of this

Rafał · Answer

The best would be to expose catalog item before loading it I would then call ds=catalog get datasetname after that I could load any version I want and of course ask for list of available versions

Rafał · Answer

For exampl if I had raw DataSet then I could take filepath in order to get top level root directory of VersionedDataSet

datajoely · Answer

so if you do `catalog datasets dataset name ` you get access to the lazy object not the data

datajoely · Answer

so you can do everything < avan sh> suggests in an interactive session

Rafał · Answer

Oh that s great news for me

datajoely · Answer

but I think it makes sense for me to make this much easier for users

datajoely · Answer

we do for partitioned dataset no reason not to here

Rafał · Answer

I see and agree it is quite complicated Especially to load the single item in order to resolve all others versions

datajoely · Answer

yes

datajoely · Answer

it s trivial to make a little convenience function

datajoely · Answer

but not accessible to newbies

Rafał · Answer

I thought that one has to load the single item Actually it works without loading any item ``` dataset = catalog datasets your dataset name pattern = dataset get versioned path version paths = sorted dataset glob function pattern reverse=True ```

Rafał · Answer

Would be great to parse the paths and expose only versions but I think I would manage to do that Thank you guys

Rafał · Answer

Actually this method works only for local filepath In the case of `s3` protocol I got the follwoing error ``` sorted dataset glob function pattern reverse=True File opt miniconda3 envs deep identifier lib python3 8 site packages fsspec asyn py line 91 in wrapper return sync self loop func args kwargs File opt miniconda3 envs deep identifier lib python3 8 site packages fsspec asyn py line 71 in sync raise return result File opt miniconda3 envs deep identifier lib python3 8 site packages fsspec asyn py line 25 in runner result 0 = await coro File opt miniconda3 envs deep identifier lib python3 8 site packages s3fs core py line 624 in glob if path startswith AttributeError PurePosixPath object has no attribute startswith ````

Rafał · Answer

This helped ``` dataset = catalog datasets your dataset name pattern = dataset get versioned path version paths = sorted dataset glob function str pattern reverse=True ```

datajoely · Answer

Yes that makes sense