ya ya exactly - like if I want to keep the last 5 ...
# beginners-need-helpp
pypeaday
03/30/2022, 3:58 PMya ya exactly - like if I want to keep the last 5 versions, and then versions at the beginning of the month for the past 12 months, and then annual versions for the last X years or whatever...
d
datajoely
03/30/2022, 4:00 PMInteresting question - it's not come up before to my knowledge
p
pypeaday
03/30/2022, 4:00 PMnever?? wow that's shocking
pypeaday
03/30/2022, 4:00 PMWaylon and I have a plugin idea if it's not natively supported
d
datajoely
03/30/2022, 4:00 PMI think if S3 has policies to do this it may be preferable
datajoely
03/30/2022, 4:01 PMbut it might be easiest in an
after_pipeline_runp
pypeaday
03/30/2022, 4:01 PMthat was what I thought the Kedro position would be - rely on underlying storage lifecycle stuff
pypeaday
03/30/2022, 4:01 PMI'd like to see it supported in Kedro abstracted from the underlying file storage to be honest
d
datajoely
03/30/2022, 4:02 PMyeah I think we avoid writing code that 'deletes' in general
p
pypeaday
03/30/2022, 4:02 PMI'm thinking also of servers where data is stored just on disk or anything
d
datajoely
03/30/2022, 4:02 PMI guess I have a question for you
p
pypeaday
03/30/2022, 4:02 PMsure, makes sense
d
datajoely
03/30/2022, 4:02 PMwould you want this routing to run as part of a lifecycle of a run
datajoely
03/30/2022, 4:02 PMor as a background process
datajoely
03/30/2022, 4:02 PMbecause we can only really support the first
datajoely
03/30/2022, 4:03 PMunless we introduced something like
kedro catalog prunep
pypeaday
03/30/2022, 4:03 PMgreat question... in my sys admin experience I'd have a daemon running on the server to do this kind of thing... in this case I think something that executes during a pipeline run is the only thing that makes sense but I now see how that might be undesired
d
datajoely
03/30/2022, 4:03 PMwhich you ran as a CI process
p
pypeaday
03/30/2022, 4:03 PMya that could be really nice
d
datajoely
03/30/2022, 4:03 PMso that would make sense as a plug-in
p
pypeaday
03/30/2022, 4:03 PMcoolio, no idea when I'd get to it but I think we could come up with something... Waylon is pretty good after all
d
datajoely
03/30/2022, 4:04 PMI've extended the
kedro catalogkedro-richp
pypeaday
03/30/2022, 4:04 PMawesome I'll keep that in mind, thank you!
d
datajoely
03/30/2022, 4:05 PMI've always wanted to do something similar for APIDataSet
datajoely
03/30/2022, 4:05 PMwe could cachce the response until it 'expires' amd then do a new API request
datajoely
03/30/2022, 4:05 PMinstead of doing a fresh call each time
datajoely
03/30/2022, 4:05 PMnot the same thing, but similar in some ways
p
pypeaday
03/30/2022, 4:06 PMon the backside of it ya very similar
pypeaday
03/30/2022, 4:06 PMI could see that being super useful - especially in testing if you wanted to cache a real api response and use it in tests in CI or something
pypeaday
03/30/2022, 4:07 PMwhich is now a different use case I guess but the caching nonetheless would be nice there
a
antony.milne
04/04/2022, 8:43 AMRelated - please do comment here if you're interested in it 🙂 https://github.com/kedro-org/kedro/issues/406
antony.milne
04/04/2022, 8:44 AMFrom what I remember from previous discussions though, this has been considered before but seemed quite hard to implement in full generality (for as many datasets as possible), so I guess there was not sufficient interest in it before for us to prioritise it compared to the effort involved
2 Views