Is there a discussion already open for how heavy handed kedr

Question

Is there a discussion already open for how heavy handed kedro is on it s requirements I have some issues on occasion where things have moved since the last kedro release but kedro does not allow new enough versions currently my build reqs is broken because of this issue on pip tools https github com jazzband pip tools issues 1617 It s fixed after 6 5 1 but kedro does not want me to use it I am able to but get pip warnings all over Will raise an issue if there isn t already one somewhere

antony.milne · Answer

A few points Your picture isn t loading unfortunately so I can t see what the error is but we have had `pip tools~=6 5` since kedro 0 17 7 so I don t think it should be causing problems here Definitely I think there should be a conversation about kedro s handling of requirements in general see https github com kedro org kedro pull 1601 discussion r893090730 so please do feel free to open an issue about that if also have thoughts on the matter I m hoping both kedro core requirements and the project template ones will become lighter in the near future which will reduce any conflicts quite a bit See https github com kedro org kedro issues 1293 and issues spun out of it In general many of my comments on this were inspired by issues created by you actually so very interested if you have any feedback on any of it

waylonwalker · Answer

The picture was just of the kedro requirements pinning to `pip tools~=6 5`

waylonwalker · Answer

So actually this points to `6 5 1` I understand the`~` now What I need is `6 6 2` The issue here is that pip updated and removed `BAR TYPES` out from under them so now `6 5` is broken if you are using the latest `pip`

waylonwalker · Answer

Is there any reason that kedro itself locks many packages like this down rather than letting a template give users a known good starting point It feels like it would be easier to fluidly follow changes like this in a starter leave kedro a little more open and if users don t use your template they can install newer packages at their own risk

waylonwalker · Answer

Even once kedro fixes this issue in the next release would have to update from 0 17 7 to 0 18 2 which is more than I want to do until we have the bandwidth to move all of our projects This particular issue is not critical I can work around but that does not mean the next wont be as easy

antony.milne · Answer

No `pip tools~=6 5` is basically the same as `>=6 5 <7` so kedro is compatible with the latest pip and pip tools

antony.milne · Answer

FYI this the third time in the last six months there has been such a pip tools vs pip conflict This is one of the reasons I am inclined to drop pip tools from kedro requirements like you suggest it really belongs in project template requirements anyway

antony.milne · Answer

We are generally quite cautious conservative about our dependency bounds to try and ensure that dependencies don t release a breaking change that breaks kedro unexpectedly IMO we are probably too conservative here see discussion linked to above but we are not being as conservative as you think generally we just pin the major version as with `pip tools` here

antony.milne · Answer

Also this is very much part of my argument that we should be making kedro framework requirements light and loose in favour of more dependencies being project side or being dropped completely like in the case of pip tools So your general point is very much valid I think please do feel free to open an issue about it If upper bounds are causing problems and make upgrading awkward then it is good to have that recorded as evidence in favour of loosening our requirements Unfortunately we do also have much evidence of dependency releases often pip tools pip breaking kedro unexpectedly in the past Overall it s a tricky balance to get right because both situations cause user problems

antony.milne · Answer

P S if you don t know about it already I very much recommend `pipdeptree` in addition to `pip tools` for figuring out what would be causing conflicts for you