https://kedro.org/ logo
Join Discord
Powered by
Is there a discussion already open for how heavy-h...
# beginners-need-help
w
Is there a discussion already open for how heavy-handed kedro is on it's requirements? I have some issues on occasion where things have moved since the last kedro release, but kedro does not allow new enough versions. currently my build-reqs is broken because of this issue on pip-tools. https://github.com/jazzband/pip-tools/issues/1617 It's fixed after 6.5.1, but kedro does not want me to use it (I am able to, but get pip warnings all over). Will raise an issue if there isn't already one somewhere.
a
A few points... * Your picture isn't loading unfortunately so I can't see what the error is but we have had 
pip-tools~=6.5
since kedro 0.17.7 so I don't think it should be causing problems here. * Definitely I think there should be a conversation about kedro's handling of requirements in general (see https://github.com/kedro-org/kedro/pull/1601#discussion_r893090730) so please do feel free to open an issue about that if also have thoughts on the matter * I'm hoping both kedro core requirements and the project template ones will become lighter in the near future which will reduce any conflicts quite a bit. See https://github.com/kedro-org/kedro/issues/1293 and issues spun out of it 🙂 (In general many of my comments on this were inspired by issues created by you actually, so very interested if you have any feedback on any of it!)
w
The picture was just of the kedro requirements pinning to 
pip-tools~=6.5
So actually this points to 
6.5.1
, I understand the`~` now. What I need is 
6.6.2
. The issue here is that pip updated and removed 
BAR_TYPES
out from under them, so now 
6.5
is broken if you are using the latest 
pip
Is there any reason that kedro itself locks many packages like this down rather than letting a template give users a known good starting point? It feels like it would be easier to fluidly follow changes like this in a starter, leave kedro a little more open, and if users don't use your template they can install newer packages at their own risk.
Even once kedro fixes this issue in the next release would have to update from 0.17.7 to 0.18.2, which is more than I want to do until we have the bandwidth to move all of our projects. This particular issue is not critical, I can work around, but that does not mean the next wont be as easy.
a
No, 
pip-tools~=6.5
is basically the same as 
>=6.5, <7
so kedro is compatible with the latest pip and pip-tools.
FYI this the third time in the last six months there has been such a pip-tools vs. pip conflict 😬 This is one of the reasons I am inclined to drop pip-tools from kedro requirements (like you suggest, it really belongs in project template requirements anyway)
We are generally quite cautious/conservative about our dependency bounds to try and ensure that dependencies don't release a breaking change that breaks kedro unexpectedly. IMO we are probably too conservative here (see discussion linked to above), but we are not being as conservative as you think - generally we just pin the major version as with 
pip-tools
here.
Also this is very much part of my argument that we should be making kedro framework requirements light and loose in favour of more dependencies being project-side (or being dropped completely, like in the case of pip-tools). So your general point is very much valid I think - please do feel free to open an issue about it. If upper bounds are causing problems and make upgrading awkward then it is good to have that recorded as evidence in favour of loosening our requirements. Unfortunately we do also have much evidence of dependency releases (often pip-tools/pip) breaking kedro unexpectedly in the past. Overall it's a tricky balance to get right because both situations cause user problems
P.S. if you don't know about it already, I very much recommend 
pipdeptree
in addition to 
pip-tools
for figuring out what would be causing conflicts for you
5 Views
PreviousNext
Powered by