Hello there! I'm new to Kedro. I was reading the r...
# beginners-need-help
b
Hello there! I'm new to Kedro. I was reading the release notes for 0.18 and noticed
kedro install
is being depreciated in favour of
pip install -r src/requirements.txt
to install dependencies. Does this effectively mean that managing dependencies via conda alone is now depreciated, given that
kedro install
currently supports installing from
src/environment.yml
?
d
Hi @User like @User suggests it's there because our research found that people found
kedro install
confusing - it did different things if you had run
kedro build-reqs
beforehand or not and it wasn't well explained. In the next major version
build-reqs
(our pip-tools powered dependency resolver) will remain - but you are free to use whatever installer you want on the (pip/conda/poetry/etc) on the compiled requirements file
b
Makes sense, thanks for clearing that up!
@User Did some testing there, and found the following: PyPI and Conda have different naming conventions between some packages (
jupyter-client
vs
jupyter_client
, etc), meaning that the resulting
requirements.txt
from
kedro build-reqs
almost certainly can't be installed by Conda. Of course, one can manually manage a
requirements.txt
or
environment.yml
without any help from Kedro that can be parsed by Conda, but you'll run into more problems with
setup.py
as
install_requires
will parse the
.txt
and search PyPI for the contents, resulting in the same problem. As of right now, it seems like doing everything with
pip
is the "correct" way (I don't know anything about poetry), or at least the path of least resistance if you want to work with the existing starters out-of-the-box.
c
It's totally opinionated but the best way to run with env in python is to use conda for python only and pip/poetry for installing package. This is what i'm doing for some months right now and that's really nice. You have the advantage of running conda by creating isolated python version X as you want without chnaging your local version or even installing one version of python if you don't have one by default on your OS. And you take advantage of poetry (or at least pip) to handle packages and dependencies.
This is the best combo to avoid conda weird dependencies check/install/resolve and time lost by waiting conda to resolve things + you just need to poetry install or pip install you requirements file whenever you need to update package inside the conda env.
I use pip with kedro, and poetry outside when it's a project without kedro or I initialize myself without any requirements file created
It resolves also a lot of boring time dealing with conda packages not updated like pip ones
b
Much obliged @User . I'll try and work your way for the next few projects and see how things go. Mamba is really good for an instant 4x improvement in resolve time if you're ever forced to use conda to manage packages. Even if you just like to manage envs with conda, I couldn't recommend mambaforge / micromamba more highly.
d
That's super cool to see - as a always we'd love to see a 'Show and tell' github discussions whenever users discover something neat!
In case useful there is a conda related discussion going on here https://github.com/kedro-org/kedro/discussions/1220