I get the error: "Could not find a version that matches ipython==7.10,>=4.0.0,>=7.23.1 (from -r /Users/robertson/dev/kedro-tutorial/src/requirements.in (line 3))"

It seems there's a direct written requirement in the requirements.in file that "ipython==7.10," but because "jupiter==1.0," this causes a need for "ipykernel==6.0.0," which in turn causes "ipython>=7.23.1," which is obviously incompatible with "ipython==7.10"

I would ignore it normally, but I thought might have been causing some issues further down the line when kedro ipython started with errors and couldn't build the .yaml files

Thanks! I guessed it. I'll give it a try 🙂

Hey @robertson809 are you working in a virtual environment?

hi again! Imagine I'm working in a project in a dedicated venv with all of the packages needed installed. For some reason I started the project out of kedro and now I want to work in my project inside kedro (I mean using kedro as a ML pipeline generator, etc). Is there a way to transfer all the packages (with version) installed out of kedro (in my venv) to my new kedro project (I guess to the requirements.txt file)? Type them in one by one could be very time-consuming!

Hi @User if the only issues are

ipython

I think you'll be fine living with this error. In general it's easiest (for me at least) to start from a fresh venv each time so I can manage my mental model of which dependencies are installed.

So do you recommend me to create a new venv, install kedro, create a new kedro project and install all of the packages needed inside the kedro project modifying the requirements.txt file and running 'kedro install'? I think it would be nice to add in a future release the functionality of transferring the packages installed in the venv to the kedro project without having to write them one by one in the requirements.txt file, in order to manage in a quick way the scenario where you start prototyping your solution outside of kedro and at some point you want to move to kedro.

Yeah this is more of a Python best practice that a Kedro one - the short answer is Python dependency management is a pain in general

@User feel free to unpin

ipython

to sth like ~=7.23.

ipykernel

has released a new version a couple of days ago which is why some of the dependencies are unhappy. We fixed it on master but it's not been released yet ofc. https://github.com/quantumblacklabs/kedro/commit/63f7c34f5826ae6e02b037115f954d2a9cb9f42b

And the last question about this issue 😅 The packages installed via 'kedro install' are not installed in the venv where kedro exists, right?

@User if you are in an active virtual environment it should show up inside the parentheses to the left

pip freeze

will show you what's already in there

and

kedro install

will install to the python environment you currently have activated

you can prove this by checking

which python

you are using

@User pointed me in the direction of this talk a while back that explained how things work and specifically why

conda environments

work well for DS workflows

Thank you very much for your help and quick response!

No worries! Good luck!

Yes, I was, but I think @Lorena gave me a good solution to the problem.

Got it! Thanks for the answer!

@User Have you tried

pipx

?? I prefer it for running kedro new over anything else. I think it solves this chicken and egg situation quite well. And avoids the issue of having a stale system kedro install giving you a different template. I have some more details in this issue here https://github.com/quantumblacklabs/kedro/issues/753

# use the latest template
pipx run kedro new

# use a git hash/branch/tag
pipx run --spec git+https://github.com/quantumblacklabs/kedro.git@0.16.1 kedro new

# use a specific version out of pypi
pipx run --spec kedro==0.16.1 kedro new

I haven't used it personally - I think it makes sense for prototyping, but I'm so familiar with the

venv

pattern it would be hard to unlearn it!

You still use venv after you have your template.

Where do you put your venv if its not inside the project?

oh I see! I thought it was more of an ephemeral env

Personally my venv lives in my conda global dir - but if we were going towards production I'd push towards dockerising my pipeline and achieve isolation that way

It's a guaranteed way to get the version you want or the latest, but not a randomly stale version. personally I think this should be in the project readme, as it is repeatable.

I see, I use conda as well which is global. I have a direnv script that creates it all for me based on the directory name so its easier to do if I create after, which is what you would do if you were using venv as well.

makes sense

dependencies in Python are a mess as @User and @User will attest!