Catalog loading by directory

Hi! I'm running a kedro project inside a Docker container (previously I've created the DockerFile of the project and built the image) just for experimenting. My project is using mlflow plugin and the thing is, when I run 'kedro run' inside the docker container I get the following error. All previous nodes run well but the execution stops once it reaches the node where the mlflow artifacts are saved

It seems like it is trying to access to the C:/ disk, but I dont know why and where it is configured to act like this.

BTW, this is my catalog:

SOLVED! I've mounted the image with data of previous experiments in the /data directory. It seems to cause the issue. If the /data directory is empty of files (except the files of the 01_raw directory) it runs with no problems.

Amazing 🎉

well done!

I am playing with

kedro pipeline package/pull

. TLDR, how do I update pulled pipelines?

kedro new # proj1
kedro new # proj2

cd proj1
kedro pipeline create newpipe
# make some nodes
kedro pipeline package newpipe
cd src/dist
python -m http.server
# hosting dist directory at 8000

# new terminal
cd proj2
kedro pipeline pull http://localhost:8000/newpipe-0.1-py3-none-any.whl

At this point everything is working. My question is how do I update this pipeline

cd proj1
# update newpipe code
# update version in __init__.py
kedro pipeline package newpipe

cd proj2
kedro pipeline pull http://localhost:8000/newpipe-0.2-py3-none-any.whl

❯ kedro pipeline pull http://localhost:8000/newpipe-0.2-py3-none-any.whl

2021-08-28 12:11:02,473 - root - INFO - Registered CLI hooks from 1 installed plugin(s): kedro-telemetry-0.1.2
2021-08-28 12:11:02,476 - kedro_telemetry.plugin - INFO - You have opted into product usage analytics.
HTTPFileSystem requires "requests" and "aiohttp" to be installed
Trying to use 'pip download'...
/proj2/.venv/bin/python -m pip download --no-deps --dest /tmp/tmpucm6o5r2 http://localhost:8000/newpipe-0.2-py3-none-any.whl
Collecting newpipe==0.2
  Downloading http://localhost:8000/newpipe-0.2-py3-none-any.whl (6.0 kB)
Saved /tmp/tmpucm6o5r2/newpipe-0.2-py3-none-any.whl
Successfully downloaded newpipe
Creating `/proj2/conf/base/parameters`:
  Creating `/proj2/conf/base/parameters/newpipe.yml`: SKIPPED (already exists)
Creating `/proj2/src/tests/pipelines/newpipe/__init__.py`: SKIPPED (already exists)
Creating `/proj2/src/tests/pipelines/newpipe/test_pipeline.py`: SKIPPED (already exists)
Creating `/proj2/src/proj2/pipelines/newpipe/nodes.py`: SKIPPED (already exists)
Creating `/proj2/src/proj2/pipelines/newpipe/README.md`: SKIPPED (already exists)
Creating `/proj2/src/proj2/pipelines/newpipe/__init__.py`: SKIPPED (already exists)
Creating `/proj2/src/proj2/pipelines/newpipe/pipeline.py`: SKIPPED (already exists)

you need to delete the files you want to update before running

kedro pipeline pull

, @User

as they are skipped by default

I think the rationale behind this is to not delete additional code you may have written

Hello guys! I introduced Kedro to a coworker and when he was testing, the command "kedro new --starter=pandas-iris" was not working. I also tried and got the same erros as it follows:

Hey @User is Git installed?

And googling suggests that the certificates on your (corporate?) machines may be funky https://stackoverflow.com/questions/46887802/exit-status-128-while-cloning-the-git-repository-while-getting-go-package-from-g

Is it generally recommended to group multiple different pipelines in the same repo ?

Yes most projects use a monorepo set up. However our modular pipeline pattern allows you to package/push/pull pipelines between projects. https://kedro.readthedocs.io/en/stable/06_nodes_and_pipelines/03_modular_pipelines.html

Hello everyone. I'm having a problem saving a versioned spark.SparkDataSet. I'm saving a new dataset and receive the error message "kedro.io.core.VersionNotFoundError: Did not find any versions for SparkDataSet". (The code works if I change the type for pandas with a small part of the data) Does someone already had the same issue or know possible solutions?

@User So this error is returned if Kedro can't find an older version of dataset

Some questions (1) Are you able to write not read (2) Does it work with versioning disabled?

(1) I'm writing the dataset not reading (2) works with versioning disable

I didn't expect this error in a write operation. I already use other versioned datasets in the project, but this is the first SparkDataSet.

Yeah it's a little weird

The other are pickle

Is there any issue if we clear the directory created by the SparkVersionedDataset?

No problem

then can we wipe that directory and see if that works?

Yes. I already wipe the directory and run with and without the versioned dataset. (1) With the versioned Spark Data Set, create the timestamp folder and written data on the folder but result in the exception (2) Without a versioned Spark Data Set, everything works as expected Versioned dataset: Alsformattedmodeloutput

Okay thank you @User let me ask one of the devs - thanks for checking

Thanks @User - How about within a monorepo.. would you say it makes sense to have separate projects within a monorepo, or pile all the pipelines together into a single project?

I really it think it depends on how much logic/shared catalog exists between the pipelines. I think I would err on one repo per Kedro project, but utilise multiple pipelines and configuration environments within that project.