When I run kedro with

kedro run

my first pipeline data_processing has a node which outputs prokka_bins. This data is one of the outputs of the pipeline data_processing.

namespace="data_processing", 
inputs=["partition_prokka_faa", "partition_prokka_gff"], 
outputs=["hypo_prot","prokka_bins"]

I was wondering how I can access the data from my new pipeline

edge_processing

inputs="prokka_bins"

when this code is within a node in edge_processing, I get the error.

ValueError: Pipeline input(s) {'edge_processing.prokka_bins'} not found in the DataCatalog

Other than essentially duplicating an entry in my DataCatalog with

edge_processing.prokka_bins

which makes things more messy, is there any way I can access this data in my second pipeline?

This feels like a namespace issue

Hello, are there any publications/papers describing the core architecture of kedro and the main principles? And If so, where can I find them?

https://kedro.readthedocs.io/en/stable/faq/architecture_overview.html https://kedro.readthedocs.io/en/stable/faq/kedro_principles.html I don't think we have a sophisticated paper for that, these docs may covers part of it.

If you are new to kedro, I suggest just try going through the tutorials and come back to the individual component

@noklam Case solved. Documentation bug: https://kedro.readthedocs.io/en/stable/tutorial/set_up_experiment_tracking.html#set-up-the-session-store ...says to modify

./src/settings.py

but it should be

./src/{project name}/settings.py

Nice catch! The settings file should be generated from the template though, are you migrating from some older project?

Otherwise I can take care of it.

No, this was starting fresh. I just started editing the file name. Also didn't help that the addition looks like a complete file (imports and all) so it seemed less surprising that it was an empty file.

Sorry about the confusion, glad you have found out the source of error at the end.

Hi all, I have created a namespace modular pipeline and specified some outputs which are consumed by other pipelines/nodes. My problem is that this output is also used within the namespace pipeline as an input to other internal nodes. As a result, when I look at kedro-viz, the output dataset is disjointed and not showing as an output of the namespace pipeline. Is this a bug, feature or am I doing something wrong here? Images attached illustrate my point.

So I think you can fix this with nested namespaces. I.e. put everything in one namespace, then your two sub parts in namespace.one and namespace.two

I have just tried this an unfortunately this does not seem to help.

Hmm this is odd - what version of kedro-viz are you running?

I'm on 4.7.1

Hmm is your project sensitive or could you put it on a repo for me to have a look?

Thanks for your help, mystery solved!

@Spikeyhog or @datajoely can you share what the solution ended up being? I've had this issue as well but just accepted the floating nodes as a feature.

Its about name spacing your datasets that live on the boundary between pipelines in the catalog

I cant share the repo but it's easy to debug with

kedro viz --autoreload

Can you share what the viz ends up looking like for this example? Do the floating "competitions" and "games" datasets disappear and stay inside "Statsbomb Open Data", or do new arrows appear pointing to them outside "Statsbomb Open Data"?

Hello... I am trying to figure out two things because I simply do not know where to find some sample code... 1. I want to train n times and choose best model, encoder etc. Is it better to do this with Kedro or with Kedro Mlflow? I understand Kedro has versioning but Kedro Mlflow offers more advanced tracking options. 2. I don't really get it how will Kedro work deployed on single server. I mean it will be deployed on a server with no Internet connection as a package but I want to be able to run all project or only parts of a project. If I have a good model I might just want to predict. How can I achieve this with a single package?

The solution results in the former when the named pipelines are collapsed, so the floating nodes does indeed seem to be a feature. This also happens in the demo project.

Hello, is there a good way to access context.params while defining a pipeline?

Do you mean within a node?

If you do it's the

params

syntax in the inputs. If you mean a pipeline then the answer is sort of, you can use hooks for this.

Yes, in a pipeline. My current approach is to use ConfigLoader and get project path from project_module.__file__. There used to be a get_current_session but seems like it was removed in 0.18.

can you not use a hook for this?

Thanks for the suggestion. I’m not very familiar with hooks. Still not sure how to go about it, but I’ll give the docs a read.

My rule is if you're accessing the context directly you've gone too far! Hooks usually do what you need