message has been deleted

I changed the name of file from kedro.py to kedro_hello.py now 'nomodule' error is gone. but new error occurred.

searched and found this https://github.com/kedro-org/kedro/issues/1409

I see, if you name your file as

kedro.py

Python is confused that this is the kedro module instead of the installed one.

We have a work in progress to fix this. https://github.com/kedro-org/kedro/issues/1429 For now, please try to follow the suggested solution in the Github Issue that you found.

Hello folks. I've been using kedro for a while now. But not too much using it in the interactive mode. Now, is there a better way of computing the nodes that reach a memory dataset, and then retrieving the memory dataset, without changing the catalog so that it is written somewhere?

You could try running a partial pipeline until the memory datasets you're interested in jupyter using

session.run

with

to_outputs

arg. But this will only return a dictionary of your datasets, not as memory datasets retrievable in catalog. Reference to

session.run

function specs: https://kedro.readthedocs.io/en/stable/kedro.framework.session.session.KedroSession.html#kedro.framework.session.session.KedroSession.run

Can you explain what you are trying to achieve here?

The use case is simple. I want to run a pipeline upto a dateset, but I don't want to write it anywhere. The

session.run

only returns an empty dict, and from what I understand, only datasets with some catalog issues are returned. from the session.run docstring:

Returns:
    Any node outputs that cannot be processed by the ``DataCatalog``.
    These are returned in a dictionary, where the keys are defined
    by the node outputs.

Unless I'm using the catalog wrong

You'll have to add the list of datasets you want to access in the

to_outputs

arg for it to return them in the dictionary. Also what noklam might be looking to know the reason you're trying to do this.

I've been running it as :

session.run(to_outputs=['my_dataset'])

And the return value is an empty dict. The pipeline runs smoothly, and everything is defined in the catalog.

My use case in debugging actually. There is something wrong with a node somewhere, and I'm bratracking datasets to see where the issue lies.

In that case I would recommend breakpoints as a much better way of backtracking

yep, this is what I've been doing 😄 But I'm just after loading pretty much all datasets before to see at least in what region of the pipeline stuff changed.

I have a local development catalog/env that saves everything on disk, but some intermediary steps not.

So I was wondering if there was a way to reach those without having to write them to disk.

One other case that might not work is when debugging in production/remote systems. I had to do similar console runs to understand what was going wrong.

Huum, the system is running locally, but I prefer using ipython in the console rather then jupyter lab/notebook. Could that be an issue?

By default when you run a pipeline, any free output will be return in a dictionary.

At first, I thought the

session.run

idea was this: Have a generic function that can run nodes, pipelines, everything needed for a set of inputs and/or a set of outputs, and kedro would take care of running everything. But the return values are just for those that have a catalog issue, and there is no way to access the catalog for those runs (I think) . So I'm always lost on how to use the

session.run

, specifically due to the fact that I can't reach those memory datasets interactively unless I'm always persisting everything in the catalog.

Huuum. What do you mean by default 😄 😅?

I quickly checked a few things, looks like if it's being persisted/present in catalog it is available in the catalog object and if it's a free dataset(not present in catalog yml ) it is returned in dictionary

I mean

result  = session.run()

, the result will store any free output in the pipeline.

As @avan-sh said.

FYI this is just the sort of thing (jumping into a certain point of a pipeline in jupyter) that we're hoping to make easier in future. I think this sort of debugging is a pretty common thing to do. https://github.com/kedro-org/kedro/issues/1075

nice. I'll check out the PR.

But, just my 2 cents here, the

session.run

api seems to match perfectly the use cases, if it returned all the datasets data are a result from a node/pipeline/set of inputs or a list of the wanted outputs. Now,

session.run

shouldn't change, obviously, but the api and the arguments in the signature seem very ergonomic to me.

Do you mean it should just return every datasets instead of the free outputs only (current implementation)?