In that case I would recommend breakpoints as a much better way of backtracking

yep, this is what I've been doing 😄 But I'm just after loading pretty much all datasets before to see at least in what region of the pipeline stuff changed.

I have a local development catalog/env that saves everything on disk, but some intermediary steps not.

So I was wondering if there was a way to reach those without having to write them to disk.

One other case that might not work is when debugging in production/remote systems. I had to do similar console runs to understand what was going wrong.

Huum, the system is running locally, but I prefer using ipython in the console rather then jupyter lab/notebook. Could that be an issue?

By default when you run a pipeline, any free output will be return in a dictionary.

At first, I thought the

session.run

idea was this: Have a generic function that can run nodes, pipelines, everything needed for a set of inputs and/or a set of outputs, and kedro would take care of running everything. But the return values are just for those that have a catalog issue, and there is no way to access the catalog for those runs (I think) . So I'm always lost on how to use the

session.run

, specifically due to the fact that I can't reach those memory datasets interactively unless I'm always persisting everything in the catalog.

Huuum. What do you mean by default 😄 😅?

I quickly checked a few things, looks like if it's being persisted/present in catalog it is available in the catalog object and if it's a free dataset(not present in catalog yml ) it is returned in dictionary

I mean

result  = session.run()

, the result will store any free output in the pipeline.

As @avan-sh said.

FYI this is just the sort of thing (jumping into a certain point of a pipeline in jupyter) that we're hoping to make easier in future. I think this sort of debugging is a pretty common thing to do. https://github.com/kedro-org/kedro/issues/1075

nice. I'll check out the PR.

But, just my 2 cents here, the

session.run

api seems to match perfectly the use cases, if it returned all the datasets data are a result from a node/pipeline/set of inputs or a list of the wanted outputs. Now,

session.run

shouldn't change, obviously, but the api and the arguments in the signature seem very ergonomic to me.

Do you mean it should just return every datasets instead of the free outputs only (current implementation)?

yep. I mean, viewing the

session.run

as a helper for interactive inspection and development when using outputs from other nodes/pipelines.

session.run

is actually the core API for kedro's pipeline now. The use case of debugging is a valid one, especially without the access of a debugger. The current concept of "free outputs" is essentially pipeline's output (node's output that has no consumer, thus intermediate output are not included) - set of catalog entries (anything defined in the catalog). From my experience, this is not convenient as the intermediate output could be useful for debugging, but I think the main reason is that kedro tries to be memory efficient. I think it's one area we could improve on interactive workflow.

Hey All, For one of my use-case, I have to add a node that creates histograms for each pipeline run on a dataset and then second node stitch existing stitched-histogram(from previous run) with new histogram created from previous node something like

hist_train_candidates = node(
        func=monitor_training_cand,
        inputs="training_candidates",
        outputs="training_candidates_hist",
        name="hist_train_candidates_node")

    stitch_train_candidates = node(
        func=stitch_training_cand,
        inputs=["training_candidates_hist", "stitched_hist"],
        outputs="stitched_hist",
        name="stitch_train_candidates_node")

    create_report = node(
        func=create_report_cand,
        inputs="stitch_train_candidates",
        outputs="stitch_report",
        name="create_report_node")

Having output same as your input isn't supported, but I do need it for my usecase. Plus I have to create custom versioned dataset for stitch_hist which is leading to(after applying a hack of different name catalog entry but points to same file location ) *`raise VersionNotFoundError(f"Did not find any versions for {self}") kedro.io.core.VersionNotFoundError: Did not find any versions for HistogramDataSet(filepath=/Users/Project/data/08_reporting/stitch_train_candidates.json, protocol=file, version=Version(load=None, save='2022-04-21T12.17.11.537Z'))`* Any suggestion on how to better handle this scenario?

You can't have the same input as an output, the DAG needs to be acyclic by definition. What you can do is define two datasets that point to the same path

Then in this scenario, the dataset that I am using a custom versioned dataset and for the very first run of node

stitch_train_candidates = node(
        func=stitch_training_cand,
        inputs=["training_candidates_hist", "stitched_hist"],
        outputs="stitch_train_candidates",
        name="stitch_train_candidates_node")

stitched_hist isn't available and I am getting this error

raise VersionNotFoundError(f"Did not find any versions for {self}")
kedro.io.core.VersionNotFoundError: Did not find any versions for HistogramDataSet(filepath=/Users/Project/data/08_reporting/stitch_train_candidates.json, protocol=file, version=Version(load=None, save='2022-04-21T12.17.11.537Z'))

How can i fix that?

If I'm reading this correctly - you've never written any data to that location so it can't read any versions?

yes, so I don't understand why I am getting this. something wrong with how i am creating this versioned dataset

class HistogramDataSet(AbstractVersionedDataSet):

    def __init__(self, filepath: str, version: Version = None, credentials: Dict[str, Any] = None):
        _credentials = deepcopy(credentials) or {}
        protocol, path = get_protocol_and_path(filepath)
        self._protocol = protocol
        self._fs = fsspec.filesystem(self._protocol, **_credentials)
        super().__init__(
            filepath=PurePosixPath(path),
            version=version,
            exists_function=self._fs.exists,
            glob_function=self._fs.glob, )

    def _load(self):
        load_path = get_filepath_str(self._filepath, self._protocol)
        log.info(f'load_path: {load_path}')
        try:
            with self._fs.open(load_path) as f:
                return json.load(f)
        except FileNotFoundError:
            return None

    def _save(self, data) -> None:
        """Saves data to the specified filepath."""
        save_path = get_filepath_str(self._filepath, self._protocol)
        with self._fs.open(save_path, mode="w") as f:
            json.dump(data, f, default=dumper)
        self._invalidate_cache()

for versioning I am using kedro functionality

What you're doing is ok as far I see, but you might need to catch

VersionNotFoundError

instead of

FileNotFoundError

.

Hi everyone, I've been using kedro for a little while , and followed EngineerOne videos , i had a question about programatically adding datasets, for one of my projects i have a hierarchy of files that is growing in a structured way where i have recordings that are being added for new subjects (data/sub01/recordings.txt) , in one of the videos of Dataengineerone he does so by changing the ProjectContext class in the run.py file , but it seems that in the recents version this file is no more. I wanna be able to create the datasets automatically from params , and create corresponding nodes from params, I was thinking of 4 solutions: 1-Find the Equivalent of the ProjectContextClass, iv'e been wondering if this class is still used in a new file or if there is the equivalent in the new version of kedro 2-Use jinja2 in the catalog, if i use jinja2 i've been wondering then how i could load the parameters for iterating over them and creating the catalog entries, 3-Create a custom class, but then i've been wondering how to return a dictionary of callables like the partitionDataset does, 4- Use hooks as proposed in a past question, but sincerely i still never used them, which solution is the best ? or is there another simpler one ?

dynamic datasets

Hi, everybody! I came back to the project I set up last week, going through the Kedro tutorial, which was working just fine. When I try to execute

kedro run

, I get the following ValueError:

ValueError: Pipeline input(s) {'params:data_science.model_options_experimental', 'params:data_science.active_modelling_pipeline.model_options'} not found in the DataCatalog

What does your parameters.yml look like?

I have not been able to figure out what changed in the meanwhile. Any ideas are more than welcome.

This did change between 0.17.7 and 0.18.0 it's in the release notes