But the timestamp should change with each run right?

So simply remove the existing file will make it runs again

Yes

So I need to remove anything that is outside of the timestamp(s) directory?

What file are there now?

Do a ls or screenshot will help

So I have -data.parquet -data.parquet/

And there are timestamps below the data.parquet/

Can you also post the error you get?

It’s from work so let me strip out the sensitive stuff

@noklam DataSetError: Save path

s3bucketname/project_folder/data/layer/data.parquet/2022-04-18T21.43.00.910Z/data.parquet

for ParquetDataSet(filepath=s3bucketname/project_folder/data/layer/data.parquet, load_args={}, protocol=s3, save_args={}, version=Version(load=None, save='2022-04-18T21.43.00.910Z')) must not exist if versioning is enabled.

That's a pretty good approximation with some of the path stuff abstracted

The thing is...shouldn't the timestamp advance and automatically take care of the error?

Is there now a file (not a dir) called

s3bucketname/project_folder/data/layer/data.parquet

?

yes

s3bucketname/project_folder/data/layer/data.parquet/

can you try removing the file? If version is turned on it should always be

filename/timestamp/filename

that structure exists

if you go further it becomes

s3bucketname/project_folder/data/layer/data.parquet/timestamp/data.parquet

I changed the name of file from kedro.py to kedro_hello.py now 'nomodule' error is gone. but new error occurred.

searched and found this https://github.com/kedro-org/kedro/issues/1409

I see, if you name your file as

kedro.py

Python is confused that this is the kedro module instead of the installed one.

We have a work in progress to fix this. https://github.com/kedro-org/kedro/issues/1429 For now, please try to follow the suggested solution in the Github Issue that you found.

Hello folks. I've been using kedro for a while now. But not too much using it in the interactive mode. Now, is there a better way of computing the nodes that reach a memory dataset, and then retrieving the memory dataset, without changing the catalog so that it is written somewhere?

You could try running a partial pipeline until the memory datasets you're interested in jupyter using

session.run

with

to_outputs

arg. But this will only return a dictionary of your datasets, not as memory datasets retrievable in catalog. Reference to

session.run

function specs: https://kedro.readthedocs.io/en/stable/kedro.framework.session.session.KedroSession.html#kedro.framework.session.session.KedroSession.run

Can you explain what you are trying to achieve here?

The use case is simple. I want to run a pipeline upto a dateset, but I don't want to write it anywhere. The

session.run

only returns an empty dict, and from what I understand, only datasets with some catalog issues are returned. from the session.run docstring:

Returns:
    Any node outputs that cannot be processed by the ``DataCatalog``.
    These are returned in a dictionary, where the keys are defined
    by the node outputs.

Unless I'm using the catalog wrong

You'll have to add the list of datasets you want to access in the

to_outputs

arg for it to return them in the dictionary. Also what noklam might be looking to know the reason you're trying to do this.

I've been running it as :

session.run(to_outputs=['my_dataset'])

And the return value is an empty dict. The pipeline runs smoothly, and everything is defined in the catalog.

My use case in debugging actually. There is something wrong with a node somewhere, and I'm bratracking datasets to see where the issue lies.