For example, say your file is called data.csv If you turn on versioning, it will be data.csv/timestamp/data.csv instead

Also where is it fetching the time stamp from?

The timestamp will be a default timestamp, each time you run kedro pipeline it generates a timestamp

You just need to remove the existing data.cvs which cause name collision

So if I make no change to my code, the node I run will have a name collision with versioning on. But if I change the code, the version time stamp changes?

Not sure if I am following, what code change are you referring?

Ok…let me back up. So I have a node that I’ve made some changes to. I save those and have versioning on. When I run the node the first time it runs fine. But if I make no change and rerun it, I get the error.

Perhaps node is incorrect as well. I have a segment of the pipeline I’m running using to_nodes

So this is probably because when you run the pipeline first time, you have no version on. It generates a file called path/filename

That’s true I didn’t. I enabled it after the fact

And now u turn on versioning, it tries to generate a path called path/filename/timestamp/filename, but the existing file collide with the existing file

The filesystem does not allow the same name being reused

But the timestamp should change with each run right?

So simply remove the existing file will make it runs again

Yes

So I need to remove anything that is outside of the timestamp(s) directory?

What file are there now?

Do a ls or screenshot will help

So I have -data.parquet -data.parquet/

And there are timestamps below the data.parquet/

Can you also post the error you get?

It’s from work so let me strip out the sensitive stuff

@noklam DataSetError: Save path

s3bucketname/project_folder/data/layer/data.parquet/2022-04-18T21.43.00.910Z/data.parquet

for ParquetDataSet(filepath=s3bucketname/project_folder/data/layer/data.parquet, load_args={}, protocol=s3, save_args={}, version=Version(load=None, save='2022-04-18T21.43.00.910Z')) must not exist if versioning is enabled.

That's a pretty good approximation with some of the path stuff abstracted

The thing is...shouldn't the timestamp advance and automatically take care of the error?

Is there now a file (not a dir) called

s3bucketname/project_folder/data/layer/data.parquet

?

yes

s3bucketname/project_folder/data/layer/data.parquet/

can you try removing the file? If version is turned on it should always be

filename/timestamp/filename

that structure exists