SOLVED: I tried to save multiple files to the same folder.
Hello all. I'm trying to upgrade Kedro from 0.15.9 all the way to 0.17.5. I'm having issues saving parquet files in S3. I think it has to do with a "S3 metadata eventual consistency" issue. "Caused by: java.io.FileNotFoundException: No such file or directory: s3a://kedrobucket/supply_chain_data_asset_matt/data/03_primary/api/part-00000-09420dd5-677d-421f-9b15-555b2d648c05-c000.snappy.parquet
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved."
I'm going package by package to figure out what is causing the issue. Anyone else ran into this?
in in my requirements.in. But
ImportError: cannot import name 'get_installed_distributions'
What is the latest version of pip supported by kedro==0.17.5