Any idea what type of object it is when it fails?
# beginners-need-help
d
Any idea what type of object it is when it fails?
r
Even with just the first node it fails. The first node has pandas parquet as inputs. I haven't created any custom datasets in any pipeline.
d
That's super odd
if not sensitive, can you push to repo for me to review?
(tomorrow :P)
r
Unfortunately, it is sensitive. I'll see if I can change it to a shareable version. It might take a few days though.
d
another alternative would be to list any ways we deviate from a standard kedro project
d
@datajoely I have faced the same issue while persisting Pandas dataframe as pickle files in the catalog.yml file
r
To replicate the error, just insert a node with
func=lambda x: x
in a pipeline and run that pipeline on ParallelRunner. I used this example (https://github.com/datajoely/modular-spaceflights/tree/main/src/modular_spaceflights/pipelines/data_ingestion) and added a dummy
lambda
node in the "Data ingestion" pipeline.
d
Hi RRoger - I think we've worked out the issue
It's related to this change from 0.17.6 we need to tweak the pickledataset implementation, you can roll back to 0.17.5 or avoid using PickleDataSet with ParallelRunner
4 Views