https://kedro.org/ logo
#beginners-need-help
Title
# beginners-need-help
d

datajoely

01/19/2022, 10:09 PM
Any idea what type of object it is when it fails?
r

Rroger

01/19/2022, 10:39 PM
Even with just the first node it fails. The first node has pandas parquet as inputs. I haven't created any custom datasets in any pipeline.
d

datajoely

01/19/2022, 10:39 PM
That's super odd
if not sensitive, can you push to repo for me to review?
(tomorrow :P)
r

Rroger

01/19/2022, 10:58 PM
Unfortunately, it is sensitive. I'll see if I can change it to a shareable version. It might take a few days though.
d

datajoely

01/20/2022, 10:32 AM
another alternative would be to list any ways we deviate from a standard kedro project
d

Dhaval

01/22/2022, 9:55 AM
@datajoely I have faced the same issue while persisting Pandas dataframe as pickle files in the catalog.yml file
r

RRoger

01/29/2022, 11:27 AM
To replicate the error, just insert a node with
func=lambda x: x
in a pipeline and run that pipeline on ParallelRunner. I used this example (https://github.com/datajoely/modular-spaceflights/tree/main/src/modular_spaceflights/pipelines/data_ingestion) and added a dummy
lambda
node in the "Data ingestion" pipeline.
d

datajoely

01/31/2022, 9:37 AM
Hi RRoger - I think we've worked out the issue
It's related to this change from 0.17.6 we need to tweak the pickledataset implementation, you can roll back to 0.17.5 or avoid using PickleDataSet with ParallelRunner
4 Views