So it would be great to get a stack trace to understand what's going on. Could you try wrapping your

ParquetDataSet

in a

PartitionedDataSet

would allow you to write smaller chunks and won't fail