Hello you may be able to use

PartitionedDataSet

to do this with pandas. Spark will do this automatically.