How to manage large partitioned datasets