So also remember that spark is doing its own distribution optimisation in the background - we're telling it which chunks it can run once isolated branches have completed, but it will be also doing it's own resource management behind the scenes. Another angle is to maybe look at optimising the spark cluster or digging into the query plans of some of the long running processes.