I wonder if it's worth defining a custom dataset that works on ranges

and then you run the several pipelines for each range

Is the error you're worried about related to logic or memory?

because I guess we could take diff approaches depending on those variables

I'm anticipating either of those error types. It's gravitational wave data, so big files and the signal can be inconsistent at times, so just trying make a plan for things going wrong

In general I'd explore two things: (1) Get good at logging out exceptions/warnings things that can be used to understand why things failed. Potentially the

on_pipeline_error

hook is really useful here (2) Try and set up a data profiling pipeline that allows you to do low-cost analysis of the data that can maybe give you an idea of what may cause problems downstream (3) for logic errors, you may want to use exception handlers to gracefully swallow and log errors rather than killing the process