Would it be appropriate to store those intermediate steps to disk next to the notebook? You could make your own function for running that asks the section of pipeline you are running for its

all_outptuts

then make pickle or parquet datasets for each of them on the fly. They won't exist in the project's catalog but only in memory. You might even be able to use the

tempfile

module to let the os clean up for you after your done.