Title
#beginners-need-help
waylonwalker

waylonwalker

08/05/2021, 3:42 AM
Would it be appropriate to store those intermediate steps to disk next to the notebook? You could make your own function for running that asks the section of pipeline you are running for its
all_outptuts
then make pickle or parquet datasets for each of them on the fly. They won't exist in the project's catalog but only in memory. You might even be able to use the
tempfile
module to let the os clean up for you after your done.