https://kedro.org/ logo
#beginners-need-help
Title
# beginners-need-help
l

limdauto

01/26/2022, 10:47 PM
You can emulate this with
after_node_run
hook: https://kedro.readthedocs.io/en/stable/kedro.framework.hooks.specs.NodeSpecs.html#kedro.framework.hooks.specs.NodeSpecs.after_node_run -- use MemoryDataSet as data flow in Kedro but save to disk in after_node_run hook as backup. I think you can also do this async (eg in another process) so you don't block the pipeline execution.