antony.milne11/10/2021, 2:29 PM
explains a bit what's going on here:
The second part about memory datasets is what's relevant here. As Joel said, default for parallel runner is that
Ensure that all data sets are serializable and that we do not have non proxied memory data sets being used as outputs as their content not be synchronized across threads.
is used rather than
for where this happens). In theory you could specify this dataset type explicitly in the catalog, but the fact that it's private means that's probably not a good idea, and I've never seen anyone do so. Just don't define them in the catalog and they will default to
and everything should work ok 🙂