08/05/2021, 2:35 AM
@User , I am not going out and creating a static dataframe and passing it as a MemoryDatasets. However, we have lots of pipelines inside a project, each with a ton of nodes which aggregate data. Its not uncommon for us to have 6 different aggregation nodes, which then only get passed into further aggregation nodes down stream. Thus there is never a need to save the output of the first aggregation nodes to a storage location. I am just assuming that the passing of the data without it being saved between nodes is being kept in a MemoryDataset until it is used by later nodes. Does that make sense? I do know that I cannot call these unsaved datasets via the catalog methods you mentioned above (at least not that I have found yet).