antheas
08/30/2022, 6:35 AMazrael
09/01/2022, 3:11 PMwilliamc
09/01/2022, 9:25 PMA node cannot have the same inputs and outputs
error, or would you suggest a different approach? Thanks!datajoely
09/02/2022, 7:30 AMdatajoely
09/02/2022, 7:31 AMPetitLepton
09/04/2022, 9:09 AMwilliamc
09/04/2022, 9:52 PMwilliamc
09/04/2022, 9:52 PMvenncit
09/07/2022, 8:41 AMnoklam
09/07/2022, 10:51 PMnoklam
09/07/2022, 10:52 PMvenncit
09/08/2022, 7:50 AMdatajoely
09/08/2022, 8:03 AMToniMaroni
09/09/2022, 8:41 AMnoklam
09/09/2022, 12:43 PMuser
09/14/2022, 5:51 AMCarlo Gonzalez
09/14/2022, 2:53 PMdatajoely
09/14/2022, 2:54 PMCarlo Gonzalez
09/14/2022, 3:02 PMdatajoely
09/14/2022, 3:03 PMdatajoely
09/14/2022, 3:03 PMCarlo Gonzalez
09/14/2022, 3:04 PMuser
09/17/2022, 11:02 AMCarlo Gonzalez
09/23/2022, 3:53 PMdatajoely
09/23/2022, 3:54 PMdatajoely
09/23/2022, 3:56 PMCarlo Gonzalez
09/23/2022, 3:58 PMrafael.gildin
09/23/2022, 6:43 PMantheas
09/24/2022, 11:25 AMantheas
09/24/2022, 11:42 AMantheas
09/24/2022, 11:42 AMrafael.gildin
09/27/2022, 2:54 PMclass CustomSequentialRunner2(SequentialRunner):
def _run(
self,
pipeline: Pipeline,
catalog: DataCatalog,
hook_manager: PluginManager,
session_id: str = None,
) -> None:
nodes = pipeline.nodes
done_nodes = set()
failed_nodes = set()
load_counts = Counter(chain.from_iterable(n.inputs for n in nodes))
for exec_index, node in enumerate(nodes):
try:
run_node(node, catalog, hook_manager, self._is_async, session_id)
done_nodes.add(node)
except Exception:
failed_nodes.add(node)
# decrement load counts and release any data sets we've finished with
for data_set in node.inputs:
load_counts[data_set] -= 1
if load_counts[data_set] < 1 and data_set not in pipeline.inputs():
catalog.release(data_set)
for data_set in node.outputs:
if data_set not in failed_nodes:
if load_counts[data_set] < 1 and data_set not in pipeline.outputs():
catalog.release(data_set)
self._logger.info(
"Completed %d out of %d tasks", exec_index + 1, len(nodes)
)
antheas
09/30/2022, 3:18 PMrafael.gildin
10/06/2022, 12:47 PMantheas
10/06/2022, 12:55 PMrafael.gildin
10/20/2022, 9:13 PM