Title
#beginners-need-help
WolVez

WolVez

10/08/2021, 7:07 PM
@User is there a dataset to load tons of log (json) files in s3 and take advantage of async? Looking at how
--async
is implemented with
_run_node_async
, there doesn't seem to be a good way to achieve this that I can think of, other than making my own dataset which applies an async get while running in non async mode (as stacking async on async is bad). However, this must be a common use case you have overcome? I feel like adding a hook to append to the node would be a good thing here.