I went one step further and added a while loop to wait for

self._fs.ls(load_path)

to find something in the S3 bucket where my csv dataframe is, to no luck.