kedro pipelines are running without issue, running on spark

Okay creating a thread

running on spark 3.2.2 on S3

And if you do pip freeze

I am able to save viz output to json file, it still hangs at the same place but after breaking the run I could see the json file is created, I am also able to load the output json file

Can you do a pip freeze in your container? Additionally your requiremets.txt may be helpful.

did you get pip freeze result, I cant open the message.txt

Ah I just saw that was a HDFS warning not error sorry I've been looking at the wrong thing

the process abruptly hangs and never starts a session

This is super unusual

let me try doing that

Thank you