kedro pipelines are running without issue running on spark

Question

datajoely · Answer

Okay creating a thread

datajoely · Answer

Can you print out your HDFS version

skuma458 · Answer

running on spark 3 2 2 on S3

datajoely · Answer

And if you do pip freeze

skuma458 · Answer

I am able to save viz output to json file it still hangs at the same place but after breaking the run I could see the json file is created I am also able to load the output json file

datajoely · Answer

Can you do a pip freeze in your container Additionally your requiremets txt may be helpful

skuma458 · Answer

did you get pip freeze result I cant open the message txt

datajoely · Answer

Ah I just saw that was a HDFS warning not error sorry I ve been looking at the wrong thing

datajoely · Answer

When you run Viz the browser launched but is blank

skuma458 · Answer

the process abruptly hangs and never starts a session

datajoely · Answer

This is super unusual

datajoely · Answer

So given I m not sure of the root cause it would be helpful if you could comment out half your pipeline and see if it that works

datajoely · Answer

Then we sort of do a binary search approach to pinpoint which part of the pipeline is causing issues

skuma458 · Answer

let me try doing that

datajoely · Answer

Thank you

datajoely · Answer

I ve not seen this before