https://kedro.org/ logo
#advanced-need-help
Title
# advanced-need-help
s

skuma458

08/25/2022, 6:54 AM
kedro pipelines are running without issue, running on spark
d

datajoely

08/25/2022, 6:55 AM
Okay creating a thread
Can you print out your HDFS version?
s

skuma458

08/25/2022, 7:08 AM
running on spark 3.2.2 on S3
d

datajoely

08/25/2022, 7:09 AM
And if you do pip freeze
s

skuma458

08/25/2022, 7:14 AM
I am able to save viz output to json file, it still hangs at the same place but after breaking the run I could see the json file is created, I am also able to load the output json file
d

datajoely

08/25/2022, 7:15 AM
Can you do a pip freeze in your container? Additionally your requiremets.txt may be helpful.
s

skuma458

08/25/2022, 7:24 AM
did you get pip freeze result, I cant open the message.txt
d

datajoely

08/25/2022, 7:27 AM
Ah I just saw that was a HDFS warning not error sorry I've been looking at the wrong thing
When you run Viz the browser launched but is blank?
s

skuma458

08/25/2022, 7:28 AM
the process abruptly hangs and never starts a session
d

datajoely

08/25/2022, 7:28 AM
This is super unusual
So given I'm not sure of the root cause it would be helpful if you could comment out half your pipeline and see if it that works
Then we sort of do a binary search approach to pinpoint which part of the pipeline is causing issues
s

skuma458

08/25/2022, 7:31 AM
let me try doing that
d

datajoely

08/25/2022, 7:31 AM
Thank you
I've not seen this before
3 Views