https://kedro.org/ logo
Title
k

khern

02/03/2022, 3:15 AM
Hello, beginner here. I wanted to use hive context inside my node. I have enabled hive support in building spark session. Then inside the node I tried calling spark session:
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

df = spark.sql("SELECT * FROM ....")
But I got the error:
ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1641884194120_477874 to YARN : Application rejected by queue placement policy
l

limdauto

02/03/2022, 9:48 AM
Hi this is a spark issue. Make sure you can execute spark sql outside of kedro first.