Hello, beginner here. I wanted to use hive context inside my node. I have enabled hive support in building spark session. Then inside the node I tried calling spark session:

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

df = spark.sql("SELECT * FROM ....")

But I got the error:

ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1641884194120_477874 to YARN : Application rejected by queue placement policy

Hi this is a spark issue. Make sure you can execute spark sql outside of kedro first.

@User this links might be useful as well: - https://kedro.readthedocs.io/en/latest/11_tools_integration/01_pyspark.html - https://github.com/quantumblacklabs/kedro-starter-pyspark-iris