Spark – Initial job has not accepted any resources; check your cluster UI

After setting up a new Spark cluster running on Yarn, I’ve come across an error Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources while running Spark application via spark-submit and also tried with PySpark but got the same error.

Initial job has not accepted any resources

As you noticed when you get this warning, your application keeps running and wait for the resources indefinitely. You need to kill a job in order to terminate.

This issue is mainly caused due to the resource unavailable to run your application on the cluster, I have resolved this issue and able to run the Spark application on the cluster successfully and though it would be helpful sharing here.

1. Disable dynamic resource allocation

If you have enough nodes, cores and memory resources available to yarn, Spark uses dynamic allocation to create spark workers and by default this dynamic allocation is enabled.

When you have a small cluster with limited resources, you can disable this option and allocate resources as per your need. Make sure to allocate resources less than the actual resources available.


spark-submit --conf spark.dynamicAllocation.enabled=false

When you turn off the dynamic allocation, you need to explicitly allocate the resources. Below is Python (PySpark) spark-submit command with minimum config.


spark-submit --deploy-mode cluster --master yarn \
        --driver-memory 3g --executor-memory 3g \
        --num-executors 2 --executor-cores 2 \
        --conf spark.dynamicAllocation.enabled=false \
        readcsv.py

Below is spark-submit for scala with minimum config.


spark-submit --deploy-mode cluster --master yarn \
        --driver-memory 3g --executor-memory 3g \
        --num-executors 2 --executor-cores 2 \
        --conf spark.dynamicAllocation.enabled=false \
        --jar application.jar \
        --class com.example.ReadCSV

2. By Pragmatically

You can also turn-off the dynamic resource allocation by pragmatically.


// By Pragmatically
conf = SparkConf().setAppName("SparkByExamples.com")
        .set("spark.shuffle.service.enabled", "false")
        .set("spark.dynamicAllocation.enabled", "false")

3. Starting Slave server

If you are running the Spark standalone cluster, you can also get this error when you are not running your slaves.

Start your slave servers by passing master URL


// Starting Slave server
start-slave.sh

Hope you able to resolve this error and able to run your application.

Happy Learning !!

Tags:

Naveen (NNK)

I am Naveen (NNK) working as a Principal Engineer. I am a seasoned Apache Spark Engineer with a passion for harnessing the power of big data and distributed computing to drive innovation and deliver data-driven insights. I love to design, optimize, and managing Apache Spark-based solutions that transform raw data into actionable intelligence. I am also passion about sharing my knowledge in Apache Spark, Hive, PySpark, R etc.

Leave a Reply

You are currently viewing Spark – Initial job has not accepted any resources; check your cluster UI