Pyspark: Exception: Java gateway process exited before sending the driver its port number

  • Post author:
  • Post category:PySpark
  • Post last modified:March 27, 2024
  • Reading time:4 mins read
You are currently viewing Pyspark: Exception: Java gateway process exited before sending the driver its port number

Problem: While running PySpark application through spark-submit, Spyder or even from PySpark shell I am getting Pyspark: Exception: Java gateway process exited before sending the driver its port number.

Advertisements

Solution: Pyspark: Exception: Java gateway process exited before sending the driver its port number

In order to run PySpark (Spark with Python) you would need to have Java installed on your Mac, Linux or Windows, without Java installation & not having JAVA_HOME environment variable set with Java installation path or not having PYSPARK_SUBMIT_ARGS, you would get Exception: Java gateway process exited before sending the driver its port number.

Set PYSPARK_SUBMIT_ARGS

Set PYSPARK_SUBMIT_ARGS with master, this resolves Exception: Java gateway process exited before sending the driver its port number.


export PYSPARK_SUBMIT_ARGS="--master local[3] pyspark-shell"

vi ~/.bashrc , add the above line and reload the bashrc file using source ~/.bashrc

Incase if issue still doesn’t resolve, check your Java installation and JAVA_HOME environment variable.

Install Open JDK

Why you need Java to run PySpark?

Spark basically written in Scala and later on due to its industry adaptation it’s API PySpark released for Python using Py4J. Py4J is a Java library that is integrated within PySpark and allows python to dynamically interface with JVM objects, hence to run PySpark you also need Java to be installed along with Python, and Apache Spark.

Use below commands to install OpenJDK or Oracle JDK on Linux Ubuntu.


# To Install Open JDK
sudo add-apt-repository ppa:openjdk-r/ppa
sudo apt-get update
sudo apt-get install openjdk-11-jdk

# To Install Oracke JDK varsion 8
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

Set JAVA_HOME Environment Variable

Now export JAVA_HOME with the java installation directory.


export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

vi ~/.bashrc , add the above line and reload the bashrc file using source ~/.bashrc

Happy Learning