You are currently viewing What Java & Scala Versions are Supported by Spark 3.5.0

Apache Spark 3.5.0 support and compatibility with different Java and Scala versions evolve with new releases to take advantage of language features and improvements. So, it is very important to understand what Java and Scala versions Spark 3.5.0 supports.

In this article, let’s understand the basic Java and Scala versions supported by Spark 3.5.0 versions and what issues you might come across by not having these supported versions.

In simple words, Spark 3.5.0 supports Java 8, 11, and 17 and Scala 2.12 and 2.13. By not having these versions, you will have issues like installations, performance degradation, inability to use some new features of Spark, and run-time errors.

Related: Spark compatibility matrix

What Java Versions are Supported in Spark 3.5.0?

Spark is compatible with both Windows and UNIX-like systems such as Linux and Mac OS. It is designed to operate on platforms supporting the required Java versions, encompassing JVMs on x86_64 and ARM64 architectures. Running Spark locally on a single machine is straightforward; you only need to ensure that Java is installed and accessible via the system PATH or by setting the JAVA_HOME environment variable to the Java installation directory.

Spark 3.5.0 officially supports Java 8, 11, and 17. Java 8 is commonly used, while Java 11 and 17 are also supported. Java 8 before version 8u371 support is deprecated as of Spark 3.5.0. However, with each minor release, it’s important to check the specific release notes for any updates or changes in supported Java versions.

What Scala Versions are Supported in Spark 3.5.0?

Spark 3.5 primarily supports Scala 2.12.x and 2.13.x. Before Spark 3.0, it supported Scala 2.11.x. However, Spark 3.0 moves towards Scala 2.12.x as the primary Scala version.

Issues of Using Unsupported Versions

Using an incorrect or unsupported Java or Scala version with Spark might result in various issues or errors when running Spark applications or working within the Spark environment. Some possible consequences include:

  1. Compatibility Issues: Spark might not function correctly or may encounter compatibility issues if it’s designed to work with specific versions of Java. This can lead to unexpected errors or failures during execution.
  2. Performance Degradation: Using an incompatible Java version might result in reduced performance or efficiency due to potential inefficiencies in leveraging the latest Java features or optimizations.
  3. Unsupported Features: Newer versions of Spark might rely on certain features or enhancements available in specific Java versions. Using an older or unsupported Java version could prevent you from utilizing these features.
  4. Runtime Errors: Incompatible Java versions might cause runtime errors, class linkage errors, or other exceptions during the execution of Spark jobs or while interacting with Spark APIs.

To avoid these issues, it’s crucial to refer to the official Spark documentation or release notes for any updates, changes, or specific version recommendations, especially for patch releases within the Spark 3.5.x series or for newer versions. Spark’s compatibility and supported versions might evolve with each release, incorporating new language features and optimizations.

Java and Scala Compatible Versions with Earlier Spark Releases

Here’s a table summarizing Spark versions along with their compatible Java and Scala versions:

Spark VersionCompatible Java VersionsCompatible Scala Versions
Spark 1.xJava 7 or laterScala 2.10.x
Spark 2.0 – 2.3Java 7 or laterScala 2.11.x
Spark 2.4.xJava 8, 11Scala 2.11.x, 2.12.x
Spark 3.0.xJava 8, 11, 16Scala 2.12.x
Spark 3.1.xJava 8, 11, 16Scala 2.12.x, 2.13.x
Spark 3.2.xJava 8, 11, 16Scala 2.12.x, 2.13.x
Spark 3.3.xJava 8, 11, 16Scala 2.12.x, 2.13.x
Spark 3.4.xJava 8, 11, 16Scala 2.12.x, 2.13.x
Spark 3.5.xJava 8, 11, 17Scala 2.12.x, 2.13.x

Please note that while these are common compatibilities for each Spark version, it’s always advisable to refer to the official Spark documentation or release notes for the most accurate and updated information regarding compatible Java and Scala versions for a specific Spark release. Compatibility might vary slightly or be enhanced in minor releases within a major Spark version.

Conclusion

In summary, the support and compatibility of Apache Spark 3.5.0 with different Java and Scala versions continuously evolve to leverage language enhancements and advancements. Spark 3.5.0 accommodates Java 8, 11, and 17, as well as Scala 2.12 and 2.13. Failing to use these specified versions may result in challenges during installations, decreased performance, the inability to utilize new Spark features, and encountering runtime errors. Understanding and aligning with these supported versions is essential for a seamless Spark 3.5.0 experience.

Happy Learning !!

Naveen Nelamali

Naveen Nelamali (NNK) is a Data Engineer with 20+ years of experience in transforming data into actionable insights. Over the years, He has honed his expertise in designing, implementing, and maintaining data pipelines with frameworks like Apache Spark, PySpark, Pandas, R, Hive and Machine Learning. Naveen journey in the field of data engineering has been a continuous learning, innovation, and a strong commitment to data integrity. In this blog, he shares his experiences with the data as he come across. Follow Naveen @ LinkedIn and Medium