Sparkling Water – java.lang.NoClassDefFoundError: scala/tools/nsc/interpreter/InteractiveReader

While Running H2O Sparkling Water (Machine Learning Models) in the Spark cluster, you would probably get exceptionjava.lang.NoClassDefFoundError: scala/tools/nsc/interpreter/InteractiveReader and program fails.


Exception in thread "main" java.lang.NoClassDefFoundError: scala/tools/nsc/interpreter/InteractiveReader
	at water.api.scalaInt.ScalaCodeHandler.createInterpreterInPool(ScalaCodeHandler.scala:145)
	at water.api.scalaInt.ScalaCodeHandler$$anonfun$initializeInterpreterPool$1.apply(ScalaCodeHandler.scala:139)
	at water.api.scalaInt.ScalaCodeHandler$$anonfun$initializeInterpreterPool$1.apply(ScalaCodeHandler.scala:138)
	at scala.collection.immutable.Range.foreach(Range.scala:160)
	at water.api.scalaInt.ScalaCodeHandler.initializeInterpreterPool(ScalaCodeHandler.scala:138)
	at water.api.scalaInt.ScalaCodeHandler.<init>(ScalaCodeHandler.scala:42)
	at water.api.scalaInt.ScalaCodeHandler$.registerEndpoints(ScalaCodeHandler.scala:171)
	at water.api.CoreRestAPI$.registerEndpoints(CoreRestAPI.scala:32)
	at water.api.RestAPIManager.register(RestAPIManager.scala:39)
	at water.api.RestAPIManager.registerAll(RestAPIManager.scala:31)
	at org.apache.spark.h2o.backends.internal.InternalH2OBackend.init(InternalH2OBackend.scala:43)
	at org.apache.spark.h2o.H2OContext$H2OContextClientBased.initBackend(H2OContext.scala:450)
	at org.apache.spark.h2o.H2OContext.init(H2OContext.scala:150)
	at org.apache.spark.h2o.H2OContext$.getOrCreate(H2OContext.scala:608)
	at org.apache.spark.h2o.H2OContext$.getOrCreate(H2OContext.scala:636)
	at com.sparkbyexamples.spark.SparklingWaterExample$.delayedEndpoint$com$sparkbyexamples$spark$SparklingWaterExample$1(SparklingWaterExample.scala:13)
	at com.sparkbyexamples.spark.SparklingWaterExample$delayedInit$body.apply(SparklingWaterExample.scala:6)
	at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App$$anonfun$main$1.apply(App.scala:76)
	at scala.App$$anonfun$main$1.apply(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
	at scala.App$class.main(App.scala:76)
	at com.sparkbyexamples.spark.SparklingWaterExample$.main(SparklingWaterExample.scala:6)
	at com.sparkbyexamples.spark.SparklingWaterExample.main(SparklingWaterExample.scala)
Caused by: java.lang.ClassNotFoundException: scala.tools.nsc.interpreter.InteractiveReader
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 26 more

</pre>



Solution

I had this issue when I was running Sparkling Water with below configuration

  • Sparkling Water – sparkling-water-3.28.0.3-1-2.4
  • Spark – spark-2.4.4-bin-hadoop2.7 (with winutils)
  • Scala – 2.11.11
  • OS – Windows 10

In my case, after adding Scala compiler maven dependency, my issues have been resolved and have not seen this exception anymore.

   <dependency>
       <groupId>org.scala-lang</groupId>
       <artifactId>scala-compiler</artifactId>
       <version>2.11.11</version>
   </dependency>

You can also download this jar and submit with spark-submit.

In case, if your issue has not resolved, please comment with Spark, Sparkling Water and Scala version you are using, I will happy to help.

Happy Learning !!

NNK

SparkByExamples.com is a Big Data and Spark examples community page, all examples are simple and easy to understand and well tested in our development environment Read more ..

Leave a Reply