NoClassDefFoundError: org/apache/spark/Logging in SparkSession.getOrCreate

[Brad Cox <br...@gmail.com>]

I'm experimenting with Spark 2.0.1 for the first time and hitting a problem right out of the gate.

My main routine starts with this which I think is the standard idiom.

		SparkSession sparkSession = SparkSession
				.builder()
				.master("local")
				.appName("DecisionTreeExample")
				.getOrCreate();

Running this in the eclipse debugger, execution fails in getOrCreate() with this exception

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at org.apache.spark.sql.SparkSession.<init>(SparkSession.scala:122)
	at org.apache.spark.sql.SparkSession.<init>(SparkSession.scala:77)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:840)
	at titanic.DecisionTreeExample.main(DecisionTreeExample.java:54)

java.lang.NoClassDefFoundError means a class is not found at run time that was present at
compile time. I've googled everything I can think of and found no solutions. Can someone
help? Thanks!

These are my spark-relevant dependencies:

		<dependency>
			<groupId>org.apache.spark</groupId>
			<artifactId>spark-core_2.11</artifactId>
			<version>2.0.1</version>
		</dependency>
		<dependency>
			<groupId>org.apache.spark</groupId>
			<artifactId>spark-mllib_2.11</artifactId>
			<version>2.0.1</version>
		</dependency>
		<dependency>
			<groupId>org.apache.spark</groupId>
			<artifactId>spark-sql_2.11</artifactId>
			<version>2.0.1</version>
		</dependency>

 

Dr. Brad J. Cox    Cell: 703-594-1883 Skype: dr.brad.cox





---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: NoClassDefFoundError: org/apache/spark/Logging in SparkSession.getOrCreate

[Saisai Shao <sa...@gmail.com>]

Not sure why your code will search Logging class under org/apache/spark,
this should be “org/apache/spark/internal/Logging”, and it changed long
time ago.


On Sun, Oct 16, 2016 at 3:25 AM, Brad Cox <br...@gmail.com> wrote:

> I'm experimenting with Spark 2.0.1 for the first time and hitting a
> problem right out of the gate.
>
> My main routine starts with this which I think is the standard idiom.
>
>                 SparkSession sparkSession = SparkSession
>                                 .builder()
>                                 .master("local")
>                                 .appName("DecisionTreeExample")
>                                 .getOrCreate();
>
> Running this in the eclipse debugger, execution fails in getOrCreate()
> with this exception
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/spark/Logging
>         at java.lang.ClassLoader.defineClass1(Native Method)
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>         at java.security.SecureClassLoader.defineClass(
> SecureClassLoader.java:142)
>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>         at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>         at org.apache.spark.sql.SparkSession.<init>(
> SparkSession.scala:122)
>         at org.apache.spark.sql.SparkSession.<init>(SparkSession.scala:77)
>         at org.apache.spark.sql.SparkSession$Builder.
> getOrCreate(SparkSession.scala:840)
>         at titanic.DecisionTreeExample.main(DecisionTreeExample.java:54)
>
> java.lang.NoClassDefFoundError means a class is not found at run time that
> was present at
> compile time. I've googled everything I can think of and found no
> solutions. Can someone
> help? Thanks!
>
> These are my spark-relevant dependencies:
>
>                 <dependency>
>                         <groupId>org.apache.spark</groupId>
>                         <artifactId>spark-core_2.11</artifactId>
>                         <version>2.0.1</version>
>                 </dependency>
>                 <dependency>
>                         <groupId>org.apache.spark</groupId>
>                         <artifactId>spark-mllib_2.11</artifactId>
>                         <version>2.0.1</version>
>                 </dependency>
>                 <dependency>
>                         <groupId>org.apache.spark</groupId>
>                         <artifactId>spark-sql_2.11</artifactId>
>                         <version>2.0.1</version>
>                 </dependency>
>
>
>
> Dr. Brad J. Cox    Cell: 703-594-1883 Skype: dr.brad.cox
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>