Linkage error - duplicate class definition

[_soumya_ <so...@gmail.com>]

Facing a funny issue with the Spark class loader. Testing out a basic
functionality on a vagrant VM with spark running - looks like it's
attempting to ship the jar to a remote instance (in this case local) and
somehow is encountering the jar twice? 

14/07/11 23:27:59 INFO DAGScheduler: Got job 0 (count at
GenerateSEOContent.java:75) with 1 output partitions (allowLocal=false)
14/07/11 23:27:59 INFO DAGScheduler: Final stage: Stage 0(count at
GenerateSEOContent.java:75)
14/07/11 23:27:59 INFO DAGScheduler: Parents of final stage: List()
14/07/11 23:27:59 INFO DAGScheduler: Missing parents: List()
14/07/11 23:27:59 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[1] at map
at GenerateSEOContent.java:67), which has no missing parents
14/07/11 23:27:59 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0
(MappedRDD[1] at map at GenerateSEOContent.java:67)
14/07/11 23:27:59 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
14/07/11 23:27:59 INFO TaskSetManager: Starting task 0.0:0 as TID 0 on
executor localhost: localhost (PROCESS_LOCAL)
14/07/11 23:27:59 INFO TaskSetManager: Serialized task 0.0:0 as 3287 bytes
in 4 ms
14/07/11 23:27:59 INFO Executor: Running task ID 0
14/07/11 23:27:59 INFO Executor: Fetching
http://10.141.141.10:36365/jars/rickshaw-spark-0.0.1-SNAPSHOT.jar with
timestamp 1405121278732
14/07/11 23:27:59 INFO Utils: Fetching
http://10.141.141.10:36365/jars/rickshaw-spark-0.0.1-SNAPSHOT.jar to
/tmp/fetchFileTemp2298196547032055523.tmp
14/07/11 23:27:59 INFO Executor: Adding
file:/tmp/spark-defa5d35-1853-492f-b8e0-e7ac30a370b1/rickshaw-spark-0.0.1-SNAPSHOT.jar
to class loader
14/07/11 23:27:59 ERROR Executor: Exception in task ID 0
java.lang.LinkageError: loader (instance of 
org/apache/spark/executor/ChildExecutorURLClassLoader$userClassLoader$):
attempted  duplicate class definition for name:
"com/evocalize/rickshaw/spark/util/HdfsUtil"
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at
org.apache.spark.executor.ChildExecutorURLClassLoader$userClassLoader$.findClass(ExecutorURLClassLoader.scala:42)
	at
org.apache.spark.executor.ChildExecutorURLClassLoader.findClass(ExecutorURLClassLoader.scala:50)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:270)
	at
org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:60)



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Linkage-error-duplicate-class-definition-tp9482.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Linkage error - duplicate class definition

[Hafiz Mujadid <ha...@gmail.com>]

Have you solved this problem?




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Linkage-error-duplicate-class-definition-tp9482p21260.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org