You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Koert Kuipers <ko...@tresata.com> on 2016/04/15 18:38:18 UTC

ClassFormatError in latest spark 2 SNAPSHOT build

not sure why, but i am getting this today using spark 2 snapshots...
i am on java 7 and scala 2.11

16/04/15 12:35:46 WARN TaskSetManager: Lost task 2.0 in stage 3.0 (TID 15,
localhost): java.lang.ClassFormatError: Duplicate field name&signature in
class file
org/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificMutableProjection
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
    at
org.codehaus.janino.ByteArrayClassLoader.findClass(ByteArrayClassLoader.java:66)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at
org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown
Source)
    at
org.apache.spark.sql.catalyst.expressions.codegen.GenerateMutableProjection$$anonfun$create$2.apply(GenerateMutableProjection.scala:140)
    at
org.apache.spark.sql.catalyst.expressions.codegen.GenerateMutableProjection$$anonfun$create$2.apply(GenerateMutableProjection.scala:139)
    at
org.apache.spark.sql.execution.aggregate.AggregationIterator.generateProcessRow(AggregationIterator.scala:178)
    at
org.apache.spark.sql.execution.aggregate.AggregationIterator.<init>(AggregationIterator.scala:197)
    at
org.apache.spark.sql.execution.aggregate.SortBasedAggregationIterator.<init>(SortBasedAggregationIterator.scala:39)
    at
org.apache.spark.sql.execution.aggregate.SortBasedAggregate$$anonfun$doExecute$1$$anonfun$3.apply(SortBasedAggregate.scala:80)
    at
org.apache.spark.sql.execution.aggregate.SortBasedAggregate$$anonfun$doExecute$1$$anonfun$3.apply(SortBasedAggregate.scala:71)
    at
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$23.apply(RDD.scala:768)
    at
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$23.apply(RDD.scala:768)
    at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
    at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:72)
    at org.apache.spark.scheduler.Task.run(Task.scala:86)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:239)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$

Re: ClassFormatError in latest spark 2 SNAPSHOT build

Posted by Reynold Xin <rx...@databricks.com>.

Can you post the generated code?

df.queryExecution.debug.codeGen()

(Or something similar to that)

On Friday, April 15, 2016, Koert Kuipers <ko...@tresata.com> wrote:

> not sure why, but i am getting this today using spark 2 snapshots...
> i am on java 7 and scala 2.11
>
> 16/04/15 12:35:46 WARN TaskSetManager: Lost task 2.0 in stage 3.0 (TID 15,
> localhost): java.lang.ClassFormatError: Duplicate field name&signature in
> class file
> org/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificMutableProjection
>     at java.lang.ClassLoader.defineClass1(Native Method)
>     at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>     at
> org.codehaus.janino.ByteArrayClassLoader.findClass(ByteArrayClassLoader.java:66)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>     at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown
> Source)
>     at
> org.apache.spark.sql.catalyst.expressions.codegen.GenerateMutableProjection$$anonfun$create$2.apply(GenerateMutableProjection.scala:140)
>     at
> org.apache.spark.sql.catalyst.expressions.codegen.GenerateMutableProjection$$anonfun$create$2.apply(GenerateMutableProjection.scala:139)
>     at
> org.apache.spark.sql.execution.aggregate.AggregationIterator.generateProcessRow(AggregationIterator.scala:178)
>     at
> org.apache.spark.sql.execution.aggregate.AggregationIterator.<init>(AggregationIterator.scala:197)
>     at
> org.apache.spark.sql.execution.aggregate.SortBasedAggregationIterator.<init>(SortBasedAggregationIterator.scala:39)
>     at
> org.apache.spark.sql.execution.aggregate.SortBasedAggregate$$anonfun$doExecute$1$$anonfun$3.apply(SortBasedAggregate.scala:80)
>     at
> org.apache.spark.sql.execution.aggregate.SortBasedAggregate$$anonfun$doExecute$1$$anonfun$3.apply(SortBasedAggregate.scala:71)
>     at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$23.apply(RDD.scala:768)
>     at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$23.apply(RDD.scala:768)
>     at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
>     at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
>     at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)
>     at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)
>     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:72)
>     at org.apache.spark.scheduler.Task.run(Task.scala:86)
>     at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:239)
>     at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>     at java.util.concurrent.ThreadPoolExecutor$
>