You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Aris V (JIRA)" <ji...@apache.org> on 2016/08/16 21:14:20 UTC

[jira] [Created] (SPARK-17092) DataFrame with large number of columns causing code generation error

Aris V created SPARK-17092:
------------------------------

             Summary: DataFrame with large number of columns causing code generation error
                 Key: SPARK-17092
                 URL: https://issues.apache.org/jira/browse/SPARK-17092
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.0.0
         Environment: On vanilla Spark hadoop 2.7 Scala 2.11 in Linux CentOS, cluster with 9 slaves. Amazon AWS node size m3.2xlarge.

            Reporter: Aris V


On vanilla Spark hadoop 2.7 Scala 2.11:

When I use randomSplit on a DataFrame with several hundreds of columns, I get Janino code generation errors. The lowest number of columns that triggers the bug is around 500 or less.

The error message:

```
Caused by: org.codehaus.janino.JaninoRuntimeException: Code of method "(Lorg/apache/spark/sql/catalyst/InternalRow;Lorg/apache/spark/sql/catalyst/InternalRow;)I" of class "org.apache.spark.sql.catalyst.ex
pressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB
```

Here is a small code sample which causes it in spark-shell

```
import org.apache.spark.sql.types.{DoubleType, StructType}
import org.apache.spark.sql.{Row, SparkSession}

val COLMAX: Double = 500.0
val ROWSIZE: Int = 1000

val intToRow: Int => Row = (i: Int) => Row.fromSeq(Range.Double.inclusive(1.0, COLMAX, 1.0).toSeq)
val schema: StructType = (1 to COLMAX.toInt).foldLeft(new StructType())((s, i) => s.add(i.toString, DoubleType, nullable = true))
val rdds = spark.sparkContext.parallelize((1 to ROWSIZE).map(intToRow))
val df = spark.createDataFrame(rdds, schema)
val Array(left, right) = df.randomSplit(Array(.8,.2))

// This crashes
left.count
```






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org