[jira] [Updated] (SPARK-16624) Generated SpecificColumnarIterator code can exceed JVM size limit for cached DataFrames

["吴志龙 (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-16624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

吴志龙 updated SPARK-16624:
------------------------
    Summary: Generated SpecificColumnarIterator code can exceed JVM size limit for cached DataFrames  (was: cd /home/product/bi_etl/dump_data/dp_edw/)

> Generated SpecificColumnarIterator code can exceed JVM size limit for cached DataFrames
> ---------------------------------------------------------------------------------------
>
>                 Key: SPARK-16624
>                 URL: https://issues.apache.org/jira/browse/SPARK-16624
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.2
>         Environment: ./bin/spark-sql --master=yarn --queue queue3 --driver-memory 4g --driver-java-options -XX:MaxPermSize=1g --num-executors 8  --executor-memory 4g --conf spark.yarn.am.memory=4096m --hiveconf hive.cli.print.header=false -S -e “large sql command”
>            Reporter: 吴志龙
>             Fix For: 1.6.2
>
>
> java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException: Code of method "(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection" grows beyond 64 KB
> SQL: sql size 12K 
> data:more tha 300 million the data



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org