[jira] [Commented] (SPARK-22226) Code generation fails for dataframes with 10000 columns

["Marco Gaido (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-22226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16197036#comment-16197036 ] 

Marco Gaido commented on SPARK-22226:
-------------------------------------

[~srowen] I know that there are many ticket for this, but I wanted to submit a PR to solve this specific issue I mentioned in the description. This won't solve completely the error above (for datasets with 20.000 of columns for instance), but it will allow to support a larger number of columns than now. Thus I created this JIRA for that PR. If this is not the right approach, may you please tell me what I should do?
Thanks.

> Code generation fails for dataframes with 10000 columns
> -------------------------------------------------------
>
>                 Key: SPARK-22226
>                 URL: https://issues.apache.org/jira/browse/SPARK-22226
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Marco Gaido
>
> Code generation for very wide datasets can fail because of the Constant Pool limit reached.
> This can be caused by many reasons. One of them is that we are currently splitting the definition of the generated methods among several {{NestedClass}} but all these methods are called in the main class. Since we have entries added to the constant pool for each method invocation, this is limiting the number of rows and is leading for very wide dataset to:
> {noformat}
> org.codehaus.janino.JaninoRuntimeException: Constant pool for class org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificMutableProjection has grown past JVM limit of 0xFFFF
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org