You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Davies Liu (JIRA)" <ji...@apache.org> on 2016/05/18 20:48:13 UTC

[jira] [Assigned] (SPARK-15392) The default value of size estimation is not good

     [ https://issues.apache.org/jira/browse/SPARK-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Davies Liu reassigned SPARK-15392:
----------------------------------

    Assignee: Davies Liu

> The default value of size estimation is not good
> ------------------------------------------------
>
>                 Key: SPARK-15392
>                 URL: https://issues.apache.org/jira/browse/SPARK-15392
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 2.0.0
>            Reporter: Davies Liu
>            Assignee: Davies Liu
>
> We use  autoBroadcastJoinThreshold + 1L as the default value of size estimation, that is not good in 2.0, because we will calculate the size based on size of schema, then the estimation could be less than autoBroadcastJoinThreshold if you have an SELECT on top of an DataFrame created from RDD.
> We should use an even bigger default value, for example, MaxLong.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org