You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Hong Shen (JIRA)" <ji...@apache.org> on 2015/03/02 08:11:05 UTC

[jira] [Updated] (SPARK-6104) spark SQL shuffle OOM

     [ https://issues.apache.org/jira/browse/SPARK-6104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Shen updated SPARK-6104:
-----------------------------
    Summary: spark SQL shuffle OOM  (was: spark sql shuffle OOM)

> spark SQL shuffle OOM
> ---------------------
>
>                 Key: SPARK-6104
>                 URL: https://issues.apache.org/jira/browse/SPARK-6104
>             Project: Spark
>          Issue Type: Improvement
>    Affects Versions: 1.2.0
>            Reporter: Hong Shen
>
> Currently, spark shuffle can use ExternalAppendOnlyMap to combine data that have fetched, but it depend on ShuffleDependency's aggregator, while spark sql's shuffle have not define aggregator, so it will easily cause OOM in large scale of data.
> In our cluster, we have used spark sql, but it often appear OOM, it's hard to reset spark.default.parallelism every day. I think we should add aggregator for spark sql shuffle.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org