[jira] [Comment Edited] (SPARK-6104) spark SQL shuffle OOM

["Hong Shen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-6104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14342872#comment-14342872 ] 

Hong Shen edited comment on SPARK-6104 at 3/2/15 7:25 AM:
----------------------------------------------------------

Can you link to the duplicate issue?


was (Author: shenhong):
Can you link to the Duplicate issue?

> spark SQL shuffle OOM
> ---------------------
>
>                 Key: SPARK-6104
>                 URL: https://issues.apache.org/jira/browse/SPARK-6104
>             Project: Spark
>          Issue Type: Improvement
>    Affects Versions: 1.2.0
>            Reporter: Hong Shen
>
> Currently, spark shuffle can use ExternalAppendOnlyMap to combine data that have fetched, but it depend on ShuffleDependency's aggregator, while spark SQL's shuffle have not define aggregator, so it will easily cause OOM in large scale of data.
> In our cluster, we have used spark SQL, but it often appear OOM, it's hard to reset spark.default.parallelism every day. I think we should add aggregator for spark SQL shuffle.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org