You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2015/02/26 05:23:05 UTC
[jira] [Commented] (SPARK-6026) Eliminate the bypassMergeThreshold
parameter and associated hash-ish shuffle within the Sort shuffle code
[ https://issues.apache.org/jira/browse/SPARK-6026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337849#comment-14337849 ]
Reynold Xin commented on SPARK-6026:
------------------------------------
I need to think about this more. I think there are still cases where hashing is substantially faster than sorting, in particular when the number of partitions is very small (in that case you just write stuff out, rather than doing an expensive sort).
> Eliminate the bypassMergeThreshold parameter and associated hash-ish shuffle within the Sort shuffle code
> ---------------------------------------------------------------------------------------------------------
>
> Key: SPARK-6026
> URL: https://issues.apache.org/jira/browse/SPARK-6026
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.3.0
> Reporter: Kay Ousterhout
>
> The bypassMergeThreshold parameter (and associated use of a hash-ish shuffle when the number of partitions is less than this) is basically a workaround for SparkSQL, because the fact that the sort-based shuffle stores non-serialized objects is a deal-breaker for SparkSQL, which re-uses objects. Once the sort-based shuffle is changed to store serialized objects, we should never be secretly doing hash-ish shuffle even when the user has specified to use sort-based shuffle (because of its otherwise worse performance).
> [~rxin][~adav], masters of shuffle, it would be helpful to get agreement from you on this proposal (and also a sanity check that I've correctly characterized the issue).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org