You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2016/06/01 05:44:12 UTC

[jira] [Created] (SPARK-15690) Fast single-node in-memory shuffle

Reynold Xin created SPARK-15690:
-----------------------------------

             Summary: Fast single-node  in-memory shuffle
                 Key: SPARK-15690
                 URL: https://issues.apache.org/jira/browse/SPARK-15690
             Project: Spark
          Issue Type: New Feature
          Components: Shuffle, SQL
            Reporter: Reynold Xin


An increasing number of Spark users are using the system to process data on a single-node. When in a single node operating against intermediate data that fits in memory, the existing shuffle code path can become a big bottleneck.

Ideally, Spark should be able to use in-memory radix sort to do data shuffling on a single node



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org