You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Chendi.Xue (JIRA)" <ji...@apache.org> on 2019/04/09 06:46:00 UTC

[jira] [Created] (SPARK-27412) Add a new shuffle manager to use Persistent Memory as shuffle and spilling storage

Chendi.Xue created SPARK-27412:
----------------------------------

             Summary: Add a new shuffle manager to use Persistent Memory as shuffle and spilling storage
                 Key: SPARK-27412
                 URL: https://issues.apache.org/jira/browse/SPARK-27412
             Project: Spark
          Issue Type: New Feature
          Components: Shuffle, Spark Core
    Affects Versions: 3.0.0
            Reporter: Chendi.Xue


Add a new shuffle manager called "PmemShuffleManager", by using which, we can use Persistent Memory Device as storage for shuffle and external sorter spilling.

In this implementation, we leveraged Persistent Memory Development Kit(PMDK) to support transaction write with high performance.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org