You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "BoYang (Jira)" <ji...@apache.org> on 2020/10/11 05:41:00 UTC

[jira] [Created] (SPARK-33114) Add metadata in MapStatus to support custom shuffle manager

BoYang created SPARK-33114:
------------------------------

             Summary: Add metadata in MapStatus to support custom shuffle manager
                 Key: SPARK-33114
                 URL: https://issues.apache.org/jira/browse/SPARK-33114
             Project: Spark
          Issue Type: Improvement
          Components: Shuffle
    Affects Versions: 3.0.1
            Reporter: BoYang


Current MapStatus class is tightly bound with local (sort merge) shuffle which uses BlockManagerId to store the shuffle data location. It could not support other custom shuffle manager implementation.

We could add "metadata" to MapStatus and allow different shuffle manager implementation to store information related to them.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org