[jira] [Created] (SPARK-33037) Add "spark.shuffle.manager" value to knownManagers

["BoYang (Jira)" <ji...@apache.org>]

BoYang created SPARK-33037:
------------------------------

             Summary: Add "spark.shuffle.manager" value to knownManagers
                 Key: SPARK-33037
                 URL: https://issues.apache.org/jira/browse/SPARK-33037
             Project: Spark
          Issue Type: Improvement
          Components: Shuffle
    Affects Versions: 3.0.1, 2.4.7
            Reporter: BoYang


Spark has a hardcode list to contain known shuffle managers, which has two values now. It does not contain user's custom shuffle manager which is set through Spark config "spark.shuffle.manager".

 

We hit issue when set "spark.shuffle.manager" with our own shuffle manager plugin (Uber Remote Shuffle Service implementation, https://github.com/uber/RemoteShuffleService).

 

Need to add "spark.shuffle.manager" config value to the known managers list as well.

 

The know managers list is in code:

common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
{quote}private final List<String> knownManagers = Arrays.asList(
  "org.apache.spark.shuffle.sort.SortShuffleManager",
  "org.apache.spark.shuffle.unsafe.UnsafeShuffleManager");
{quote}
 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org