[jira] [Created] (SPARK-7317) ShuffleHandle needs to be exposed

["Mridul Muralidharan (JIRA)" <ji...@apache.org>]

Mridul Muralidharan created SPARK-7317:
------------------------------------------

             Summary: ShuffleHandle needs to be exposed
                 Key: SPARK-7317
                 URL: https://issues.apache.org/jira/browse/SPARK-7317
             Project: Spark
          Issue Type: Improvement
          Components: Shuffle
            Reporter: Mridul Muralidharan
            Assignee: Mridul Muralidharan
            Priority: Minor


ShuffleHandle is marked private[spark] - while a lot of code which depends on it, and exposes it, is DeveloperApi.
While the actual implementation can remain private[spark], the handle class itself should be exposed so that Rdd's can leverage it.

Example: 
a) ShuffleDependency.shuffleHandle exposes a ShuffleHandle
b) ShuffleManager instance is exposed via SparkEnv.get.shuffleManager
c) SparkEnv.get.shuffleManager.getReader is exposed which needs handle as param : and can be used to write RDD's which leverage shuffle without needing to go through spark's shuffle based rdd's.

So all the machinery for custom RDD to leverage shuffle exists - except for specifying the ShuffleHandle class itself in dependencies.
This allows for customizations in user code on how to leverage shuffle.
For example, specialized join implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org