[jira] [Commented] (SPARK-7317) ShuffleHandle needs to be exposed

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524469#comment-14524469 ] 

Apache Spark commented on SPARK-7317:
-------------------------------------

User 'mridulm' has created a pull request for this issue:
https://github.com/apache/spark/pull/5857

> ShuffleHandle needs to be exposed
> ---------------------------------
>
>                 Key: SPARK-7317
>                 URL: https://issues.apache.org/jira/browse/SPARK-7317
>             Project: Spark
>          Issue Type: Improvement
>          Components: Shuffle
>            Reporter: Mridul Muralidharan
>            Assignee: Mridul Muralidharan
>            Priority: Minor
>
> ShuffleHandle is marked private[spark] - while a lot of code which depends on it, and exposes it, is DeveloperApi.
> While the actual implementation can remain private[spark], the handle class itself should be exposed so that Rdd's can leverage it.
> Example: 
> a) ShuffleDependency.shuffleHandle exposes a ShuffleHandle
> b) ShuffleManager instance is exposed via SparkEnv.get.shuffleManager
> c) SparkEnv.get.shuffleManager.getReader is exposed which needs handle as param : and can be used to write RDD's which leverage shuffle without needing to go through spark's shuffle based rdd's.
> So all the machinery for custom RDD to leverage shuffle exists - except for specifying the ShuffleHandle class itself in dependencies.
> This allows for customizations in user code on how to leverage shuffle.
> For example, specialized join implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org