You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2015/05/02 06:24:11 UTC
[jira] [Resolved] (SPARK-7317) ShuffleHandle needs to be exposed
[ https://issues.apache.org/jira/browse/SPARK-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reynold Xin resolved SPARK-7317.
--------------------------------
Resolution: Fixed
Fix Version/s: 1.4.0
> ShuffleHandle needs to be exposed
> ---------------------------------
>
> Key: SPARK-7317
> URL: https://issues.apache.org/jira/browse/SPARK-7317
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Reporter: Mridul Muralidharan
> Assignee: Mridul Muralidharan
> Priority: Minor
> Fix For: 1.4.0
>
>
> ShuffleHandle is marked private[spark] - while a lot of code which depends on it, and exposes it, is DeveloperApi.
> While the actual implementation can remain private[spark], the handle class itself should be exposed so that Rdd's can leverage it.
> Example:
> a) ShuffleDependency.shuffleHandle exposes a ShuffleHandle
> b) ShuffleManager instance is exposed via SparkEnv.get.shuffleManager
> c) SparkEnv.get.shuffleManager.getReader is exposed which needs handle as param : and can be used to write RDD's which leverage shuffle without needing to go through spark's shuffle based rdd's.
> So all the machinery for custom RDD to leverage shuffle exists - except for specifying the ShuffleHandle class itself in dependencies.
> This allows for customizations in user code on how to leverage shuffle.
> For example, specialized join implementations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org