[jira] [Created] (SPARK-15352) Topology aware block replication

["Shubham Chopra (JIRA)" <ji...@apache.org>]

Shubham Chopra created SPARK-15352:
--------------------------------------

             Summary: Topology aware block replication
                 Key: SPARK-15352
                 URL: https://issues.apache.org/jira/browse/SPARK-15352
             Project: Spark
          Issue Type: New Feature
          Components: Block Manager, Mesos, Spark Core, YARN
            Reporter: Shubham Chopra


With cached RDDs, Spark can be used for online analytics where it is used to respond to online queries. But loss of RDD partitions due to node/executor failures can cause huge delays in such use cases as the data would have to be regenerated.
Cached RDDs, even when using multiple replicas per block, are not currently resilient to node failures when multiple executors are started on the same node. Block replication currently chooses a peer at random, and this peer could also exist on the same host. 
This effort would add topology aware replication to Spark that can be enabled with pluggable strategies. For ease of development/review, this is being broken down to three major work-efforts:
1.	Making peer selection for replication pluggable
2.	Providing pluggable implementations for providing topology and topology aware replication
3.	Pro-active replenishment of lost blocks




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org