[GitHub] [spark] JulySouthWind commented on a change in pull request #30164: [SPARK-32919][SHUFFLE] Driver side changes for coordinating push based shuffle by selecting external shuffle services for merging partitions

[GitBox <gi...@apache.org>]

JulySouthWind commented on a change in pull request #30164:
URL: https://github.com/apache/spark/pull/30164#discussion_r516451905



##########
File path: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala
##########
@@ -74,6 +74,12 @@ class BlockManagerMasterEndpoint(
   // Mapping from block id to the set of block managers that have the block.
   private val blockLocations = new JHashMap[BlockId, mutable.HashSet[BlockManagerId]]
 
+  // Mapping from host name to shuffle (mergers) services
+  private val mergerLocations = new mutable.LinkedHashMap[String, BlockManagerId]()

Review comment:
       Maybe, we can add a shuffle service master role to manage the shuffle service worker.  App driver requests shuffle service workers from  shuffle service master. The master can  customize the politics to assign the workers for apps. Also, the master  manages the blacklist of shuffle service worker, adding or removing shuffle workers. In addition, the master can notify worker to delete data of completed apps by heart beat. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org