[GitHub] [spark] LuciferYang commented on a diff in pull request #39879: [SPARK-42336][CORE] Use OpenHashMap instead of HashMap in ResourceAllocator

["LuciferYang (via GitHub)" <gi...@apache.org>]

LuciferYang commented on code in PR #39879:
URL: https://github.com/apache/spark/pull/39879#discussion_r1096051656


##########
core/src/main/scala/org/apache/spark/resource/ResourceAllocator.scala:
##########
@@ -38,11 +37,13 @@ private[spark] trait ResourceAllocator {
    * For task resources ([[org.apache.spark.scheduler.ExecutorResourceInfo]]), this value
    * can be a multiple, such that each address can be allocated up to [[slotsPerAddress]]
    * times.
-   *
-   * TODO Use [[org.apache.spark.util.collection.OpenHashMap]] instead to gain better performance.
    */
   private lazy val addressAvailabilityMap = {
-    mutable.HashMap(resourceAddresses.map(_ -> slotsPerAddress): _*)
+    val map: OpenHashMap[String, Int] = new OpenHashMap[String, Int]()

Review Comment:
   This is a `lazy val`. How much performance will it improve if it is calculated only once? Could you make a mirco-benchmark to compare it(The benchmark code does not need to be submitted to pr)?
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org