[GitHub] [spark] JoshRosen commented on a change in pull request #34351: [SPARK-37071][CORE] Make OpenHashMap serialize without reference tracking

[GitBox <gi...@apache.org>]

JoshRosen commented on a change in pull request #34351:
URL: https://github.com/apache/spark/pull/34351#discussion_r733921709



##########
File path: core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala
##########
@@ -149,17 +149,12 @@ class OpenHashMap[K : ClassTag, @specialized(Long, Int, Double) V: ClassTag](
     }
   }
 
-  // The following member variables are declared as protected instead of private for the
-  // specialization to work (specialized class extends the non-specialized one and needs access
-  // to the "private" variables).
-  // They also should have been val's. We use var's because there is a Scala compiler bug that

Review comment:
       Spark was using Scala 2.9.3 at the time that this code was originally written, so it's quite likely that this has been fixed now that Spark uses Scala 2.12.
   
   I searched the code and found that there's two places that duplicate this workaround:
   
   - [PrimitiveKeyOpenHashMap](https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/core/src/main/scala/org/apache/spark/util/collection/PrimitiveKeyOpenHashMap.scala#L120-L131)
   - [GraphXPrimitiveKeyOpenHashMap](https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/graphx/src/main/scala/org/apache/spark/graphx/util/collection/GraphXPrimitiveKeyOpenHashMap.scala#L140-L152)
   
   Once we've decided on the best fix we should make sure to apply it there as well.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org