Re: [PR] [SPARK-45605][CORE][SQL][SS][CONNECT][MLLIB][GRAPHX][DSTREAM][PROTOBUF][EXAMPLES] Replace `s.c.MapOps.mapValues` with `s.c.MapOps.view.mapValues` [spark]

["LuciferYang (via GitHub)" <gi...@apache.org>]

LuciferYang commented on code in PR #43448:
URL: https://github.com/apache/spark/pull/43448#discussion_r1411568249


##########
common/sketch/src/test/scala/org/apache/spark/util/sketch/CountMinSketchSuite.scala:
##########
@@ -56,7 +56,7 @@ class CountMinSketchSuite extends AnyFunSuite { // scalastyle:ignore funsuite
 
       val exactFreq = {
         val sampledItems = sampledItemIndices.map(allItems)
-        sampledItems.groupBy(identity).mapValues(_.length.toLong)
+        sampledItems.groupBy(identity).view.mapValues(_.length.toLong)

Review Comment:
   @cloud-fan 
   
   Thank you very much for pointing this out, it's a great observation.
   
   Let me explain first, in Scala 2.13, changing to use `.transform` would change the semantics,
   
   ```
   sampledItems.groupBy(identity).mapValues(_.length.toLong)
   ```
   and 
   
   ```
   sampledItems.groupBy(identity).view.mapValues(_.length.toLong)
   ```
   
   both return a `MapView`.
   
   
   While
   
   ```
   sampledItems.groupBy(identity).transform((_, v) => v.length.toLong)
   ```
   
   returns a `Map`.
   
   The main difference between `MapView` and `Map` lies in their computation methods and memory usage:
   
   - `MapView` is a kind of lazy computation collection view, it does not immediately calculate the result, but calculates when needed. This means that if you create a `MapView` but don't actually use it, the related calculations will not occur. This can improve performance when dealing with large data sets, as it can avoid unnecessary calculations. However, each time `MapView` is accessed, the calculation will be performed again, which may lead to performance degradation, especially in cases where multiple accesses are required.
   
   - On the other hand, `Map` is an immediate computation collection, it calculates all the results at creation and stores the results in memory. This means that whether or not you actually use this `Map`, the related calculations will occur. This may lead to higher memory usage, but in cases where the same `Map` needs to be accessed multiple times, it can provide better performance because the results have been calculated and stored in memory, no need to recalculate each time it is accessed.
   
   I will keep an eye on this case and adjust the cases where `Map` is suitable for use.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org