[GitHub] attilapiros commented on a change in pull request #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSuccess

[GitBox <gi...@apache.org>]

attilapiros commented on a change in pull request #22024: [SPARK-25034][CORE] Remove allocations in onBlockFetchSuccess
URL: https://github.com/apache/spark/pull/22024#discussion_r257702269
 
 

 ##########
 File path: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala
 ##########
 @@ -160,7 +160,13 @@ private[spark] class TorrentBroadcast[T: ClassTag](obj: T, id: Long)
           releaseLock(pieceId)
         case None =>
           bm.getRemoteBytes(pieceId) match {
-            case Some(b) =>
+            case Some(splitB) =>
+
+              // Checksum computation and further computations require the data
+              // from the ChunkedByteBuffer to be merged, so we we merge it now.
+              val bMergedData = splitB.toArray
 
 Review comment:
   As I see when checksum is not enabled then merge is not needed.
   And you can go further you can extend the checksum calculation to work on an array of chunks as I think it is just calling Alder32's `update` in a iteration for all the chunks.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org