[GitHub] [spark] dongjoon-hyun edited a comment on pull request #34846: [SPARK-37593][CORE] Optimize HeapMemoryAllocator to avoid memory waste when using G1GC

[GitBox <gi...@apache.org>]

dongjoon-hyun edited a comment on pull request #34846:
URL: https://github.com/apache/spark/pull/34846#issuecomment-1002336872


   The new proposed code focused only on fitting to JVM G1HeapRegionSize and it claims that it will reduce the Spark's pageSize (humongous) to smaller ones. In this case, there is no guarantee that new defaultPageSizeBytes is better than the previous calculated ones (whatever they were in the production) in terms of the executor time (if no GC occurs because there is enough memory in the extreme cases).
   ```
     private lazy val defaultPageSizeBytes = Utils.maybeG1HeapRegionSize match {
        case Some(heapRegionSize) if tungstenMemoryMode == MemoryMode.ON_HEAP =>
          heapRegionSize - Platform.LONG_ARRAY_OFFSET
   ```
   
   If the smaller page size always wins, we may want to have a default value, `1MB` (the minimum value of G1HeapRegionSize) - Platform.LONG_ARRAY_OFFSET, for `spark.buffer.pageSize` configuration, instead of this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org