[GitHub] [spark] WangGuangxin edited a comment on pull request #34846: [SPARK-37593][CORE] Optimize HeapMemoryAllocator to avoid memory waste in humongous allocation when using G1GC

[GitBox <gi...@apache.org>]

WangGuangxin edited a comment on pull request #34846:
URL: https://github.com/apache/spark/pull/34846#issuecomment-992585451


   > @WangGuangxin . Do you think you can add one specific example about the case to the PR description?
   
   Hi @dongjoon-hyun ,  We can demo it using following piece of code.
   
   ```
   public static void bufferSizeTest(boolean optimize) {
       long totalAllocatedSize = 0L;
       int blockSize = 1024 * 1024 * 4; // 4m
       if (optimize) {
         blockSize -= 16;
       }
       List<long[]> buffers = new ArrayList<>();
       while (true) {
         long[] arr = new long[blockSize/8];
         buffers.add(arr);
         totalAllocatedSize += blockSize;
         System.out.println("Total allocated size: " + totalAllocatedSize);
       }
     }
   ```
   
   Run it using following jvm params
   ```
   java -Xmx100m -XX:+UseG1GC -XX:G1HeapRegionSize=4m -XX:-UseGCOverheadLimit -verbose:gc -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -Xss4m -XX:+ExitOnOutOfMemoryError -XX:ParallelGCThreads=4 -XX:ConcGCThreads=4
   ```
   
   with optimized = false
   ```
   Total allocated size: 46137344
   [GC pause (G1 Humongous Allocation) (young) 44M->44M(100M), 0.0007091 secs]
   [GC pause (G1 Evacuation Pause) (young) (initial-mark)-- 48M->48M(100M), 0.0021528 secs]
   [GC concurrent-root-region-scan-start]
   [GC concurrent-root-region-scan-end, 0.0000021 secs]
   [GC concurrent-mark-start]
   [GC pause (G1 Evacuation Pause) (young) 48M->48M(100M), 0.0011289 secs]
   [Full GC (Allocation Failure)  48M->48M(100M), 0.0017284 secs]
   [Full GC (Allocation Failure)  48M->48M(100M), 0.0013437 secs]
   Terminating due to java.lang.OutOfMemoryError: Java heap space
   ```
   
   with optimzied = true
   ```
   Total allocated size: 96468624
   [GC pause (G1 Humongous Allocation) (young)-- 92M->92M(100M), 0.0024416 secs]
   [Full GC (Allocation Failure)  92M->92M(100M), 0.0019883 secs]
   [GC pause (G1 Evacuation Pause) (young) (initial-mark) 96M->96M(100M), 0.0004282 secs]
   [GC concurrent-root-region-scan-start]
   [GC concurrent-root-region-scan-end, 0.0000040 secs]
   [GC concurrent-mark-start]
   [GC pause (G1 Evacuation Pause) (young) 96M->96M(100M), 0.0003269 secs]
   [Full GC (Allocation Failure)  96M->96M(100M), 0.0012409 secs]
   [Full GC (Allocation Failure)  96M->96M(100M), 0.0012607 secs]
   Terminating due to java.lang.OutOfMemoryError: Java heap space
   ```
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org