[jira] [Created] (TEZ-3634) reduce the buffer sizes in PipelinedSorter by a small amount

["Siddharth Seth (JIRA)" <ji...@apache.org>]

Siddharth Seth created TEZ-3634:
-----------------------------------

             Summary: reduce the buffer sizes in PipelinedSorter by a small amount
                 Key: TEZ-3634
                 URL: https://issues.apache.org/jira/browse/TEZ-3634
             Project: Apache Tez
          Issue Type: Improvement
            Reporter: Siddharth Seth
            Assignee: Siddharth Seth


When running in large JVMs with the G1GC, the regionsize maxes out at 32MB.

PipelinedSorter does allocations of 32MB and 256MB. The actual allocation ends up being 24 bytes over 32M/256M. As a result the 32M allocation ends up taking 2 regions (or 64M), and the 256M ends up taking 9 regions instead of 8. Also leads to fragmentation of the heap, resulting in more Full GCs than required.

{code}
Live humongous region 2 size 33554456 start 0x00007f4918000000 length 2 with remset 0 code roots 0 is marked 0 reclaim candidate 0 type array 1
{code}

{code}
Dead humongous region 111 size 268435480 start 0x00007f49f2000000 length 9 with remset 0 code roots 0 is marked 0 reclaim candidate 1 type array 1
{code}





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)