[jira] [Updated] (TEZ-3634) reduce the buffer sizes in PipelinedSorter by a small amount

["Siddharth Seth (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TEZ-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siddharth Seth updated TEZ-3634:
--------------------------------
    Attachment: TEZ-3634.01.patch

[~rajesh.balamohan], [~gopalv] - could you please take a look.

> reduce the buffer sizes in PipelinedSorter by a small amount
> ------------------------------------------------------------
>
>                 Key: TEZ-3634
>                 URL: https://issues.apache.org/jira/browse/TEZ-3634
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>         Attachments: TEZ-3634.01.patch
>
>
> When running in large JVMs with the G1GC, the regionsize maxes out at 32MB.
> PipelinedSorter does allocations of 32MB and 256MB. The actual allocation ends up being 24 bytes over 32M/256M. As a result the 32M allocation ends up taking 2 regions (or 64M), and the 256M ends up taking 9 regions instead of 8. Also leads to fragmentation of the heap, resulting in more Full GCs than required.
> {code}
> Live humongous region 2 size 33554456 start 0x00007f4918000000 length 2 with remset 0 code roots 0 is marked 0 reclaim candidate 0 type array 1
> {code}
> {code}
> Dead humongous region 111 size 268435480 start 0x00007f49f2000000 length 9 with remset 0 code roots 0 is marked 0 reclaim candidate 1 type array 1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)