[jira] [Updated] (TEZ-3673) Allocate smaller buffers in UnorderedPartitionedKVWriter

["Harish Jaiprakash (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TEZ-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harish Jaiprakash updated TEZ-3673:
-----------------------------------
    Attachment: TEZ-3673.02.patch

* Using configured value for buffer limit instead of 32 mb.
* Using a configuration to fraction of buffers in a spill.
* Changed buffer allocation logic, if the buffer limit is set and available memory >= buffer limit * 2, it will use buffer limit as size always. Over allocating or under allocating depending on the left over size.

Counter + warning for too many spills pending, I'll add them once Rajesh's optimization is in.

> Allocate smaller buffers in UnorderedPartitionedKVWriter
> --------------------------------------------------------
>
>                 Key: TEZ-3673
>                 URL: https://issues.apache.org/jira/browse/TEZ-3673
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Harish Jaiprakash
>            Assignee: Harish Jaiprakash
>         Attachments: TEZ-3673.01.patch, TEZ-3673.02.patch
>
>
> UnorderedPartitionedKVWriter allocates in bigger chunks. It may or may not get filled up. In PipelinedSorter, we start off with 32MB chunks. But UnorderedPartitionedKVWriter can be worse as it allocates bigger blocks. Need to revisit this allocation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)