[jira] [Commented] (TEZ-3673) Allocate smaller buffers in UnorderedPartitionedKVWriter

["Harish Jaiprakash (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991949#comment-15991949 ] 

Harish Jaiprakash commented on TEZ-3673:
----------------------------------------

[~seth] Please take a look. Couple of questions for computeNumBuffersAndSize:

* Should I subtract 64 bytes in all sizePerBuffer assignments? Currently only if sizePerBuffer is set because of bufferLimit parameter it does the correction. Assuming that bufferLimit was set to be some multiple of region size. Reason being that any other value might not be at region size boundary and correction may not be helpful.
* There is no logging or exception for wrong buffer merge percent, it silently corrects to be within limit. Is that alright?

> Allocate smaller buffers in UnorderedPartitionedKVWriter
> --------------------------------------------------------
>
>                 Key: TEZ-3673
>                 URL: https://issues.apache.org/jira/browse/TEZ-3673
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Harish Jaiprakash
>            Assignee: Harish Jaiprakash
>         Attachments: TEZ-3673.01.patch, TEZ-3673.02.patch, TEZ-3673.03.patch, TEZ-3673.04.patch
>
>
> UnorderedPartitionedKVWriter allocates in bigger chunks. It may or may not get filled up. In PipelinedSorter, we start off with 32MB chunks. But UnorderedPartitionedKVWriter can be worse as it allocates bigger blocks. Need to revisit this allocation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)