You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@tez.apache.org by "Harish Jaiprakash (JIRA)" <ji...@apache.org> on 2017/03/28 14:57:41 UTC

[jira] [Updated] (TEZ-3673) Allocate smaller buffers in UnorderedPartitionedKVWriter

     [ https://issues.apache.org/jira/browse/TEZ-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harish Jaiprakash updated TEZ-3673:
-----------------------------------
    Attachment: TEZ-3673.01.patch

Using 32 mb buffer size and aggregating buffers for spills.

> Allocate smaller buffers in UnorderedPartitionedKVWriter
> --------------------------------------------------------
>
>                 Key: TEZ-3673
>                 URL: https://issues.apache.org/jira/browse/TEZ-3673
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Harish Jaiprakash
>            Assignee: Harish Jaiprakash
>         Attachments: TEZ-3673.01.patch
>
>
> UnorderedPartitionedKVWriter allocates in bigger chunks. It may or may not get filled up. In PipelinedSorter, we start off with 32MB chunks. But UnorderedPartitionedKVWriter can be worse as it allocates bigger blocks. Need to revisit this allocation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)