[jira] [Comment Edited] (TEZ-3140) Reduce AM memory usage while serialization

["Rohini Palaniswamy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167433#comment-15167433 ] 

Rohini Palaniswamy edited comment on TEZ-3140 at 2/25/16 4:56 PM:
------------------------------------------------------------------

Avoided the need for a intermediate byte array by directly writing into the DataOutputBuffer used by Hadoop.  DataOutputBuffer is

{code}
@InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
@InterfaceStability.Unstable
{code}

but we use it directly in Tez in lots of places. So I hope that is fine.  Added an else clause logic just for completeness sake and changed it to reuse a 8K buffer.


was (Author: rohini):
Avoided the need for a intermediate byte array by directly writing into the DataOutputBuffer used by Hadoop.  DataOutputBuffer is

@InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
@InterfaceStability.Unstable

but we use it directly in Tez in lots of places. So I hope that is fine.  Added an else clause logic just for completeness sake and changed it to reuse a 4K buffer.

> Reduce AM memory usage while serialization
> ------------------------------------------
>
>                 Key: TEZ-3140
>                 URL: https://issues.apache.org/jira/browse/TEZ-3140
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.7.1, 0.8.3
>
>         Attachments: TEZ-3140-1.patch
>
>
>    There is an unnecessary copy of userpayload byte array during serialization.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)