You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-issues@hadoop.apache.org by "He Tianyi (JIRA)" <ji...@apache.org> on 2015/09/22 05:30:04 UTC

[jira] [Commented] (MAPREDUCE-6488) Make buffer size in PipeMapRed configurable

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901860#comment-14901860 ] 

He Tianyi commented on MAPREDUCE-6488:
--------------------------------------

Went through this again, buffer size has nothing to do with this.
Please mark as Invalid.

> Make buffer size in PipeMapRed configurable
> -------------------------------------------
>
>                 Key: MAPREDUCE-6488
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6488
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: He Tianyi
>            Assignee: He Tianyi
>
> Default value of buffer size is 128K in {{PipeMapRed}}.
> When mapper input record is large enough that it won't fit in buffer, {{MapRunner}} blocks until written. If child process and input reader are both slow (due to calculation and decompress), then process of decoding and reading will rarely overlap with each other, hurting performance.
> I suppose we should make the buffer size configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)