You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-issues@hadoop.apache.org by "He Tianyi (JIRA)" <ji...@apache.org> on 2015/09/22 04:26:04 UTC

[jira] [Created] (MAPREDUCE-6488) Make buffer size in PipeMapRed configurable

He Tianyi created MAPREDUCE-6488:
------------------------------------

             Summary: Make buffer size in PipeMapRed configurable
                 Key: MAPREDUCE-6488
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6488
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
            Reporter: He Tianyi
            Assignee: He Tianyi


Default value of buffer size is 128K in {{PipeMapRed}}.

When mapper input record is large enough that it won't fit in buffer, {{MapRunner}} blocks until written. If child process and input reader are both slow (due to calculation and decompress), then process of decoding and reading will rarely overlap with each other, hurting performance.

I suppose we should make the buffer size configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)