You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-dev@hadoop.apache.org by "Todd Lipcon (Created) (JIRA)" <ji...@apache.org> on 2011/10/24 22:52:33 UTC

[jira] [Created] (MAPREDUCE-3252) MR2: Map tasks rewrite data once even if output fits in sort buffer

MR2: Map tasks rewrite data once even if output fits in sort buffer
-------------------------------------------------------------------

                 Key: MAPREDUCE-3252
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3252
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: mrv2, task
    Affects Versions: 0.23.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Critical


I found that, even if the output of a map task fits entirely in its sort buffer, it was rewriting the output entirely rather than just renaming the first spill into place. This is due to RawLocalFileSystem.rename() falling back to a copy if renameTo() fails. The first rename attempt was failing because no one has called mkdir for the output directory yet.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira