[jira] [Updated] (MAPREDUCE-6166) Reducers do not catch bad map output transfers during shuffle if data shuffled directly to disk

["Eric Payne (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Payne updated MAPREDUCE-6166:
----------------------------------
    Attachment: MAPREDUCE-6166.v1.201411221941.txt

This patch makes a change to OnDiskMapOutput#shuffle to wrap the input stream with IFileInputStream#readWithChecksum. The result is that when a corrupted map output file is shuffled directly to disk on the reduce side, the corruption is caught immediately, and the map is tried again (due to "Too many shuffle failures").

This behavior is similar to what happens when a corrupted file is shuffled to memory as opposed to disk.

> Reducers do not catch bad map output transfers during shuffle if data shuffled directly to disk
> -----------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6166
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6166
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 2.6.0
>            Reporter: Eric Payne
>            Assignee: Eric Payne
>         Attachments: MAPREDUCE-6166.v1.201411221941.txt
>
>
> In very large map/reduce jobs (50000 maps, 2500 reducers), the intermediate map partition output gets corrupted on disk on the map side. If this corrupted map output is too large to shuffle in memory, the reducer streams it to disk without validating the checksum. In jobs this large, it could take hours before the reducer finally tries to read the corrupted file and fails. Since retries of the failed reduce attempt will also take hours, this delay in discovering the failure is multiplied greatly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)