[jira] [Assigned] (MAPREDUCE-3927) Shuffle hang when set map.failures.percent

["Jason Lowe (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Lowe reassigned MAPREDUCE-3927:
-------------------------------------

    Assignee: Bhallamudi Venkata Siva Kamesh

Overall looks good.  Couple of comments:

In Job.java, this code:

{code}
if (reportedAfterCompletion
    && status.getState() == JobStatus.State.SUCCEEDED) {
  mapProgress = mapProgress != 1.0f ? 1.0f : mapProgress;
  reduceProgress = reduceProgress != 1.0f ? 1.0f : reduceProgress;
}
{code}

would be better written as:

{code}
if (reportedAfterCompletion
    && status.getState() == JobStatus.State.SUCCEEDED) {
  mapProgress = 1.0f;
  reduceProgress = 1.0f;
}
{code}

Also I noticed that when one or more map task fails (after max attempts have failed), the map completion percentage of the job isn't accurate until the job completes.  (e.g.: reducer progress shows we're well past the shuffle stage even though map progress is <100%).  If that were done properly, I'm not sure we'd need the SUCCEEDED == progress 1.0 fixup in Job.java.
                
> Shuffle hang when set map.failures.percent
> ------------------------------------------
>
>                 Key: MAPREDUCE-3927
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3927
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 2.0.0-alpha, 2.0.1-alpha, trunk
>            Reporter: MengWang
>            Assignee: Bhallamudi Venkata Siva Kamesh
>            Priority: Critical
>              Labels: patch
>         Attachments: MAPREDUCE-3927-1.patch, MAPREDUCE-3927.patch, MAPREDUCE-3927.patch
>
>
> When set mapred.max.map.failures.percent and there does have some failed maps, then shuffle will hang

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira