[jira] Updated: (MAPREDUCE-114) All reducer tasks are finished, while some mapper tasks are still running

["Alexey Spiridonov (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexey Spiridonov updated MAPREDUCE-114:
----------------------------------------

    Attachment: hadoop-bug-useless-task.png
                hadoop-bug-overview.png

I had encountered this very same issue.

The task in the attached screenshot had finished computing, and apparently had successfully sent data to the reducers (I checked some reducer logs). However, it lost the tracker before it notified the reducer about task completion. As a result, the job is stuck waiting for the replacement map tasks to finish, even though they are not needed.

(I cut out hostnames from the screenshots for privacy reasons)

> All reducer tasks are finished, while some mapper tasks are still running
> -------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-114
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-114
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Qi Liu
>         Attachments: hadoop-bug-overview.png, hadoop-bug-useless-task.png
>
>
> In a high load environment (i.e. multiple jobs are queued up to be executed), when all reducer tasks of a job are finished, some mapper tasks of the same job may still running (possibly re-executed due to lost task tracker, etc).
> This should not happen when a job has at least one reducer task. When all reducer tasks are in SUCCEEDED state, the Hadoop JobTracker should kill all running mapper tasks, since execution would be meaningless. The job should also switch to SUCCEEDED state when all reducer tasks of that job succeeded successfully.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.