You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Qi Liu (JIRA)" <ji...@apache.org> on 2009/07/22 22:01:14 UTC

[jira] Created: (MAPREDUCE-790) TaskTracker blacklisted by one job should be blacklisted for all other jobs in the queue

TaskTracker blacklisted by one job should be blacklisted for all other jobs in the queue
----------------------------------------------------------------------------------------

                 Key: MAPREDUCE-790
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-790
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: jobtracker
    Affects Versions: 0.20.1
            Reporter: Qi Liu


Once a task tracker is blacklisted by one job, it is still being used by all other jobs in the queue. A blacklisted task tracker could be a signal of marginal node, and thus it should be blacklisted for all jobs at least temporarily. Also, even if one task tracker has been blacklisted globally due to too many failures, the blacklists of the jobs in the queue are not affected, and thus will continue to use the bad task tracker. This could result job failure.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-790) TaskTracker blacklisted by one job should be blacklisted for all other jobs in the queue

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734324#action_12734324 ] 

Arun C Murthy commented on MAPREDUCE-790:
-----------------------------------------

We do roll-up per-job blacklists to create a global one. Once a tasktracker is globally blacklisted we do not assign any more tasks from _any_ jobs to it. Once improvement might be to immediately kill all running tasks on a tasktracker as soon as it's blacklisted (per-job basis or globally).

> TaskTracker blacklisted by one job should be blacklisted for all other jobs in the queue
> ----------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-790
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-790
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>    Affects Versions: 0.20.1
>            Reporter: Qi Liu
>
> Once a task tracker is blacklisted by one job, it is still being used by all other jobs in the queue. A blacklisted task tracker could be a signal of marginal node, and thus it should be blacklisted for all jobs at least temporarily. Also, even if one task tracker has been blacklisted globally due to too many failures, the blacklists of the jobs in the queue are not affected, and thus will continue to use the bad task tracker. This could result job failure.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-790) TaskTracker blacklisted by one job should be blacklisted for all other jobs in the queue

Posted by "Qi Liu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734328#action_12734328 ] 

Qi Liu commented on MAPREDUCE-790:
----------------------------------

>From our experience, that is not the case, though we are using 0.18.3.

Here is what had happened. We submitted 10 jobs at the same time. Since all jobs are pretty big jobs, they are queued except one job is running. After 3 or 4 jobs finished, we have two globally blacklisted tasktrackers. However, we can still see tasks from the remaining jobs assigned to those globally blacklisted nodes. It appears that the global blacklist is not actively synchronized with job-specific blacklist.

Also, I would like to know if the globally blacklisted task trackers will be unblacklisted automatically after some time, or until that task tracker is restarted?

> TaskTracker blacklisted by one job should be blacklisted for all other jobs in the queue
> ----------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-790
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-790
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>    Affects Versions: 0.20.1
>            Reporter: Qi Liu
>
> Once a task tracker is blacklisted by one job, it is still being used by all other jobs in the queue. A blacklisted task tracker could be a signal of marginal node, and thus it should be blacklisted for all jobs at least temporarily. Also, even if one task tracker has been blacklisted globally due to too many failures, the blacklists of the jobs in the queue are not affected, and thus will continue to use the bad task tracker. This could result job failure.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (MAPREDUCE-790) TaskTracker blacklisted by one job should be blacklisted for all other jobs in the queue

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy resolved MAPREDUCE-790.
-------------------------------------

    Resolution: Duplicate

See HADOOP-4305 for details on global blacklisting of tasktrackers.

> TaskTracker blacklisted by one job should be blacklisted for all other jobs in the queue
> ----------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-790
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-790
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>    Affects Versions: 0.20.1
>            Reporter: Qi Liu
>
> Once a task tracker is blacklisted by one job, it is still being used by all other jobs in the queue. A blacklisted task tracker could be a signal of marginal node, and thus it should be blacklisted for all jobs at least temporarily. Also, even if one task tracker has been blacklisted globally due to too many failures, the blacklists of the jobs in the queue are not affected, and thus will continue to use the bad task tracker. This could result job failure.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.