You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Doug Cutting (JIRA)" <ji...@apache.org> on 2008/01/10 18:48:35 UTC

[jira] Created: (HADOOP-2573) limit running tasks per job

limit running tasks per job
---------------------------

                 Key: HADOOP-2573
                 URL: https://issues.apache.org/jira/browse/HADOOP-2573
             Project: Hadoop
          Issue Type: New Feature
          Components: mapred
            Reporter: Doug Cutting
             Fix For: 0.17.0


It should be possible to specify a limit to the number of tasks per job permitted to run simultaneously.  If, for example, you have a cluster of 50 nodes, with 100 map task slots and 100 reduce task slots, and the configured limit is 25 simultaneous tasks/job, then four or more jobs will be able to run at a time.  This will permit short jobs to pass longer-running jobs.  This also avoids some problems we've seen with HOD, where nodes are underutilized in their tail, and it should permit improved input locality.



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2573) limit running tasks per job

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557767#action_12557767 ] 

Doug Cutting commented on HADOOP-2573:
--------------------------------------

I think a static limit for all jobs would be useful and best to implement first.  After some experience with this, we would be better able to address its shortcomings.  Possible future extensions might be:
- dynamically altering the limit, e.g., limit=max(min.tasks.per.job, numSlots/numJobsOutstanding)
 -- ramping up the limit slowly, so that a users's sequential jobs don't have all their slots immediately taken when one job completes
 -- ramping down the limit slowly, so that tasks are given an opportunity to finish normally before they are killed.
- incorporating job priority into the limit


> limit running tasks per job
> ---------------------------
>
>                 Key: HADOOP-2573
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2573
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Doug Cutting
>             Fix For: 0.17.0
>
>
> It should be possible to specify a limit to the number of tasks per job permitted to run simultaneously.  If, for example, you have a cluster of 50 nodes, with 100 map task slots and 100 reduce task slots, and the configured limit is 25 simultaneous tasks/job, then four or more jobs will be able to run at a time.  This will permit short jobs to pass longer-running jobs.  This also avoids some problems we've seen with HOD, where nodes are underutilized in their tail, and it should permit improved input locality.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Issue Comment Edited: (HADOOP-2573) limit running tasks per job

Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557770#action_12557770 ] 

tdunning@veoh.com edited comment on HADOOP-2573 at 1/10/08 12:28 PM:
---------------------------------------------------------------

(oops... yes, doug anticipated this in his comment and I didn't read very well)

Presumably the limit could be made dynamic.  The limit could be max(static_limit, number of cores in cluster / # active jobs) 


On further reflection, I should note that my big jobs are all limited in pretty much the way that Doug suggests because they are processing a few large files that are unsplittable.  This limits the number of slots these big jobs can eat up.

The result is pretty OK.  My little jobs with lots of maps can slide through the cracks most of the time and everything runs pretty well.



      was (Author: tdunning@veoh.com):
    
Presumably the limit could be made dynamic.  The limit could be max(static_limit, number of cores in cluster / # active jobs) 

  
> limit running tasks per job
> ---------------------------
>
>                 Key: HADOOP-2573
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2573
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Doug Cutting
>             Fix For: 0.17.0
>
>
> It should be possible to specify a limit to the number of tasks per job permitted to run simultaneously.  If, for example, you have a cluster of 50 nodes, with 100 map task slots and 100 reduce task slots, and the configured limit is 25 simultaneous tasks/job, then four or more jobs will be able to run at a time.  This will permit short jobs to pass longer-running jobs.  This also avoids some problems we've seen with HOD, where nodes are underutilized in their tail, and it should permit improved input locality.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2573) limit running tasks per job

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557758#action_12557758 ] 

Doug Cutting commented on HADOOP-2573:
--------------------------------------

Some discussion of this issue may be found at:

http://www.nabble.com/question-about-file-glob-in-hadoop-0.15-tt14702242.html#a14741794


> limit running tasks per job
> ---------------------------
>
>                 Key: HADOOP-2573
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2573
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Doug Cutting
>             Fix For: 0.17.0
>
>
> It should be possible to specify a limit to the number of tasks per job permitted to run simultaneously.  If, for example, you have a cluster of 50 nodes, with 100 map task slots and 100 reduce task slots, and the configured limit is 25 simultaneous tasks/job, then four or more jobs will be able to run at a time.  This will permit short jobs to pass longer-running jobs.  This also avoids some problems we've seen with HOD, where nodes are underutilized in their tail, and it should permit improved input locality.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2573) limit running tasks per job

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557757#action_12557757 ] 

Arun C Murthy commented on HADOOP-2573:
---------------------------------------

I'd like to throw *job priority* into this festering pool...

At least changing the job-priority (done by the cluster-admin) should  in a change in number of max_slots... thoughts?



> limit running tasks per job
> ---------------------------
>
>                 Key: HADOOP-2573
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2573
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Doug Cutting
>             Fix For: 0.17.0
>
>
> It should be possible to specify a limit to the number of tasks per job permitted to run simultaneously.  If, for example, you have a cluster of 50 nodes, with 100 map task slots and 100 reduce task slots, and the configured limit is 25 simultaneous tasks/job, then four or more jobs will be able to run at a time.  This will permit short jobs to pass longer-running jobs.  This also avoids some problems we've seen with HOD, where nodes are underutilized in their tail, and it should permit improved input locality.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2573) limit running tasks per job

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557775#action_12557775 ] 

Doug Cutting commented on HADOOP-2573:
--------------------------------------

> The limit could be max(static_limit, number of cores in cluster / # active jobs)

Jinx!

> limit running tasks per job
> ---------------------------
>
>                 Key: HADOOP-2573
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2573
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Doug Cutting
>             Fix For: 0.17.0
>
>
> It should be possible to specify a limit to the number of tasks per job permitted to run simultaneously.  If, for example, you have a cluster of 50 nodes, with 100 map task slots and 100 reduce task slots, and the configured limit is 25 simultaneous tasks/job, then four or more jobs will be able to run at a time.  This will permit short jobs to pass longer-running jobs.  This also avoids some problems we've seen with HOD, where nodes are underutilized in their tail, and it should permit improved input locality.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2573) limit running tasks per job

Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557770#action_12557770 ] 

Ted Dunning commented on HADOOP-2573:
-------------------------------------


Presumably the limit could be made dynamic.  The limit could be max(static_limit, number of cores in cluster / # active jobs) 


> limit running tasks per job
> ---------------------------
>
>                 Key: HADOOP-2573
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2573
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Doug Cutting
>             Fix For: 0.17.0
>
>
> It should be possible to specify a limit to the number of tasks per job permitted to run simultaneously.  If, for example, you have a cluster of 50 nodes, with 100 map task slots and 100 reduce task slots, and the configured limit is 25 simultaneous tasks/job, then four or more jobs will be able to run at a time.  This will permit short jobs to pass longer-running jobs.  This also avoids some problems we've seen with HOD, where nodes are underutilized in their tail, and it should permit improved input locality.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2573) limit running tasks per job

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557724#action_12557724 ] 

Doug Cutting commented on HADOOP-2573:
--------------------------------------

This addresses issues raised in HADOOP-2510.

> limit running tasks per job
> ---------------------------
>
>                 Key: HADOOP-2573
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2573
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Doug Cutting
>             Fix For: 0.17.0
>
>
> It should be possible to specify a limit to the number of tasks per job permitted to run simultaneously.  If, for example, you have a cluster of 50 nodes, with 100 map task slots and 100 reduce task slots, and the configured limit is 25 simultaneous tasks/job, then four or more jobs will be able to run at a time.  This will permit short jobs to pass longer-running jobs.  This also avoids some problems we've seen with HOD, where nodes are underutilized in their tail, and it should permit improved input locality.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.