You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Xiao Kang (JIRA)" <ji...@apache.org> on 2010/04/06 04:32:27 UTC

[jira] Created: (MAPREDUCE-1674) Some new features for CapacityTaskScheduler

Some new features for CapacityTaskScheduler
-------------------------------------------

                 Key: MAPREDUCE-1674
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1674
             Project: Hadoop Map/Reduce
          Issue Type: New Feature
          Components: contrib/capacity-sched
    Affects Versions: 0.20.2
            Reporter: Xiao Kang


Some new features for CapacityTaskScheduler developed at Baidu.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1674) Some new features for CapacityTaskScheduler

Posted by "Xiao Kang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853721#action_12853721 ] 

Xiao Kang commented on MAPREDUCE-1674:
--------------------------------------

Sorry, wrong wiki syntext used, correct here:

# *support queue priority* : higher priority queue get capacity first
# *support queue refresh* : reload queue configuration at run time, including change queue properties and adding/removing queues
# *queue over capacity controlled* : add a queue property to indicate whether this queue can use more slot than its configured capacity
# *seperate web page for capacity-scheduler* : just as fair-scheduler, a new web page dadicated for capacity-scheduler is added
# *job capacity* : allowing user to specify per job map/reduce capacity both in cluster/tasktracker level
# *job initialization* : do not initialize a job if it will not get a chance to run if initialized


> Some new features for CapacityTaskScheduler
> -------------------------------------------
>
>                 Key: MAPREDUCE-1674
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1674
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/capacity-sched
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>
> Some new features for CapacityTaskScheduler developed at Baidu.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1674) Some new features for CapacityTaskScheduler

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853762#action_12853762 ] 

Hemanth Yamijala commented on MAPREDUCE-1674:
---------------------------------------------

Since Hadoop 0.21, there has been a good deal of changes in the capacity scheduler, primarily to support hierarchical queues. Since the 'Affects version' was set to an earlier version, I just thought of calling this out so you are aware.

> Some new features for CapacityTaskScheduler
> -------------------------------------------
>
>                 Key: MAPREDUCE-1674
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1674
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/capacity-sched
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>
> Some new features for CapacityTaskScheduler developed at Baidu.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1674) Some new features for CapacityTaskScheduler

Posted by "Xiao Kang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853720#action_12853720 ] 

Xiao Kang commented on MAPREDUCE-1674:
--------------------------------------

Details as follows:

# support queue priority : higher priority queue get capacity first

# support queue refresh : reload queue configuration at run time, including change queue properties and adding/removing queues

# queue over capacity controlled : add a queue property to indicate whether this queue can use more slot than its configured capacity

# seperate web page for capacity-scheduler : just as fair-scheduler, a new web page dadicated for capacity-scheduler is added

# job capacity : allowing user to specify per job map/reduce capacity both in cluster/tasktracker level

# job initialization : do not initialize a job if it will not get a chance to run if initialized


> Some new features for CapacityTaskScheduler
> -------------------------------------------
>
>                 Key: MAPREDUCE-1674
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1674
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/capacity-sched
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>
> Some new features for CapacityTaskScheduler developed at Baidu.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1674) Some new features for CapacityTaskScheduler

Posted by "Xiao Kang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854428#action_12854428 ] 

Xiao Kang commented on MAPREDUCE-1674:
--------------------------------------

Thank you for remind.

I have read about the interesting features in hadoop-0.21. We will follow the new features. Among the six features I listed above, 2 & 3 have been implemeted in hadoop-0.21.

Since great changes have been made to capacity-scheduler, QueueManager and TaskScheduler, it's difficult to migrate our improvents made against hadoop-0.20.2 to the latest version. If they are still valuable for the earlier version, sub task will be created for each.

> Some new features for CapacityTaskScheduler
> -------------------------------------------
>
>                 Key: MAPREDUCE-1674
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1674
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/capacity-sched
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>
> Some new features for CapacityTaskScheduler developed at Baidu.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1674) Some new features for CapacityTaskScheduler

Posted by "Xiao Kang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853723#action_12853723 ] 

Xiao Kang commented on MAPREDUCE-1674:
--------------------------------------

Creating a sub task for each feature may be more convient for discuss.

> Some new features for CapacityTaskScheduler
> -------------------------------------------
>
>                 Key: MAPREDUCE-1674
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1674
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/capacity-sched
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>
> Some new features for CapacityTaskScheduler developed at Baidu.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1674) Some new features for CapacityTaskScheduler

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853719#action_12853719 ] 

Arun C Murthy commented on MAPREDUCE-1674:
------------------------------------------

More details please? *smile*

> Some new features for CapacityTaskScheduler
> -------------------------------------------
>
>                 Key: MAPREDUCE-1674
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1674
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/capacity-sched
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>
> Some new features for CapacityTaskScheduler developed at Baidu.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.