You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Scott Chen (JIRA)" <ji...@apache.org> on 2009/10/22 00:20:59 UTC

[jira] Created: (MAPREDUCE-1129) Assign multiple Map and Reduce tasks in Fairscheduler

Assign multiple Map and Reduce tasks in Fairscheduler
-----------------------------------------------------

                 Key: MAPREDUCE-1129
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1129
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: contrib/fair-share
    Affects Versions: 0.20.1
            Reporter: Scott Chen


In Hadoop-0.20, the period of heartbeat becomes much longer. 
Fairscheduler assigns at most one Map and one Reduce task per heartbeat.
This makes the cluster become very inefficient. Often time only half of the slots are used.

One idea is that we make Fairscheduler detect this situation (cluster under used) and scheduler more tasks in a heartbeat.
Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1129) Assign multiple Map and Reduce tasks in Fairscheduler

Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768562#action_12768562 ] 

Aaron Kimball commented on MAPREDUCE-1129:
------------------------------------------

Matei: On our roadmap. It's a tricky backport, so it's taking a while to test and bake in.

> Assign multiple Map and Reduce tasks in Fairscheduler
> -----------------------------------------------------
>
>                 Key: MAPREDUCE-1129
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1129
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/fair-share
>    Affects Versions: 0.20.1
>            Reporter: Scott Chen
>
> In Hadoop-0.20, the period of heartbeat becomes much longer. 
> Fairscheduler assigns at most one Map and one Reduce task per heartbeat.
> This makes the cluster become very inefficient. Often time only half of the slots are used.
> One idea is that we make Fairscheduler detect this situation (cluster under used) and scheduler more tasks in a heartbeat.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (MAPREDUCE-1129) Assign multiple Map and Reduce tasks in Fairscheduler

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon resolved MAPREDUCE-1129.
------------------------------------

    Resolution: Duplicate

Resolving as duplicate since this is in MAPREDUCE-706. Regarding the backport for Cloudera's distro, we will definitely include this in 0.20 in CDH3, and it's looking like we may backport for CDH2 in the coming weeks as well.

> Assign multiple Map and Reduce tasks in Fairscheduler
> -----------------------------------------------------
>
>                 Key: MAPREDUCE-1129
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1129
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/fair-share
>    Affects Versions: 0.20.1
>            Reporter: Scott Chen
>
> In Hadoop-0.20, the period of heartbeat becomes much longer. 
> Fairscheduler assigns at most one Map and one Reduce task per heartbeat.
> This makes the cluster become very inefficient. Often time only half of the slots are used.
> One idea is that we make Fairscheduler detect this situation (cluster under used) and scheduler more tasks in a heartbeat.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1129) Assign multiple Map and Reduce tasks in Fairscheduler

Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775347#action_12775347 ] 

Matei Zaharia commented on MAPREDUCE-1129:
------------------------------------------

Todd, are you guys also doing a backport for vanilla (Apache) Hadoop 0.20?

> Assign multiple Map and Reduce tasks in Fairscheduler
> -----------------------------------------------------
>
>                 Key: MAPREDUCE-1129
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1129
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/fair-share
>    Affects Versions: 0.20.1
>            Reporter: Scott Chen
>
> In Hadoop-0.20, the period of heartbeat becomes much longer. 
> Fairscheduler assigns at most one Map and one Reduce task per heartbeat.
> This makes the cluster become very inefficient. Often time only half of the slots are used.
> One idea is that we make Fairscheduler detect this situation (cluster under used) and scheduler more tasks in a heartbeat.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1129) Assign multiple Map and Reduce tasks in Fairscheduler

Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768532#action_12768532 ] 

Matei Zaharia commented on MAPREDUCE-1129:
------------------------------------------

This feature is implemented as part of MAPREDUCE-706. I think some Cloudera folks were porting this to Hadoop 0.20, but I'm not sure what the status is.

> Assign multiple Map and Reduce tasks in Fairscheduler
> -----------------------------------------------------
>
>                 Key: MAPREDUCE-1129
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1129
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/fair-share
>    Affects Versions: 0.20.1
>            Reporter: Scott Chen
>
> In Hadoop-0.20, the period of heartbeat becomes much longer. 
> Fairscheduler assigns at most one Map and one Reduce task per heartbeat.
> This makes the cluster become very inefficient. Often time only half of the slots are used.
> One idea is that we make Fairscheduler detect this situation (cluster under used) and scheduler more tasks in a heartbeat.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1129) Assign multiple Map and Reduce tasks in Fairscheduler

Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768500#action_12768500 ] 

Allen Wittenauer commented on MAPREDUCE-1129:
---------------------------------------------

We're seeing exactly the same issue.

> Assign multiple Map and Reduce tasks in Fairscheduler
> -----------------------------------------------------
>
>                 Key: MAPREDUCE-1129
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1129
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/fair-share
>    Affects Versions: 0.20.1
>            Reporter: Scott Chen
>
> In Hadoop-0.20, the period of heartbeat becomes much longer. 
> Fairscheduler assigns at most one Map and one Reduce task per heartbeat.
> This makes the cluster become very inefficient. Often time only half of the slots are used.
> One idea is that we make Fairscheduler detect this situation (cluster under used) and scheduler more tasks in a heartbeat.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1129) Assign multiple Map and Reduce tasks in Fairscheduler

Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768530#action_12768530 ] 

Zheng Shao commented on MAPREDUCE-1129:
---------------------------------------

One simple idea is to change the interface of LoadManager:

From:

[code}
boolean LoadManager.canAssignMap(...)
boolean LoadManager.canAssignReduce(...)
{code}

To:

[code}
int LoadManager.canAssignMapCount(...)
int LoadManager.canAssignReduceCount(...)
{code}

The LoadManager returns the number of map(or reduce) tasks that can be assigned to this TaskTracker, so we can assign multiple tasks in a single heartbeat.
The default implementation of the new functions can directly call the old functions to return 0 or 1, to provide backward compatibility with user-defined LoadManagers.


> Assign multiple Map and Reduce tasks in Fairscheduler
> -----------------------------------------------------
>
>                 Key: MAPREDUCE-1129
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1129
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/fair-share
>    Affects Versions: 0.20.1
>            Reporter: Scott Chen
>
> In Hadoop-0.20, the period of heartbeat becomes much longer. 
> Fairscheduler assigns at most one Map and one Reduce task per heartbeat.
> This makes the cluster become very inefficient. Often time only half of the slots are used.
> One idea is that we make Fairscheduler detect this situation (cluster under used) and scheduler more tasks in a heartbeat.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.