You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2009/09/02 03:12:32 UTC
[jira] Created: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Extend FairShare scheduler to fair-share memory usage in the cluster
--------------------------------------------------------------------
Key: MAPREDUCE-944
URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/fair-share
Reporter: dhruba borthakur
The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur reassigned MAPREDUCE-944:
------------------------------------------
Assignee: dhruba borthakur
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: LoadManager.txt, LoadManager2.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated MAPREDUCE-944:
---------------------------------------
Tags: (was: fb)
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: LoadManager.txt, LoadManager2.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751777#action_12751777 ]
Matei Zaharia commented on MAPREDUCE-944:
-----------------------------------------
+1 looks good to me, except for one minor detail: I'd change the JavaDoc on canLaunchTask from "Can a given {@link TaskTracker} run another new task?" to "Can a given {@link TaskTracker} run another new task from a given job? This method is provided for use by LoadManagers that take into account jobs' individual resource needs when placing tasks." This way the purpose of the method, and its difference from canAssignMap/canAssignReduce, is better explained. It might also be good to look at the comments for canAssignMap/Reduce and make them say that these methods are used to check whether any attempt to find a job with maps/reduces should be made.
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Attachments: LoadManager.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated MAPREDUCE-944:
---------------------------------------
Attachment: LoadManager.txt
Implemented Matei's proposal. I think the current unit tests will test that this code refactoring does not break existing functionality.
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Attachments: LoadManager.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated MAPREDUCE-944:
---------------------------------------
Attachment: LoadManager2.txt
Incorporated Matie's review comments.
Vinod: The goal that we have in mind is slightly different from what the capacity scheduler has done (pl correct me if I am wrong). Unlike the capacity scheduler, there is no assumption that the user knows (upfront, before submitting job) how much memory/CPU/network that job will need. A realtime stream of resource usage will be fed into the new LoadManager that can then dynamically decide whether to run another task on that machine or not. Your feedback and experience vis-a-vis that Capacity scheduler is very valuable here... let's continue the conversation via MAPREDUCE-961
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Attachments: LoadManager.txt, LoadManager2.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12752105#action_12752105 ]
Vinod K V commented on MAPREDUCE-944:
-------------------------------------
I see in the patch attached that only one concrete implementation CapBasedLoadManager is done for the LoadManager which in turn doesn't take into account any resource usage. I guess you are planning a proper implementation for this feature regarding fair-share of memory usage in another JIRA.
Some points still not dealt with in this JIRA. I bring about these points so as to know if you are thinking or have already thought anything about this.
- Job configuration about how users specify the resource usage. Some memory related configuration properties are added to the framework while working for memory monitoring on TTs as well as memory usage based scheduling in CapacityTaskScheduler. You may want to reuse some/all of it.
- Capturing the scheduling decisions involved when we are not able to find a task from a Schedulable because of lack of resources on a given TaskTasker.
Regarding the latter, the current patch just returns null, which is similar to the decision CapacityTaskScheduler used to take in previous versions - i.e. block the TT till it can be given a task from the job at the head of the queue/pool. Sometime back, we investigated how this approach works with FairScheduler and realized some important implications. For e.g, because the order of jobs might change significantly in consecutive iterations of FairScheduler, just returning null may not work at all. Eventually we may end up waiting for a long time if significant number of jobs ask for high amount of resources.
Thoughts?
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Attachments: LoadManager.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750822#action_12750822 ]
dhruba borthakur commented on MAPREDUCE-944:
--------------------------------------------
+1 to Matie's proposal.
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12752720#action_12752720 ]
Scott Chen commented on MAPREDUCE-944:
--------------------------------------
I wonder where why there are test errors. canLaunchTask only returns true so it shouldn't change anything. I have applied this patch and run ant clean test. It shows no test error.
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: LoadManager.txt, LoadManager2.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750775#action_12750775 ]
Matei Zaharia commented on MAPREDUCE-944:
-----------------------------------------
I think that rather than passing a LoadManager to the Schedulables and having them check whether they can run, it's better design to put all the logic in the LoadManager and ask it whether it's okay to run a particular job. I'd do this the following way:
* LoadManager contains a method called canLaunchTask(JobInProgress job, TaskType type, TaskTracker Status tt)
* In JobSchedulable.assignTasks, before looking for a task, the JobSchedulable first checks whether scheduler.getLoadManager().canLaunchTask(job, type, tt) is true. If it isn't,
* The default implementation of canLaunchTask always returns true, but in the memory-aware LoadManager, it can be made to return false.
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750221#action_12750221 ]
dhruba borthakur commented on MAPREDUCE-944:
--------------------------------------------
One simple way to do this would be to pass in the LoadManager object into the Schedulable.assignTask().
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated MAPREDUCE-944:
---------------------------------------
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: LoadManager.txt, LoadManager2.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753503#action_12753503 ]
dhruba borthakur commented on MAPREDUCE-944:
--------------------------------------------
This is not really a bug fix. It is mostly a refactoring of the code so that a certain functionality is separated out more cleanly. HADOOP-961 will benefit from this JIRA and it will have quite a few unit tests associated with it.
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: LoadManager.txt, LoadManager2.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-944:
------------------------------------
Tags: (was: fb)
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: LoadManager.txt, LoadManager2.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753174#action_12753174 ]
Konstantin Boudnik commented on MAPREDUCE-944:
----------------------------------------------
It'd be good to have a note on why there's no new tests added within that fix.
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: LoadManager.txt, LoadManager2.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12752976#action_12752976 ]
Hadoop QA commented on MAPREDUCE-944:
-------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12418880/LoadManager2.txt
against trunk revision 812546.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/52/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/52/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/52/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/52/console
This message is automatically generated.
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: LoadManager.txt, LoadManager2.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated MAPREDUCE-944:
--------------------------------
Status: Open (was: Patch Available)
The test failures were due to HADOOP-6243, which is fixed now. Resubmitting to Hudson should do.
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: LoadManager.txt, LoadManager2.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated MAPREDUCE-944:
--------------------------------
Status: Patch Available (was: Open)
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: LoadManager.txt, LoadManager2.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753516#action_12753516 ]
Hudson commented on MAPREDUCE-944:
----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #28 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/28/])
. Extend the LoadManager API of the fair-share scheduler
to support regulating tasks for a job based on resources currently in use
by that job. (dhruba)
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: LoadManager.txt, LoadManager2.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12752365#action_12752365 ]
Hadoop QA commented on MAPREDUCE-944:
-------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12418880/LoadManager2.txt
against trunk revision 812287.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/console
This message is automatically generated.
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: LoadManager.txt, LoadManager2.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-944) Extend FairShare scheduler to
fair-share memory usage in the cluster
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-944:
------------------------------------
Tags: fb
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: LoadManager.txt, LoadManager2.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.