You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Patrick Wendell (Created) (JIRA)" <ji...@apache.org> on 2011/11/22 05:04:40 UTC
[jira] [Created] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Port Fair Scheduler to MR2
--------------------------
Key: MAPREDUCE-3451
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: scheduler
Reporter: Patrick Wendell
The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413905#comment-13413905 ]
Arun C Murthy commented on MAPREDUCE-3451:
------------------------------------------
Digging more - looks like I don't see the fair-scheduler tests show up in hadoop-yarn-server-resourcemanager-3.0.0-SNAPSHOT-tests.jar.
Is that the reason why we don't see a compilation failure even though my IDE complains?
Some maven work remaining, perhaps?
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254273#comment-13254273 ]
Aaron T. Myers commented on MAPREDUCE-3451:
-------------------------------------------
bq. For the documentation pointer (hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/) - I'm unfamiliar with the markup being used in these documents (it looks like the Velocity engine maybe?).
Take a look at this page for info on the markup: http://maven.apache.org/doxia/references/apt-format.html
bq. Is there any way to render these files locally as I'm writing some preliminary docs for the FS?
>From the repo root, `cd hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site' and run `mvn site'. You can then look at the rendered page by browsing to "file:///<path to repo>/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/target/site/<file name>.html"
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276225#comment-13276225 ]
Matei Zaharia commented on MAPREDUCE-3451:
------------------------------------------
Having a single very feature-rich scheduler is certainly useful, but that's a separate discussion. My point was just that some people want compatibility with the MR1 Fair Scheduler as a feature -- not having that is an obstacle to their adopting MR2.
I also think it would be hard to design a single scheduler that will support all the features people might want in the future, for the same reason that it was hard to get different development groups to build a single scheduler in MR1. You just can't anticipate all the needs, and how various features will interact -- for example, Yahoo! had to remove preemption from the capacity scheduler because it complicated the design. We are also planning to add some features to the Fair Scheduler ourselves, such as multi-resource fairness, that might require major changes to other schedulers even if we were to wait on them adopting the existing features.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257986#comment-13257986 ]
Hadoop QA commented on MAPREDUCE-3451:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522923/MAPREDUCE-3451.v3.patch.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 21 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapreduce.security.TestJHSSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2259//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2259//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2259//console
This message is automatically generated.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159619#comment-13159619 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
I have a few questions about the way that schedulers are implemented in MR2.
It's not clear to me whether the Scheduler class used by the Resource Manager is responsible for only inter-application scheduling or both inter- ant intra-application scheduling. It looks to me like only the former is supported, because the resource requests (via allocate()) do not give any meta-data regarding the user initiating the request. So if we wanted to enforce Fair Sharing for jobs requested from a single AM, this doesn't seem possible. When I look at the Capacity Scheduler, for instance, it seems like allocation requests are assigned to queues based only on the application from which they are made, not the actual user launching the job.
Is the idea that, in a shared cluster, each administrative unit has its own MR Application Master running?
I have some follow-up but first want to see if this is actually a correct interpretation of the design spec.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159847#comment-13159847 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
Ah, perhaps my issue is a broader misunderstanding of the MR2 architecture.
I thought the Application Master was a long lived service for a particular application (given its name). I.e. for MapReduce you would expect to have a single AM running and coordinating submissions for several jobs. Your last sentence seems to suggest this is incorrect. Is a new AM instantiated for *each* MR job?
Also, you mentioned priorities. I can't find any details on the scheduling semantic associated with priorities. I'm assuming those are supposed to convey the scheduler ordering for requests from a particular AM? I.e. all priority 1 containers should be scheduled before the first priority 2 container? This made no sense to me before, but it makes more sense if there is a new AM for each job.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413477#comment-13413477 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
We have several pieces of follow-up around this, and one of them can be looking at moving SchedulerApp to an interface (this would involve changing the other two schedulers as well to some extent).
I'd like to see that change occur in tandem with better documentation for common scheduling components like SchedulerApp. The design goals of the RM scheduling code are currently opaque to anyone trying to add new code. Better documentation would make it clearer how to hook in a new scheduler the right way.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412408#comment-13412408 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
So I spent some time offline talking with Tom and I think we're ready to move forward with committing this to MR2. The primary goals are:
1) Let people who use the MR1 fair scheduler have a drop in equivalent for MR2
- and -
2) Keep as much code shared as possible between MR2 schedulers given that the overlap in substantial functionality
I think we're clearly accomplishing (1) and well on our way to (2), but subsequent patches might find more opportunity for code re-use.
This also doesn't include some necessities like the Web UI and some nice-to-have extensions like hierarchical queues. That said I'd like to get this committed as a starting point build from there, given that it's already a pretty large commit.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed MAPREDUCE-3451.
------------------------------------
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.2-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated MAPREDUCE-3451:
---------------------------------------
Attachment: MAPREDUCE-3451.v4.patch.txt
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated MAPREDUCE-3451:
---------------------------------------
Component/s: mrv2
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413401#comment-13413401 ]
Hudson commented on MAPREDUCE-3451:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2526 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2526/])
MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 1361020)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361020
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413878#comment-13413878 ]
Arun C Murthy commented on MAPREDUCE-3451:
------------------------------------------
Unfortunately we've introduced findbugs warnings, Alejandro you missed them?
I'll file another jira.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279465#comment-13279465 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
I'm not sure I understand what it would mean to have a single featureful scheduler.
There is already an abstraction layer built into YARN to allow multiple scheudling policies. Aside from the logic specific to the policy itself, the code for dealing with resource management is re-used amongst schedulers. That is, when someone is using the Fair Scheduler (vs the Capacity or FIFO scheduler) they are using mostly the same RM code save for the scheduling logic itself. Furthermore, several shared classes are used by both the Fair and Capacity schedulers, such as the Queue interface and the SchedulerApp class - these common classes are "tested" by users of both schedulers.
The alternative suggested seems to be having a monolithic scheduler class that can enact different policies depending on some configuration option. Any sensible implementation of that approach would abstract out the scheduling policy and I think you'd get what's already there now.
Or maybe the idea is that everyone using Hadoop will want the same scheduling policy (modulo minor configurations like timeouts, capacities, etc.). That seems unlikely to me given the diversity of Hadoop deployments and the fact that both of the currently available schedulers in MR1 are widely used.
Also, just a note - adding preemption to the capacity scheduler will not make it equal the Fair Scheduler. These are fundamentally different approaches which will become substantailly more different when multiple resource types are added to the YARN RM stack.
I strongly agree that having single implementations of common classes is a win in terms of support and testability - but I think this implementation mostly acheives that goal.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414015#comment-13414015 ]
Arun C Murthy commented on MAPREDUCE-3451:
------------------------------------------
Patrick - the last comment/advice was to supress them (from Harsh). Why weren't they?
Now, all patch builds are failing complaining about the findbugs warnings...
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254266#comment-13254266 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
Hey Tom/Ahmed,
Thanks for the comments - I'm addressing them now. For the documentation pointer (hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/) - I'm unfamiliar with the markup being used in these documents (it looks like the Velocity engine maybe?). Is there any way to render these files locally as I'm writing some preliminary docs for the FS?
- Patrick
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261472#comment-13261472 ]
Luke Lu commented on MAPREDUCE-3451:
------------------------------------
bq. It seems like it’s not easy to trigger a 404 from within a view, rather than at the URL router.
You can trigger a 404 or any http response code anywhere (preferably in controller code as view is not guaranteed to be fully buffered for long responses by design) by calling Controller#setStatus (via context().setStatus in views (if it's not yet flushed) and then throw an WebAppException (do not use generic RuntimeException). You can bind ErrorPage.class in a webapp to any view class if you want a custom error page.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414163#comment-13414163 ]
Hudson commented on MAPREDUCE-3451:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2486 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2486/])
MAPREDUCE-3451. Amendment, excluding findbugs warnings (tucu) (Revision 1361436)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361436
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413998#comment-13413998 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
It's a one line change to the package header. The findbugs issues are discussed further up in this Jira and were also discussed during the last round of reviews.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413424#comment-13413424 ]
Hudson commented on MAPREDUCE-3451:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2479 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2479/])
MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 1361020)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361020
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-3451:
-------------------------------
Attachment: MAPREDUCE-3451.v5.patch
Per MAPREDUCE-3812, I've tweaked your patch a bit to make it use the new configs for min/max RM allocation (memory for containers).
One thing I noticed when tweaking your patch, is that there are quite a bit of trailing white-spaces and possibly a few occurrences of tab+space mix you may wanna clean up.
The diff I've added to your patch overall is below:
{code}
diff --git a/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
index 35eafc5..e888be8 100644
--- a/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
+++ b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
@@ -4,6 +4,7 @@ import java.io.File;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.server.resourcemanager.resource.Resources;
public class FairSchedulerConfiguration extends Configuration {
@@ -13,15 +14,7 @@ public class FairSchedulerConfiguration extends Configuration {
protected static final String ALLOCATION_FILE = CONF_PREFIX + "allocation.file";
protected static final String EVENT_LOG_DIR = "eventlog.dir";
-
- /** The minimum size container the scheduler will ever allocate. */
- protected static final String MINIMUM_MEMORY_MB = CONF_PREFIX + "minimum-allocation-mb";
- protected static final int DEFAULT_MINIMUM_MEMORY_MB = 512;
-
- /** The maximum size container the scheduler will ever allocate. */
- protected static final String MAXIMUM_ALLOCATION_MB = CONF_PREFIX + "maximum-allocation-mb";
- protected static final int DEFAULT_MAXIMUM_ALLOCATION_MB = 10240;
-
+
/** Whether to use the user name as the queue name (instead of "default") if
* the request does not specify a queue. */
protected static final String USER_AS_DEFAULT_QUEUE = CONF_PREFIX + "user-as-default-queue";
@@ -60,17 +53,21 @@ public class FairSchedulerConfiguration extends Configuration {
super(conf);
addResource(FS_CONFIGURATION_FILE);
}
-
+
public Resource getMinimumMemoryAllocation() {
- int mem = getInt(MINIMUM_MEMORY_MB, DEFAULT_MINIMUM_MEMORY_MB);
+ int mem = getInt(
+ YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
+ YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
return Resources.createResource(mem);
}
-
+
public Resource getMaximumMemoryAllocation() {
- int mem = getInt(MAXIMUM_ALLOCATION_MB, DEFAULT_MAXIMUM_ALLOCATION_MB);
+ int mem = getInt(
+ YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB,
+ YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB);
return Resources.createResource(mem);
}
-
+
public boolean getUserAsDefaultQueue() {
return getBoolean(USER_AS_DEFAULT_QUEUE, DEFAULT_USER_AS_DEFAULT_QUEUE);
}
{code}
bq. Okay looks like that patch fixed everything except for the findbugs, there are some member variables which are written only while holding a lock but can be read outside of the lock and findbugs doesn't like this. I consider these false positives - any idea how to suppress?
I didn't take a look at which specific mem-var you're discussing here, but what happens if one reads it when its being updated? Does your locking mech cover this scenario too? I remember making the same mistake a while ago at https://issues.apache.org/jira/browse/MAPREDUCE-1347?focusedCommentId=13003257&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13003257 but that may be irrelevant here.
In case you wish to add it to ignores, see file {{hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml}} and add appropriate entries for your discovered warnings to suppress them. I'd recommend suppressing them after the review/over another ticket and just leave as a comment for now that this may not be harmful as it looks.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276203#comment-13276203 ]
Tom White commented on MAPREDUCE-3451:
--------------------------------------
Matei - I'm not against this going in (although we should put it in a separate module), but I thought it would be a good time to have a discussion about supporting a single main scheduler in YARN. If we could combine features from the various schedulers in MR1 to YARN then we would have less code to support and test, and it would be less confusing for users. What do you think? Do you agree that this would be a useful goal?
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur resolved MAPREDUCE-3451.
-------------------------------------------
Resolution: Fixed
Fix Version/s: 2.0.1-alpha
Nice job, thanks Patrick. Committed to trunk and branch-2.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3451:
-------------------------------------
Status: Open (was: Patch Available)
Patrick I took a brief look, mostly looks great! I agree having different schedulers is very useful.
Minor nits: I don't think it's worth modifying common infrastructure such as SchedulerApp right now - maybe you can just extend them where necessary?
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178242#comment-13178242 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
This JIRA has been chunked up into smaller JIRAs to make the patch submission and review more sane.
I've attached a patch for the first of these sub-tasks.
https://issues.apache.org/jira/browse/MAPREDUCE-3600
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated MAPREDUCE-3451:
---------------------------------------
Attachment: MAPREDUCE-3451.v1.patch.txt
This is patch aggregating all sub-ticket patches and also bringing up to date with trunk.
This patch is ready for review, thanks.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258375#comment-13258375 ]
Ahmed Radwan commented on MAPREDUCE-3451:
-----------------------------------------
Thanks Patrick for the comments, +1
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414132#comment-13414132 ]
Hudson commented on MAPREDUCE-3451:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #2466 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2466/])
MAPREDUCE-3451. Amendment, excluding findbugs warnings (tucu) (Revision 1361436)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361436
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412423#comment-13412423 ]
Tom White commented on MAPREDUCE-3451:
--------------------------------------
Thanks for summarizing the goals here Patrick. I agree that having a single scheduler is not realistic at this point.
The latest change (and test) supporting the "pool" element in the allocation XML file for backwards compatibility looks good to me.
+1 (pending Jenkins)
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161270#comment-13161270 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
@Mahedev and others
A follow up question. In the MR1 code, there was a way to assign a priority to a specific job. This let users whose jobs all go to the same queue give more importance to certain jobs over others and it is used in the fair scheduler.
The MR2 code does seem to have similar support, i.e. per-AM priorities. For instance, in ApplicationSubmissionContext we see:
{noformat}
/**
* Set the <code>Priority</code> of the application.
* @param priority <code>Priority</code> of the application
*/
@Public
@Stable
public void setPriority(Priority priority);
{noformat}
The MR AM also seems to support per-job prioritization, though it doesn't look like this code is called anywhere.
{noformat}
/**
* Set the priority of a running job.
* @param priority the new priority for the job.
* @throws IOException
*/
public void setPriority(JobPriority priority)
throws IOException, InterruptedException {
{noformat}
Such priorities are *not*, however, passed downstream to the scheduler as far as I can see (they aren't reflected in SchedulerApp class). I can just patch the code to pass these to the scheduler, but I'm curious whether there is something I'm missing here, especially given the last comment which seems to contradict that priorities can be applied across jobs.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412848#comment-13412848 ]
Arun C Murthy commented on MAPREDUCE-3451:
------------------------------------------
In fact, we should probably make SchedulerApp etc. an interface and have different implementations for CS, FS etc. to minimize interference. Thoughts?
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271143#comment-13271143 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
Okay looks like that patch fixed everything except for the findbugs, there are some member variables which are written only while holding a lock but can be read outside of the lock and findbugs doesn't like this. I consider these false positives - any idea how to suppress?
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413997#comment-13413997 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
Yes the error is related to the last minute movement of that test into the Fair package.
Karthik - could you quickly patch this so we can get the build stabilized? If you can fix the findbugs quickly that would be great too - as i said earlier these are false positives but we might be able to coerce findbugs into not spouting warnings.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413379#comment-13413379 ]
Alejandro Abdelnur commented on MAPREDUCE-3451:
-----------------------------------------------
+1
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414024#comment-13414024 ]
Alejandro Abdelnur commented on MAPREDUCE-3451:
-----------------------------------------------
I'm on amending the patch
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413915#comment-13413915 ]
Karthik Kambatla commented on MAPREDUCE-3451:
---------------------------------------------
Do you suggest creating a clean-up JIRA and addressing these issues?
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241407#comment-13241407 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
Hey Ahmed,
MR2 does support multiple resource types, but as you point out, currently only memory is implemented. Right now this scheduler basically treats memory similarly to the way slots were handled in the old fair scheduler, with the only key difference being that memory can be allocated in variable size chunks.
There is no widely accepted definition of what "fairness" means with multiple resource types, but there have been proposals from the academic community (see http://static.usenix.org/events/nsdi11/tech/full_papers/Ghodsi.pdf) which could ultimately be factored into the scheduler if and when multiple resource types are introduced. Having several resource dimensions complicates the scheduling decision substantially and will require major changes to existing schedulers (such as the capacity scheduler) as well. In all of these cases, existing approaches depend on a single fungible resource about which you can express minimum capacities (or shares), target capacities (or shares), etc. These semantics don't make sense with multiple resource types.
Right now the preemption logic preempts lowest priority containers first. This is the best general-purpose metric I could think of for choosing what to preempt.
I will include a write-up about deployment in the scheduler's current form as well in an upcoming patch.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated MAPREDUCE-3451:
---------------------------------------
Attachment: MAPREDUCE-3451.v2.patch.txt
This patch should apply cleanly to trunk, thanks.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Matei Zaharia (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia reassigned MAPREDUCE-3451:
----------------------------------------
Assignee: Patrick Wendell
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413906#comment-13413906 ]
Arun C Murthy commented on MAPREDUCE-3451:
------------------------------------------
bq. Digging more - looks like I don't see the fair-scheduler tests show up in hadoop-yarn-server-resourcemanager-3.0.0-SNAPSHOT-tests.jar.
Spoke too soon, I see the following:
{noformat}
$ jar -tvf ./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/hadoop-yarn-server-resourcemanager-3.0.0-SNAPSHOT-tests.jar | grep -i fair
0 Fri Jul 13 10:37:20 PDT 2012 org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/
324 Fri Jul 13 10:37:20 PDT 2012 org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler$1.class
1463 Fri Jul 13 10:37:20 PDT 2012 org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler$MockClock.class
25975 Fri Jul 13 10:37:20 PDT 2012 org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.class
{noformat}
However, I don't see TestFSSchedulerApp, not sure why I don't see a compilation failure given the wrong package declaration.
Patrick - any idea?
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159870#comment-13159870 ]
Mahadev konar commented on MAPREDUCE-3451:
------------------------------------------
@Patrick,
Yes the priorities are for the requests by a single AM. There are no priorities across jobs(/AM's). A DAG of MR jobs in a single AM is very much a possibility and we have been talking to PIG/Oozie folks for such a possibility, but as of now an AM maps to a single job.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Tom White (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-3451:
---------------------------------
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413398#comment-13413398 ]
Hudson commented on MAPREDUCE-3451:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #2460 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2460/])
MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 1361020)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361020
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159848#comment-13159848 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
Ah, perhaps my issue is a broader misunderstanding of the MR2 architecture.
I thought the Application Master was a long lived service for a particular application (given its name). I.e. for MapReduce you would expect to have a single AM running and coordinating submissions for several jobs. Your last sentence seems to suggest this is incorrect. Is a new AM instantiated for *each* MR job?
Also, you mentioned priorities. I can't find any details on the scheduling semantic associated with priorities. I'm assuming those are supposed to convey the scheduler ordering for requests from a particular AM? I.e. all priority 1 containers should be scheduled before the first priority 2 container? This made no sense to me before, but it makes more sense if there is a new AM for each job.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257952#comment-13257952 ]
Tom White commented on MAPREDUCE-3451:
--------------------------------------
Patrick, thanks for addressing the feedback and also for adding documentation. I think it's fine to improve test coverage and revisit the configuration format in follow on JIRAs, so I'm +1 for committing this. I'm going to run this through Jenkins now.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241368#comment-13241368 ]
Ahmed Radwan commented on MAPREDUCE-3451:
-----------------------------------------
Thanks Patrick.
The patch successfully applies and compiles on trunk. The included new tests also successfully run.
I'll postpone specific code comments, since I have some basic questions:
The old fair scheduler used slots as the basic allocation units, and calculating shares and other features (like preemption) relied on this model. In MR2 we have containers with variable resource attributes. Currently the only supported attribute is memory, but it is planned to add more attributes (network bandwidth, cpu, etc.). So it is not clear for me how this mapping is handled. Are you envisioning combing such attributes for calculating shares, how is this done? and how preemption will be done in such case, for example in the old MR1, the most recent slot was killed. But now slots are not equal and this complicate the choices. I am just using preemption as an example. Can you please elaborate on that?
I also see that the patch doesn't include the web ui components. I think postponing them to follow-up work is fine. But if someone wants to test the patch on a real cluster, can you please highlight the steps needed to setup a cluster using the new fair scheduler and if there are changes in setup compared to the old MR1 version.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated MAPREDUCE-3451:
---------------------------------------
Attachment: MAPREDUCE-3451.v3.patch.txt
New patch including response to feedback and documentation file.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413901#comment-13413901 ]
Arun C Murthy commented on MAPREDUCE-3451:
------------------------------------------
I'm seeing another problem after I did a 'git pull' today after this commit.
My IDE complains bitterly that TestFSSchdulerApp has the wrong package declaration.
[~tucu00]: Should we revert this and fix the package problem and the findbugs warnings?
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated MAPREDUCE-3451:
---------------------------------------
Attachment: MAPREDUCE-3451.v6.patch
Bringing this back to trunk again and adding one small change to make config files more backwards incompatible.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated MAPREDUCE-3451:
---------------------------------------
Attachment: MAPREDUCE-3451.v8.patch
This patch extends rather than modifies the SchedulerApp class for use in the Fair Scheduler.
I cleaned up some tabs/whitespace issues in SchedulerApp.java and those are retained in this patch. But there are no code changes to SchedulerApp.java.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412941#comment-13412941 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
Hey Arun - I think extending it is a better move for now.
If we decided to move SchedulerApp to an interface, 95% of the implementation would be the same between the FS/CS. The only differences are in the way delay scheduling works for the FairScheduler.
As far as I can tell - the idea behind SchedulerApp is to deal with application lifecycle management from the perspective of the scheduler. That's helpful to have and made writing the FS simpler. This is likely to be useful for future schedulers well. In terms of maximizing code sharing I would be in favor of keeping it collectively managed.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250886#comment-13250886 ]
Ahmed Radwan commented on MAPREDUCE-3451:
-----------------------------------------
Hi Patrick,
There are also these comments/nits:
- RMContainer.java: Added a unused "import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl;"
- Many of the inline docs have spelling mistakes. Please review them.
- SchedulerApp.java: The following was removed from the docs: [Each time the scheduler
asks the application for a task at this priority, it is incremented,
and each time the application successfully schedules a task, it
is reset to 0.], please add it back.
- Add javadocs and setup and configuration docs.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated MAPREDUCE-3451:
---------------------------------------
Attachment: MAPREDUCE-3451.v7.patch
Forgot to include some whitespace fixes in last patch
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414133#comment-13414133 ]
Hudson commented on MAPREDUCE-3451:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2532 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2532/])
MAPREDUCE-3451. Amendment, excluding findbugs warnings (tucu) (Revision 1361436)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361436
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413914#comment-13413914 ]
Karthik Kambatla commented on MAPREDUCE-3451:
---------------------------------------------
Arun, in the last version of the patch, Patrick moved TestFSSchdulerApp from o.a.h.y.s.rm.scheduler to o.a.h.y.s.rm.scheduler.fair. He might have forgotten to update the package name accordingly. From what I understand, because of the wrong package name, the class file is still created one level above.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159860#comment-13159860 ]
Todd Lipcon commented on MAPREDUCE-3451:
----------------------------------------
Hi Patrick. Yes, there is an AM-per-job in the current implementation. There's some folks I know looking at bundling full workflows into a single AM, but they would still be fairly short-lived and associated with a single submitting user.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413457#comment-13413457 ]
Arun C Murthy commented on MAPREDUCE-3451:
------------------------------------------
bq. As far as I can tell - the idea behind SchedulerApp is to deal with application lifecycle management from the perspective of the scheduler.
I wish I had a chance to respond to Patrick before this was committed... IAC, we should plan to allow the schedulers to deviate in significant ways (which is the whole point of having multiple schedulers) and, as a result, minimizing interference is a key goal.
Maybe we should open a jira to make SchedulerApp an interface?
I originally kept it common across FifoScheduler and CapacityScheduler since, for the degenerate case of a single queue, they are identical (currently) and planned to pull it out as a separate one once they deviated.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412422#comment-13412422 ]
Hadoop QA commented on MAPREDUCE-3451:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536147/MAPREDUCE-3451.v7.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2576//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2576//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2576//console
This message is automatically generated.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155304#comment-13155304 ]
Arun C Murthy commented on MAPREDUCE-3451:
------------------------------------------
Thanks for taking this up Patrick. Let me know if you need help.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413670#comment-13413670 ]
Hudson commented on MAPREDUCE-3451:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1102 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1102/])
MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 1361020)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361020
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270993#comment-13270993 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
Just added a patch addressing the legitimate findbugs and javadoc warnings. There were some findbugs warnings about concurrency that seemed like false positives - not sure if there is a way to suppress them.
I can't re-create the core test failures locally... they seem related to port contention issues and Kerberos issues around the hostname of the Hudson machine. I don't see any direct relationship between these test failures and the code in this patch. Could we at least re-run this in Hudson to make sure that these tests are deterministically failing?
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255396#comment-13255396 ]
Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------
Thanks for the feedback. About to attach a patch with the following changes:
- Updated the terminology from “pool” to “queue” to be consistent.
- I added annotations to all classes.
- Fixed the debug → log statement
- I modified the web UI to give a more precise error message in the logs, but it still has a generic 500 in the web page. MRv2 uses a custom built MVC framework based on Guice dependency injection. I’m embarrassed to say that I spend 40 minutes trying to figure out how to make it render a 404 and couldn’t. It seems like it’s not easy to trigger a 404 from within a view, rather than at the URL router.
- Extra license header removed
- Noted spelling errors fixed.
- There is extensive test coverage in the MRS FS I’d like to get there eventually but probably not with this first commit. I tried to included tests of most of the core functionality. We can add this as a JIRA, though.
- I consolidated the configuration constants as you suggested. For now I’m going to leave the queue manifest file as its own thing unless this is a deal-breaker. There is a lot of logic in parsing that file which I’d prefer to leave as-is since it’s very well tested. While the term names have changed it’s essentially identical configuration to the MR1 FS, so leaving things as-is should ease the transition for people running current clusters. It’s also a fundamentally hierarchical type of configuration, and the apache conf is all flat key/value space which will be much more clunky.
I haven’t done much testing other than on a single node. One reason I want to get this in is that some are waiting on having something in trunk to start testing. The build is stable and the unit tests pass, so I hope that is sufficient to get this in there with the caveat that things may be brittle when run on a real cluster.
I also added an updated version of the documentation (even the MR1 version wasn’t totally up-to-date with changes that have been made in the configuration options). This will need some more work going forward but it should give people to the basic configuration/installation info they need to get started testing this.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270994#comment-13270994 ]
Hadoop QA commented on MAPREDUCE-3451:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12526086/MAPREDUCE-3451.v4.patch.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2369//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2369//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2369//console
This message is automatically generated.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated MAPREDUCE-3451:
---------------------------------------
Attachment: MAPREDUCE-3451.v9.patch
This patch moves a TestFSSchedulerApp into the fair/ test subdirectory.
It also reverts whitespace fixes in SchedulerApp.java (so that file is now totally unchanged from trunk). That is out of scope for this JIRA.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414409#comment-13414409 ]
Hudson commented on MAPREDUCE-3451:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1136 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1136/])
MAPREDUCE-3451. Amendment, excluding findbugs warnings (tucu) (Revision 1361436)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361436
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Patrick Hunt (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231469#comment-13231469 ]
Patrick Hunt commented on MAPREDUCE-3451:
-----------------------------------------
Patrick unfortunately while the patch applies to the current trunk, it does not compile. Could you update? Thanks.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159845#comment-13159845 ]
Mahadev konar commented on MAPREDUCE-3451:
------------------------------------------
Patrick,
I'd say it could do both. If you look at the Capacity Scheduler, it does inter application scheduling (deciding on which application gets a resource) and then also decides which priority request within the application should be satisfied first (in CS its the highest first).
Take a look at
{code}
LeafQueue.assignContainers()
ParentQueue.assignContainers()
{code}
As for the user information, its is available in the SchedulerApp.getUser().
Also, a single AM amounts to a single job currently, I am not sure what you mean by "jobs requested for a single AM".
Hope that helps.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271317#comment-13271317 ]
Hadoop QA commented on MAPREDUCE-3451:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12526140/MAPREDUCE-3451.v5.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2371//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2371//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2371//console
This message is automatically generated.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414355#comment-13414355 ]
Hudson commented on MAPREDUCE-3451:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1103 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1103/])
MAPREDUCE-3451. Amendment, excluding findbugs warnings (tucu) (Revision 1361436)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361436
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414173#comment-13414173 ]
Zhihong Ted Yu commented on MAPREDUCE-3451:
-------------------------------------------
HBASE-6395 has been created for putting TestFSSchedulerApp in the right package.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413903#comment-13413903 ]
Arun C Murthy commented on MAPREDUCE-3451:
------------------------------------------
bq. My IDE complains bitterly that TestFSSchdulerApp has the wrong package declaration.
Also, my IDE complains that Queue needs an import.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414176#comment-13414176 ]
Zhihong Ted Yu commented on MAPREDUCE-3451:
-------------------------------------------
MAPREDUCE-4445 is the correct JIRA, sorry about this.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248692#comment-13248692 ]
Tom White commented on MAPREDUCE-3451:
--------------------------------------
Overall this looks like a great addition. Initial review feedback:
* It looks like what were called "pools" in the MR1 version are now "queues", to fit with the term that the capacity scheduler uses. Is that correct? If so, then the fair scheduler code should use the same term throughout.
* What's the locking order for the fair scheduler classes and the classes that call them? It might be worth documenting it somewhere in the code.
* I noticed that not all the unit tests from MR1 have been ported. Are you planning on including the rest?
* Configuration strings are duplicated and spread through different classes. How about creating a FairSchedulerConfiguration class to hold all the constants (like capacity scheduler does)?
* Seeing that the configuration is changing compared to MR1, is it worth changing the allocations file to be in Hadoop configuration format? Having one less format to deal with would be an improvement IMO.
* Annotate all classes as @Private @Unstable. Some are not annotated at all, even though they should be considered private.
* PoolSchedulable.assignContainer has a log line which is called frequently ("Node offered to pool"), so should really be debug level.
* The Scheduler UI link gives a 500 error - I know you plan on fixing this, but I would have expected a 404.
* Nit: PoolManager has two license headers
* Nit: Resources class spelling: "Mutliply"
* Nit: SchedulerApp - schedulingOpportunities comment has been changed from javadoc comment in MR1 - change back?
What testing have you done? I tried it on a pseudo distributed cluster and successfully ran a job with the fair scheduler.
This will need some documentation (in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/) - are you planning on doing that as a separate JIRA? It would also be useful to call out the configuration differences compared to the MR1 fair scheduler. (E.g. I noticed that yarn.scheduler.fair.user-as-default-queue is new.)
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413760#comment-13413760 ]
Hudson commented on MAPREDUCE-3451:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1135 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1135/])
MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 1361020)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361020
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, MAPREDUCE-3451.v9.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira