You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/02/13 04:20:57 UTC
[jira] Created: (MAPREDUCE-2323) Add metrics to the fair scheduler
Add metrics to the fair scheduler
---------------------------------
Key: MAPREDUCE-2323
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: contrib/fair-share
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: 0.23.0
It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2323) Add metrics to the fair scheduler
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2323:
-----------------------------------
Status: Patch Available (was: Open)
> Add metrics to the fair scheduler
> ---------------------------------
>
> Key: MAPREDUCE-2323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: mr-2323-20.txt, mr-2323.txt
>
>
> It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2323) Add metrics to the fair scheduler
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2323:
-----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks for reviewing.
> Add metrics to the fair scheduler
> ---------------------------------
>
> Key: MAPREDUCE-2323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: mr-2323-20.txt, mr-2323.txt, mr-2323.txt, mr-2323.txt
>
>
> It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2323) Add metrics to the fair
scheduler
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060300#comment-13060300 ]
Hudson commented on MAPREDUCE-2323:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #734 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/734/])
MAPREDUCE-2323. Add metrics to the fair scheduler. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143252
Files :
* /hadoop/common/trunk/mapreduce/src/contrib/build-contrib.xml
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/PoolSchedulable.java
* /hadoop/common/trunk/mapreduce/src/docs/src/documentation/content/xdocs/fair_scheduler.xml
* /hadoop/common/trunk/mapreduce/CHANGES.txt
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/Pool.java
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/Schedulable.java
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairScheduler.java
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/JobSchedulable.java
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/test/org/apache/hadoop/mapred/TestFairScheduler.java
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/test/org/apache/hadoop/mapred/FakeSchedulable.java
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/PoolManager.java
> Add metrics to the fair scheduler
> ---------------------------------
>
> Key: MAPREDUCE-2323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: mr-2323-20.txt, mr-2323.txt, mr-2323.txt, mr-2323.txt
>
>
> It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2323) Add metrics to the fair scheduler
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2323:
-----------------------------------
Attachment: mr-2323.txt
Here's a patch against trunk with unit tests and docs. Above questions still stand.
> Add metrics to the fair scheduler
> ---------------------------------
>
> Key: MAPREDUCE-2323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: mr-2323-20.txt, mr-2323.txt
>
>
> It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2323) Add metrics to the fair
scheduler
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002435#comment-13002435 ]
Hadoop QA commented on MAPREDUCE-2323:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12472627/mr-2323.txt
against trunk revision 1076804.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/116//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/116//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/116//console
This message is automatically generated.
> Add metrics to the fair scheduler
> ---------------------------------
>
> Key: MAPREDUCE-2323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: mr-2323-20.txt, mr-2323.txt, mr-2323.txt
>
>
> It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2323) Add metrics to the fair
scheduler
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052094#comment-13052094 ]
Scott Chen commented on MAPREDUCE-2323:
---------------------------------------
+1 The patch looks good to me.
I think the failed tasks are unrelated.
> Add metrics to the fair scheduler
> ---------------------------------
>
> Key: MAPREDUCE-2323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: mr-2323-20.txt, mr-2323.txt, mr-2323.txt, mr-2323.txt
>
>
> It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2323) Add metrics to the fair
scheduler
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051449#comment-13051449 ]
Hadoop QA commented on MAPREDUCE-2323:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12483015/mr-2323.txt
against trunk revision 1137017.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.cli.TestMRCLI
org.apache.hadoop.fs.TestFileSystem
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/406//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/406//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/406//console
This message is automatically generated.
> Add metrics to the fair scheduler
> ---------------------------------
>
> Key: MAPREDUCE-2323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: mr-2323-20.txt, mr-2323.txt, mr-2323.txt, mr-2323.txt
>
>
> It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2323) Add metrics to the fair scheduler
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2323:
-----------------------------------
Attachment: mr-2323-20.txt
Here's a quick draft of this feature against our branch 20. Will work on updating to trunk and adding some kind of tests.
> Add metrics to the fair scheduler
> ---------------------------------
>
> Key: MAPREDUCE-2323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: mr-2323-20.txt
>
>
> It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2323) Add metrics to the fair
scheduler
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060611#comment-13060611 ]
Hudson commented on MAPREDUCE-2323:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #729 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/729/])
MAPREDUCE-2323. Add metrics to the fair scheduler. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143252
Files :
* /hadoop/common/trunk/mapreduce/src/contrib/build-contrib.xml
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/PoolSchedulable.java
* /hadoop/common/trunk/mapreduce/src/docs/src/documentation/content/xdocs/fair_scheduler.xml
* /hadoop/common/trunk/mapreduce/CHANGES.txt
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/Pool.java
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/Schedulable.java
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairScheduler.java
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/JobSchedulable.java
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/test/org/apache/hadoop/mapred/TestFairScheduler.java
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/test/org/apache/hadoop/mapred/FakeSchedulable.java
* /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/PoolManager.java
> Add metrics to the fair scheduler
> ---------------------------------
>
> Key: MAPREDUCE-2323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: mr-2323-20.txt, mr-2323.txt, mr-2323.txt, mr-2323.txt
>
>
> It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2323) Add metrics to the fair scheduler
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2323:
-----------------------------------
Attachment: mr-2323.txt
Rebased on trunk. Also fixes a bug where using "setPool" twice on a job would cause an NPE, since it would clean up the metrics on the first switch, and then NPE when cleaning it up again
> Add metrics to the fair scheduler
> ---------------------------------
>
> Key: MAPREDUCE-2323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: mr-2323-20.txt, mr-2323.txt, mr-2323.txt, mr-2323.txt
>
>
> It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2323) Add metrics to the fair
scheduler
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994016#comment-12994016 ]
Todd Lipcon commented on MAPREDUCE-2323:
----------------------------------------
Would appreciate feedback on the following:
- is the locking correct for updateMetrics? I take a lock on the scheduler and then the poolmanager, but no other locks.
- could it be finer grain? (do I need the coarse scheduler lock at all? unclear)
- should I add a configuration to disable the job-level metrics and only use pool-level metrics?
- when a user pool becomes empty, should we remove its metrics records?
> Add metrics to the fair scheduler
> ---------------------------------
>
> Key: MAPREDUCE-2323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: mr-2323-20.txt
>
>
> It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2323) Add metrics to the fair
scheduler
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999336#comment-12999336 ]
Hadoop QA commented on MAPREDUCE-2323:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12470959/mr-2323.txt
against trunk revision 1074251.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.mapred.TestDebugScript
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/30//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/30//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/30//console
This message is automatically generated.
> Add metrics to the fair scheduler
> ---------------------------------
>
> Key: MAPREDUCE-2323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: mr-2323-20.txt, mr-2323.txt
>
>
> It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2323) Add metrics to the fair scheduler
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-2323:
-----------------------------------
Attachment: mr-2323.txt
Slightly improved patch - the main code is the same, but this fixes some flaky behavior in the tests. I previously wasn't unregistering the metrics updater in fairsched's termination method, so different test cases were interacting poorly with eachother.
This should be ready for review.
> Add metrics to the fair scheduler
> ---------------------------------
>
> Key: MAPREDUCE-2323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: mr-2323-20.txt, mr-2323.txt, mr-2323.txt
>
>
> It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira