You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ramya Sunil (JIRA)" <ji...@apache.org> on 2011/07/27 21:36:09 UTC
[jira] [Created] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Missing cluster level stats on the RM UI
----------------------------------------
Key: MAPREDUCE-2738
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Fix For: 0.23.0
Cluster usage information such as the following are currently not available in the RM UI.
- Total number of apps submitted so far
- Total number of containers running/total memory usage
- Total capacity of the cluster (in terms of memory)
- Reserved memory
- Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
- Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
- Excluded NMs
- Handle to the jobhistory server
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2738:
-------------------------------------------
Fix Version/s: 0.24.0
Target Version/s: 0.23.0, 0.24.0
Affects Version/s: 0.24.0
Status: Patch Available (was: Open)
This patch adds in two new, small tables at the top of several pages on the RM. The first table is always there and shows the stats for the cluster Total Apps Submitted, Total Containers Running, Total Memory in the cluster, Memory used, Memory reserved, Total NMs, Blacklisted NM, and Excluded NMs.
The NMs are links to lists of NMs in that state. The nodes page was updated to support filtering by a given state.
Also if user level metrics are enabled in the scheduler, and web Authentication is enabled then a second table showing stats for the current user is shown. It includes (just for this user) Apps submitted, containers running, containers pending, containers reserved, memory used, memory pending, memory reserved.
I did not add in a link to the jobhistory server, because that is map/reduce specific. If this is a blocker then I will add in a new config option for yarn which will be a list of links to various servers that display UIs related to this and add them in of the left hand nav bar.
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120208#comment-13120208 ]
Robert Joseph Evans commented on MAPREDUCE-2738:
------------------------------------------------
I can see about adding in separate columns for each, that would clean things up a bit.
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108804#comment-13108804 ]
Robert Joseph Evans commented on MAPREDUCE-2738:
------------------------------------------------
I have been looking at this and I don't like using JMX. Partly because the fifo scheduler does not update JMX correctly, which I can fix, but also because the queue name must be included in the JMX name to get the info back correctly. Because of this there is no guarantee that a scheduler will have a root queue, or something like that, so it because very difficult to write something generic with JMX that can cover all cases.
Instead I would like to add in some APIs to YarnScheduler to get back the information we want. It may get the information from metrics if it feels like it. This is also very similar to what MAPREDUCE-2789 appears to need, but on a per application basis instead of on an aggregate basis for the entire cluster.
Also we could add in some APIs on a per node basis so the node manager GUI can get back to where it was before and display resources used, reserved, etc. I will talk with the owner of MAPREDUCE-2789 about potentially filing a separate JIRA to do this work on.
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2738:
-------------------------------------------
Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0)
Status: Open (was: Patch Available)
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120922#comment-13120922 ]
Hadoop QA commented on MAPREDUCE-2738:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12497783/MAPREDUCE-2738-20111005.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/948//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/948//console
This message is automatically generated.
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2738-20111005.txt, MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121899#comment-13121899 ]
Hudson commented on MAPREDUCE-2738:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #31 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/31/])
MAPREDUCE-2738. svn merge -c r1179229 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179232
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AboutBlock.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AboutPage.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlockWithMetrics.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/InfoPage.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodesPage.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2738-20111005.txt, MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107950#comment-13107950 ]
Robert Joseph Evans commented on MAPREDUCE-2738:
------------------------------------------------
I am working on a patch for this, and I am trying to use the JMX APIs to pull out the metrics I need for this. I just want to check that JMX seems like the right way to go on this. Also Some of the metrics, at least in single user mode without security seem to be broken. Reserved memory GB always is going negative I filed MAPREDUCE-3036 to look into this and I am going to put this one on hold until I can get a fix for that one in.
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans reassigned MAPREDUCE-2738:
----------------------------------------------
Assignee: Robert Joseph Evans (was: John George)
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2738:
-------------------------------------
Priority: Blocker (was: Major)
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104809#comment-13104809 ]
Robert Joseph Evans commented on MAPREDUCE-2738:
------------------------------------------------
Stealing this from John :).
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "John George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George reassigned MAPREDUCE-2738:
--------------------------------------
Assignee: John George
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: John George
> Fix For: 0.23.0
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120186#comment-13120186 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2738:
----------------------------------------------------
I checked MRV1 JT. We have Blacklisted Nodes, Graylisted Nodes(which is never used as global-blacklisting wasn't used) and Excluded Nodes. And there is no sign of Lost nodes, which is a limitation in JT UI and can be fixed in YARN. We can add separate columns for DECOMMISSIONED, LOST and UNHEALTHY nodes if the number of columns isn't a problem.
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120996#comment-13120996 ]
Robert Joseph Evans commented on MAPREDUCE-2738:
------------------------------------------------
Thanks for fixing the tests. I forgot to run them after making my change, and since I added in new columns to the data the number of tds changed. My bad. Your changes to the patch look good. +1
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2738-20111005.txt, MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2738:
-----------------------------------------------
Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0)
Status: Open (was: Patch Available)
The patch looks good now. +1 overall.
Cancelling patch for fixing test failures.
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120484#comment-13120484 ]
Hadoop QA commented on MAPREDUCE-2738:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12497696/MR-2738.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestNodesPage
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/943//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/943//console
This message is automatically generated.
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120995#comment-13120995 ]
Hudson commented on MAPREDUCE-2738:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1099 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1099/])
MAPREDUCE-2738. Added the missing cluster level statisticss on the RM web UI. Contributed by Robert Joseph Evans.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179229
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AboutBlock.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AboutPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlockWithMetrics.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/InfoPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodesPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2738-20111005.txt, MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120157#comment-13120157 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2738:
----------------------------------------------------
Gone through the patch. Looks mostly good!
Some suggestions for mostly cosmetic changes:
- I'd prefer 'good' and 'removed' nodes as the main categories. If that is okay, we can use the same in the URL path-names instead of bl, ex. Better and full-forms are easy to remember and pass around. Same in the Titles and corresponding variable names for the corresponding pages.
- Can we also print NodeState on the NodesPage? Definitely helps for the all-nodes case, also distinguishes the type of a removed node - LOST vs DECOMMISIONED.
Also ran it on a single node. Looks clean except for two minor things - (1) the titles of the new table are bolder than the other main-table - say apps-list and (2) it may be good to have some gap between the metrics-table and the main-table for each of this pages. Please see if these are doable, if it is too convoluted we can skip for now.
Thanks for taking this up! I am the biggest beneficiary in the short term, when I run benchmarks and lots of jobs :)
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121928#comment-13121928 ]
Hudson commented on MAPREDUCE-2738:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #822 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/822/])
MAPREDUCE-2738. Added the missing cluster level statisticss on the RM web UI. Contributed by Robert Joseph Evans.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179229
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AboutBlock.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AboutPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlockWithMetrics.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/InfoPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodesPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2738-20111005.txt, MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120180#comment-13120180 ]
Robert Joseph Evans commented on MAPREDUCE-2738:
------------------------------------------------
I can change it to good and removed, but that is not what was listed previously in the web UI. Our ops team wants to know how many nodes were blacklisted vs. excluded at a glance, and not have to do that calculation themselves so I would prefer to keep blacklisted vs. excluded.
Everything else should be doable. I will have an updated patch soon.
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2738:
-----------------------------------------------
Attachment: MAPREDUCE-2738-20111005.txt
Minor fixes for the tests.
I verified this on a single node setup. CLuster level stats are coming out good.
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2738-20111005.txt, MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2738:
-------------------------------------------
Attachment: MR-2738.txt
New patch addresses all of the comments.
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119355#comment-13119355 ]
Hadoop QA commented on MAPREDUCE-2738:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12497469/MR-2738.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/927//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/927//console
This message is automatically generated.
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120998#comment-13120998 ]
Hudson commented on MAPREDUCE-2738:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1021 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1021/])
MAPREDUCE-2738. Added the missing cluster level statisticss on the RM web UI. Contributed by Robert Joseph Evans.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179229
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AboutBlock.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AboutPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlockWithMetrics.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/InfoPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodesPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2738-20111005.txt, MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2738:
-----------------------------------------------
Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0)
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2738-20111005.txt, MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119384#comment-13119384 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2738:
----------------------------------------------------
Looking at the patch now.
bq. I did not add in a link to the jobhistory server, because that is map/reduce specific.
Perfect, I was about comment the same, let's not pollute RM with MR specific things.
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121916#comment-13121916 ]
Hudson commented on MAPREDUCE-2738:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #38 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/38/])
MAPREDUCE-2738. svn merge -c r1179229 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179232
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AboutBlock.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AboutPage.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlockWithMetrics.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/InfoPage.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodesPage.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2738-20111005.txt, MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2738:
-------------------------------------------
Attachment: MR-2738.txt
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2738:
-------------------------------------------
Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0)
Status: Patch Available (was: Open)
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2738:
-----------------------------------------------
Resolution: Fixed
Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0)
Status: Resolved (was: Patch Available)
I just committed this to trunk and branch-0.23. Thanks Robert!
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2738-20111005.txt, MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121004#comment-13121004 ]
Hudson commented on MAPREDUCE-2738:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1037 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1037/])
MAPREDUCE-2738. Added the missing cluster level statisticss on the RM web UI. Contributed by Robert Joseph Evans.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179229
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AboutBlock.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AboutPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlockWithMetrics.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/InfoPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodesPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2738-20111005.txt, MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2738) Missing cluster level stats on
the RM UI
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121940#comment-13121940 ]
Hudson commented on MAPREDUCE-2738:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #852 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/852/])
MAPREDUCE-2738. Added the missing cluster level statisticss on the RM web UI. Contributed by Robert Joseph Evans.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179229
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AboutBlock.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AboutPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlockWithMetrics.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/InfoPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodesPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java
> Missing cluster level stats on the RM UI
> ----------------------------------------
>
> Key: MAPREDUCE-2738
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2738
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2738-20111005.txt, MR-2738.txt, MR-2738.txt
>
>
> Cluster usage information such as the following are currently not available in the RM UI.
> - Total number of apps submitted so far
> - Total number of containers running/total memory usage
> - Total capacity of the cluster (in terms of memory)
> - Reserved memory
> - Total number of NMs - sorting based on Node IDs is an option but when there are lost NMs or restarted NMs, the node ids does not correspond to the actual value
> - Blacklisted NMs - sorting based on health-status and counting manually is not very straight forward
> - Excluded NMs
> - Handle to the jobhistory server
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira