You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Eric Payne (Created) (JIRA)" <ji...@apache.org> on 2012/01/20 17:47:42 UTC
[jira] [Created] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
ResourceManager should provide node lists in JMX output
-------------------------------------------------------
Key: MAPREDUCE-3703
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: resourcemanager
Affects Versions: 0.23.1
Reporter: Eric Payne
Assignee: Eric Payne
Priority: Critical
In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195308#comment-13195308 ]
Mahadev konar commented on MAPREDUCE-3703:
------------------------------------------
@Eric,
You are right. Reviewing the code now.
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-3703:
-------------------------------------
Component/s: mrv2
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196327#comment-13196327 ]
Hudson commented on MAPREDUCE-3703:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #447 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/447/])
MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. (Eric Payne via mahadev) - Merge r1237920 from trunk.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237921
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestRMNMInfo.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfo.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfoBeans.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Eric Payne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192291#comment-13192291 ]
Eric Payne commented on MAPREDUCE-3703:
---------------------------------------
@Hitesh,
The timing and size did not seem unreasonable during my tests in a 0.20.205 security cluster with a large number of nodes (~4000 nodes).
Size: ~800000 Megabytes
Avg time after 10 accesses to the JMX service (using wget to access the jmx service):
real: 0m0.1016s
sys: 0m0.0050s
user: 0m0.0014s
Since the load seems reasonable, and since this service is provided in previous releases, I would like to provide it in 0.23 also.
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Eric Payne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Payne updated MAPREDUCE-3703:
----------------------------------
Release Note:
New JMX Bean in ResourceManager to provide list of live node managers:
Hadoop:service=ResourceManager,name=RMNMInfo LiveNodeManagers
Status: Patch Available (was: Open)
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194418#comment-13194418 ]
Mahadev konar commented on MAPREDUCE-3703:
------------------------------------------
Actually now that I think of it, we never had anything exposed via jmx for the RM? Looks like this is the first bean we are adding. Is that right? I thought some of the work that Thomas did on web services was also exposed via jmx. No?
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196927#comment-13196927 ]
Hudson commented on MAPREDUCE-3703:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #177 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/177/])
MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. (Eric Payne via mahadev) - Merge r1237920 from trunk.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237921
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestRMNMInfo.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfo.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfoBeans.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196869#comment-13196869 ]
Hudson commented on MAPREDUCE-3703:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #942 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/942/])
MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. (Eric Payne via mahadev)
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237920
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestRMNMInfo.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfo.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfoBeans.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196881#comment-13196881 ]
Hudson commented on MAPREDUCE-3703:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #155 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/155/])
MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. (Eric Payne via mahadev) - Merge r1237920 from trunk.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237921
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestRMNMInfo.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfo.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfoBeans.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196934#comment-13196934 ]
Hudson commented on MAPREDUCE-3703:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #975 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/975/])
MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. (Eric Payne via mahadev)
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237920
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestRMNMInfo.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfo.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfoBeans.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Eric Payne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192305#comment-13192305 ]
Eric Payne commented on MAPREDUCE-3703:
---------------------------------------
Here is the (pretty-fied) JSON output of the RMNMInfo bean from the Resourcemanager:
{noformat}
{
"name" : "Hadoop:service=ResourceManager,name=RMNMInfo",
"modelerType" : "org.apache.hadoop.yarn.server.resourcemanager.RMNMInfo",
"LiveNodeManagers" : "
[
{ "HostName":"<HOSTNAME-1>"
"NodeHTTPAddress":"<HOSTNAME-1>:<PORT>"
"Rack":"/xxx.xxx.xxx.xxx"
"NodeState":"RUNNING"
"HealthStatus":"Healthy"
"LastHealthyUpdate":1327351645595
"HealthReport":""
"UsedContainers":0
"UsedMemory":0
"AvailableMemory":<MEM-IN-MEG>
}
{ "HostName":"<HOSTNAME-2>"
"NodeHTTPAddress":"<HOSTNAME-2>:<PORT>"
"Rack":"/xxx.xxx.xxx.xxx"
"NodeState":"RUNNING"
"HealthStatus":"Healthy"
"LastHealthyUpdate":1327351645755
"HealthReport":""
"UsedContainers":0
"UsedMemory":0
"AvailableMemory":<MEM-IN-MEG>
}
...
{ "HostName":"<HOSTNAME-n>"
"NodeHTTPAddress":"<HOSTNAME-n>:<PORT>"
"Rack":"/xxx.xxx.xxx.xxx"
"NodeState":"RUNNING"
"HealthStatus":"Healthy"
"LastHealthyUpdate":1327351645423
"HealthReport":""
"UsedContainers":0
"UsedMemory":0
"AvailableMemory":<MEM-IN-MEG>
}
]"
}
{noformat}
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196310#comment-13196310 ]
Hudson commented on MAPREDUCE-3703:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1690 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1690/])
MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. (Eric Payne via mahadev)
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237920
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestRMNMInfo.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfo.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfoBeans.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196382#comment-13196382 ]
Hudson commented on MAPREDUCE-3703:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #462 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/462/])
MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. (Eric Payne via mahadev) - Merge r1237920 from trunk.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237921
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestRMNMInfo.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfo.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfoBeans.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196313#comment-13196313 ]
Hudson commented on MAPREDUCE-3703:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1618 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1618/])
MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. (Eric Payne via mahadev)
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237920
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestRMNMInfo.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfo.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfoBeans.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196336#comment-13196336 ]
Hudson commented on MAPREDUCE-3703:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1635 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1635/])
MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. (Eric Payne via mahadev)
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237920
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestRMNMInfo.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfo.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfoBeans.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-3703:
-------------------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Just committed this. Thanks a lot Eric!
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Eric Payne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194853#comment-13194853 ]
Eric Payne commented on MAPREDUCE-3703:
---------------------------------------
Actually, there are several ResourceManager beans. I couldn't find any that test JMX specifically, though, so I added a test for this new one. The ResourceManager beans are accessible via http://<rm-host>:<port>/jmx and are as follows:
Hadoop:service=ResourceManager,name=UgiMetrics (login metrics)
Hadoop:service=ResourceManager,name=MetricsSystem,sub=Stats (sources/sinks/ops/time)
Hadoop:service=ResourceManager,name=QueueMetrics,q0=root,q1=<qname>
Hadoop:service=ResourceManager,name=RpcDetailedActivityForPort<port> (NumOps/AvgTime)
Hadoop:service=ResourceManager,name=ClusterMetrics (NumNodes, but not lists of nodes)
Hadoop:service=ResourceManager,name=RMNMInfo (NEW: contains list and status of live nodes).
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196324#comment-13196324 ]
Hudson commented on MAPREDUCE-3703:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #438 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/438/])
MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. (Eric Payne via mahadev) - Merge r1237920 from trunk.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237921
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestRMNMInfo.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfo.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfoBeans.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-3703:
-------------------------------------
Fix Version/s: 0.23.1
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Eric Payne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196436#comment-13196436 ]
Eric Payne commented on MAPREDUCE-3703:
---------------------------------------
Thank you Mahadev!
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Hitesh Shah (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192409#comment-13192409 ]
Hitesh Shah commented on MAPREDUCE-3703:
----------------------------------------
@Eric, 1 MB worth of data being pulled may be an issue depending on where the call is being made from. If most of the use cases are from more or less within the same network/colo, then latency may not be that big a factor ( unless someone hits the jmx url too often ). In any case, if no one has mentioned anything with respect to the jmx data size for the 0.20 branch, I guess there is no harm in adding it also to .23.
Minor comments on the json keys:
- could you change UsedMemory and AvailableMemory to UsedMemoryMB and AvailableMemoryMB.
- Also, it would be good if the json output could be changed to have the keys match the elements used in the webservice apis.
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Eric Payne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192614#comment-13192614 ]
Eric Payne commented on MAPREDUCE-3703:
---------------------------------------
The following are updates to the LiveNameNodes fields:
{noformat}
{
"name" : "Hadoop:service=ResourceManager,name=RMNMInfo",
"modelerType" : "org.apache.hadoop.yarn.server.resourcemanager.RMNMInfo",
"LiveNodeManagers" : "[
{
"HostName":"<HOSTNAME1>",
"Rack":"/xxx.xxx.xxx.xxx",
"State":"RUNNING",
"NodeId":"<HOSTNAME1>:<PORT1>",
"NodeHTTPAddress":"<HOSTNAME1>:<PORT2>",
"HealthStatus":"Healthy",
"LastHealthUpdate":1327442780185,
"HealthReport":"",
"NumContainersMB":0,
"UsedMemoryMB":0,
"AvailableMemoryMB":<MEM>
},
{ ... },
{ ... },
...
]"
}
{noformat}
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194376#comment-13194376 ]
Mahadev konar commented on MAPREDUCE-3703:
------------------------------------------
+1 lgtm!
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Hitesh Shah (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190250#comment-13190250 ]
Hitesh Shah commented on MAPREDUCE-3703:
----------------------------------------
Would it be better to just provide a count of nodes in the jmx output and have additional webservice apis ( with appropriate filter/pagination params to reduce the size of returned data ) for querying the RM for all the active/unhealthy/rebooted/... nodes?
I am aware that the jobtracker does emit this information today but on a large cluster, the amount of data being sent back in the json output could be potentially huge ( and a performance issue if any rogue layer pings the jmx servlet too many times)?
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193264#comment-13193264 ]
Hadoop QA commented on MAPREDUCE-3703:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511861/MAPREDUCE-3703.1.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1673//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1673//console
This message is automatically generated.
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3703) ResourceManager should provide
node lists in JMX output
Posted by "Eric Payne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Payne updated MAPREDUCE-3703:
----------------------------------
Attachment: MAPREDUCE-3703.1.txt
> ResourceManager should provide node lists in JMX output
> -------------------------------------------------------
>
> Key: MAPREDUCE-3703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Eric Payne
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3703.1.txt
>
>
> In 0.20.*, the JMX UI for the JobTracker (http://<JobTrackerHost>:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section.
> In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes.
> At least the list of active nodes is needed in JSON format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira