You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Sanjay Radia (JIRA)" <ji...@apache.org> on 2007/12/10 23:59:43 UTC
[jira] Created: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Additional Instrumentation for NameNode, RPC Layer and JMX support
-------------------------------------------------------------------
Key: HADOOP-2398
URL: https://issues.apache.org/jira/browse/HADOOP-2398
Project: Hadoop
Issue Type: New Feature
Components: dfs
Reporter: Sanjay Radia
Assignee: Sanjay Radia
Fix For: 0.16.0
Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2398:
---------------------------------
Status: Open (was: Patch Available)
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, metricsPatch6_3.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556187#action_12556187 ]
Hadoop QA commented on HADOOP-2398:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12372541/metricsPatch6_3.txt
against trunk revision .
@author +1. The patch does not contain any @author tags.
patch -1. The patch command could not apply the patch.
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1474/console
This message is automatically generated.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, metricsPatch6_3.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2398:
---------------------------------
Attachment: ScreenShotRPCStats.png
This screen shot shows Jconsole displaying the rpc statistics for the name node.
Note the tree on the left: it shows that name node stats and rpc stats for the name node are available.
Any of the values can graphed by clicking on the value. This screeenshot shows that
I decided to graph the RPC's Max processing and queued times.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2398:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Sanjay!
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, metricsPatch6_3.txt, metricsPatch6_4.patch, metricsPatch6_5.patch, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2398:
---------------------------------
Attachment: metricsPatch6_2.txt
Same patch against later trunk.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550652 ]
Raghu Angadi commented on HADOOP-2398:
--------------------------------------
Sanjay,
I haven't looked at the changes yet but the features are pretty neat and very useful. Could you comment on any overhead? Is it same as Simon stats.. I think Simon stats involved a synchronized section.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553503 ]
Sanjay Radia commented on HADOOP-2398:
--------------------------------------
My patch does not use any dynamic mbeans or report the metrics in a dynamic way by listing the declared metrics.
This would be very useful to do but I ran out of time for the .16 release.
I will file a new Jira to modify the metrics to use dynamic mbeans. This would make the task of adding new metrics much
simpler. One would not have to update an mbean for new metrics and also the metrics would be published automatically to the
metrics context.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555819#action_12555819 ]
Hadoop QA commented on HADOOP-2398:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12372475/metricsPatch6_2.txt
against trunk revision .
@author -1. The patch appears to contain 1 @author tags which the Hadoop community has agreed to not allow in code contributions.
javadoc -1. The javadoc tool appears to have generated messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs -1. The patch appears to cause Findbugs to fail.
core tests -1. The patch failed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1461/testReport/
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1461/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1461/console
This message is automatically generated.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2398:
---------------------------------
Attachment: ScreenShotNameNodeStats.png
This screen shot shows Jconsole displaying the name node statistics.
Note the tree on the left: it shows that name node stats and rpc stats for the name node are available.
Any of the values can graphed by clicking on the value. This screeenshot shows that
I decided to graph the Journal Sync Average Time.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, ScreenShotNameNodeStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2398:
---------------------------------
Attachment: metricsPatch6_5.patch
fixed findbugs warning
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, metricsPatch6_3.txt, metricsPatch6_4.patch, metricsPatch6_5.patch, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2398:
---------------------------------
Assignee: Senthil Subramanian (was: Sanjay Radia)
Status: Patch Available (was: Open)
Submitting: metricsPatch6_4.patch
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Senthil Subramanian
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, metricsPatch6_3.txt, metricsPatch6_4.patch, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553523 ]
Raghu Angadi commented on HADOOP-2398:
--------------------------------------
A few comments :
# Remove LOG.debug() after inovke() in RPC.java
# rpcDiscardedOps is not treated/named consistently compared to other two variables in RpcMetrics and RpcMgt. Also is is is not updated or reset.
# RpcMetrics.log should be called LOG.
# As you mention in the conversation, right now RpcMgtMBean interfaces and the access methods in RpcMgt are quired by JMX. Once we move "Dyncalic MBean" stuff most of it will go away. Same for NameNodeMgt and NameNodeMBean. registerMBean() could be in units.
# createFile() etc in NameNodeMetrics don't need to be synchronized.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558158#action_12558158 ]
Hadoop QA commented on HADOOP-2398:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12372639/metricsPatch6_4.patch
against trunk revision r611264.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs -1. The patch appears to introduce 1 new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1547/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1547/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1547/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1547/console
This message is automatically generated.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, metricsPatch6_3.txt, metricsPatch6_4.patch, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553501 ]
Sanjay Radia commented on HADOOP-2398:
--------------------------------------
The JMX part merely reads the same stats that are given to Simon.
JMX reads the stats only when a client calls the JMX MBeans' get method.
Hence no additional overhead.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2398:
---------------------------------
Attachment: metricsPatch6_4.patch
Patch against latest trunk
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, metricsPatch6_3.txt, metricsPatch6_4.patch, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2398:
---------------------------------
Status: Open (was: Patch Available)
Cancelling metricsPatch6_2.txt
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, metricsPatch6_3.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia reassigned HADOOP-2398:
------------------------------------
Assignee: Sanjay Radia (was: Senthil Subramanian)
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, metricsPatch6_3.txt, metricsPatch6_4.patch, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2398:
---------------------------------
Status: Patch Available (was: Open)
Submiting my latest patch: metricsPatch6_2.txt
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2398:
---------------------------------
Status: Patch Available (was: Open)
submitting: metricsPatch6_5.patch
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, metricsPatch6_3.txt, metricsPatch6_4.patch, metricsPatch6_5.patch, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2398:
---------------------------------
Attachment: metricsPatch6.txt
This patch add the following features
+ Additional instrumentation/metrics for
- Name node
- RPC Layer
+ Adds a utility layer for maintaining metrics (see hadoop.metrics.util.*)
- This makes it easier to add new metrics (fewer lines of code to add for each metrics
- Mains additional metrics for min and max for rate based metrics
- Consistency in naming the suffix and prefix for metrics (e.g. Foo_num_ops, Foo_ave_time etc)
- Provides an interface for monitoring systems like JMX to get read the metrics when one of JMX's
clients reads the metrics (without out having an update
thread that reads the metrics periodically)
+Publishes all the Name Node instrumentation/metrics via JMX, Java's standard monitoring
framework. This will allow one to connect JConsole to monitor the name node.
Note the cost is minimal - the mbeans are registered - there is not additonal cost unless someone
is actually viewing the metrics.
See NameNodeMgtMBean and RpcMgtMBean for the full list of instrumentation available for the name node as part of this patch
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-2398) Additional
Instrumentation for NameNode, RPC Layer and JMX support
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553523 ]
rangadi edited comment on HADOOP-2398 at 12/19/07 2:38 PM:
----------------------------------------------------------------
A few comments :
# Remove LOG.debug() after inovke() in RPC.java
# rpcDiscardedOps is not treated/named consistently compared to other two variables in RpcMetrics and RpcMgt. Also it is not updated or reset.
# RpcMetrics.log should be called LOG.
# As you mentione in the conversation, right now RpcMgtMBean interfaces and the access methods in RpcMgt are quired by JMX. Once we move to "Dyncalic MBean" stuff most of it will go away. Same for NameNodeMgt and NameNodeMBean. registerMBean() could be in utils.
# createFile() etc in NameNodeMetrics don't need to be synchronized.
Edit: Typos.
was (Author: rangadi):
A few comments :
# Remove LOG.debug() after inovke() in RPC.java
# rpcDiscardedOps is not treated/named consistently compared to other two variables in RpcMetrics and RpcMgt. Also is is is not updated or reset.
# RpcMetrics.log should be called LOG.
# As you mentione in the conversation, right now RpcMgtMBean interfaces and the access methods in RpcMgt are quired by JMX. Once we move to "Dyncalic MBean" stuff most of it will go away. Same for NameNodeMgt and NameNodeMBean. registerMBean() could be in utils.
# createFile() etc in NameNodeMetrics don't need to be synchronized.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2398:
---------------------------------
Status: Patch Available (was: Open)
Submitting metricsPatch6_3.txt
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, metricsPatch6_3.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555413#action_12555413 ]
Raghu Angadi commented on HADOOP-2398:
--------------------------------------
+1.
You might want to move all the 'HADOOP_OPTS="-Dcom.sun.management.jmxremote $HADOOP_OPTS' into one if clause in bin/hadoop, which is functionally same. But this is more of a personal preference.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559066#action_12559066 ]
Hudson commented on HADOOP-2398:
--------------------------------
Integrated in Hadoop-Nightly #366 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/366/])
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, metricsPatch6_3.txt, metricsPatch6_4.patch, metricsPatch6_5.patch, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558653#action_12558653 ]
Hadoop QA commented on HADOOP-2398:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373077/metricsPatch6_5.patch
against trunk revision r611760.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1582/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1582/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1582/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1582/console
This message is automatically generated.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, metricsPatch6_3.txt, metricsPatch6_4.patch, metricsPatch6_5.patch, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2398:
---------------------------------
Attachment: metricsPatch6_3.txt
Fixed the javadoc findbugs and the contrib failures.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, metricsPatch6_3.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2398:
---------------------------------
Attachment: metricsPatch6_1.txt
Update patch to reflect Ragu's feedback.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2398) Additional Instrumentation for
NameNode, RPC Layer and JMX support
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Radia updated HADOOP-2398:
---------------------------------
Status: Open (was: Patch Available)
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, metricsPatch6_1.txt, metricsPatch6_2.txt, metricsPatch6_3.txt, metricsPatch6_4.patch, metricsPatch6_5.patch, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-2398) Additional
Instrumentation for NameNode, RPC Layer and JMX support
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553523 ]
rangadi edited comment on HADOOP-2398 at 12/19/07 2:37 PM:
----------------------------------------------------------------
A few comments :
# Remove LOG.debug() after inovke() in RPC.java
# rpcDiscardedOps is not treated/named consistently compared to other two variables in RpcMetrics and RpcMgt. Also is is is not updated or reset.
# RpcMetrics.log should be called LOG.
# As you mentione in the conversation, right now RpcMgtMBean interfaces and the access methods in RpcMgt are quired by JMX. Once we move to "Dyncalic MBean" stuff most of it will go away. Same for NameNodeMgt and NameNodeMBean. registerMBean() could be in utils.
# createFile() etc in NameNodeMetrics don't need to be synchronized.
was (Author: rangadi):
A few comments :
# Remove LOG.debug() after inovke() in RPC.java
# rpcDiscardedOps is not treated/named consistently compared to other two variables in RpcMetrics and RpcMgt. Also is is is not updated or reset.
# RpcMetrics.log should be called LOG.
# As you mention in the conversation, right now RpcMgtMBean interfaces and the access methods in RpcMgt are quired by JMX. Once we move "Dyncalic MBean" stuff most of it will go away. Same for NameNodeMgt and NameNodeMBean. registerMBean() could be in units.
# createFile() etc in NameNodeMetrics don't need to be synchronized.
> Additional Instrumentation for NameNode, RPC Layer and JMX support
> -------------------------------------------------------------------
>
> Key: HADOOP-2398
> URL: https://issues.apache.org/jira/browse/HADOOP-2398
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Fix For: 0.16.0
>
> Attachments: metricsPatch6.txt, ScreenShotNameNodeStats.png, ScreenShotRPCStats.png
>
>
> Additional Instrumentation is needed for name node and its rpc layer. Furthermore the instrumentation should be
> visible via JMX, Java's standard monitoring tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.