You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org> on 2011/07/02 03:47:28 UTC
[jira] [Created] (HBASE-4057) Implement HBase version of "show
processlist"
Implement HBase version of "show processlist"
---------------------------------------------
Key: HBASE-4057
URL: https://issues.apache.org/jira/browse/HBASE-4057
Project: HBase
Issue Type: New Feature
Components: client, regionserver
Reporter: Nicolas Spiegelberg
Assignee: Riley Patterson
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100915#comment-13100915 ]
stack commented on HBASE-4057:
------------------------------
Nice doc.
I'll commit this tomorrow unless objection.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
> Attachments: HBASE-4057-doc.docx, HBASE-4057.patch
>
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062216#comment-13062216 ]
Nicolas Spiegelberg commented on HBASE-4057:
--------------------------------------------
@Andrew : we're trying to figure out the best way to expose this data. Perceived options:
1. Write a Java utility and expose via bin/hbase
2. Put into Ruby shell
3. HTTP-get
One of the complaints that our operations developers had was being forced to use Ruby for scripting if they go through the shell (they are mostly Python), so integrating that isn't a high pri for us. I was thinking the benefit of using HTTP-get over Java util is that we don't have to design a public RPC interface, but maybe RPC is the best solution.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062189#comment-13062189 ]
Andrew Purtell commented on HBASE-4057:
---------------------------------------
bq. I'm currently focusing on exposing the TaskMonitor to the Jetty HTTP server
-1
I don't think exposing the information via a HTTP interface is the right approach in its entirety. We have a Java client API for administrative function. The UIs / HTTP servers and the shell use those APIs. The shell should not have to use HBase RPC on the one hand and then make various and sundry HTTP transactions to pull information together. Furthermore, running the UI / HTTP servers should be optional.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100918#comment-13100918 ]
Todd Lipcon commented on HBASE-4057:
------------------------------------
Converting it to docbook, Stack?
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
> Attachments: HBASE-4057-doc.docx, HBASE-4057.patch
>
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058979#comment-13058979 ]
Andrew Purtell commented on HBASE-4057:
---------------------------------------
+1
Making more actions MonitoredTasks and extending TaskMonitor as needed (e.g. exposing it for interrogation via RPC for pretty printing by the shell) would be a natural extension of earlier work in this regard. Is this what you are thinking here Nicolas?
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4057:
-------------------------
Fix Version/s: 0.92.0
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
> Fix For: 0.92.0
>
> Attachments: HBASE-4057-doc.docx, HBASE-4057.patch
>
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100392#comment-13100392 ]
jiraposter@reviews.apache.org commented on HBASE-4057:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1750/
-----------------------------------------------------------
Review request for hbase.
Summary
-------
Includes extended TaskMonitor with consideration for high-frequency, short duration changes, a new MonitoredRPCHandler class, integration with the RPC, and filtered exposure both as and HTML table and as JSON via jamon.
This addresses bug HBASE-4057.
https://issues.apache.org/jira/browse/HBASE-4057
Diffs
-----
/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandler.java PRE-CREATION
/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java 1166510
/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java 1166510
/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 1166510
/src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon 1166510
/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1166510
/src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon 1166510
/src/main/jamon/org/apache/hbase/tmpl/common/TaskMonitorTmpl.jamon 1166510
/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java PRE-CREATION
/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredTask.java 1166510
/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredTaskImpl.java 1166510
/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java 1166510
/src/main/java/org/apache/hadoop/hbase/regionserver/RSStatusServlet.java 1166510
/src/main/resources/hbase-webapps/static/hbase.css 1166510
/src/test/java/org/apache/hadoop/hbase/monitoring/TestTaskMonitor.java 1166510
Diff: https://reviews.apache.org/r/1750/diff
Testing
-------
All unit tests passed. All exposure works as expected. Extensive load testing has been done on FB's internal branch.
Thanks,
Riley
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
> Attachments: HBASE-4057.patch
>
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100925#comment-13100925 ]
stack commented on HBASE-4057:
------------------------------
This is one of those features you didn't know you needed till you see it in action.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
> Attachments: HBASE-4057-doc.docx, HBASE-4057.patch
>
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063665#comment-13063665 ]
Andrew Purtell commented on HBASE-4057:
---------------------------------------
@Stack I don't object to having REST-ful whatever running wherever.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Riley Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062207#comment-13062207 ]
Riley Patterson commented on HBASE-4057:
----------------------------------------
I see your point. I'll rethink the means of aggregation of this data so that the shell and such can access it without HTTP, but ultimately having a JSP that exposes this stuff via HTTP could be quite useful for ease of building quick-and-dirty diagnostic tools.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100922#comment-13100922 ]
stack commented on HBASE-4057:
------------------------------
I will on commit.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
> Attachments: HBASE-4057-doc.docx, HBASE-4057.patch
>
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102163#comment-13102163 ]
Hudson commented on HBASE-4057:
-------------------------------
Integrated in HBase-TRUNK #2199 (See [https://builds.apache.org/job/HBase-TRUNK/2199/])
HBASE-4057 Implement HBase version of "show processlist"
stack :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/jamon/org/apache/hbase/tmpl/common/TaskMonitorTmpl.jamon
* /hbase/trunk/src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon
* /hbase/trunk/src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandler.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredTask.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredTaskImpl.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/RSStatusServlet.java
* /hbase/trunk/src/main/resources/hbase-webapps/static/hbase.css
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/monitoring/TestTaskMonitor.java
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
> Fix For: 0.92.0
>
> Attachments: HBASE-4057-doc.docx, HBASE-4057.patch
>
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-4057.
--------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Thank you for the nice new feature Riley. Commmitted to TRUNK
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
> Attachments: HBASE-4057-doc.docx, HBASE-4057.patch
>
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Riley Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Riley Patterson updated HBASE-4057:
-----------------------------------
Attachment: HBASE-4057.patch
Includes extended TaskMonitor with consideration for high-frequency, short duration changes, a new MonitoredRPCHandler class, integration with the RPC, and filtered exposure both as and HTML table and as JSON via jamon.
Will also put this on the review board, as it's a rather large patch.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
> Attachments: HBASE-4057.patch
>
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Riley Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100401#comment-13100401 ]
Riley Patterson commented on HBASE-4057:
----------------------------------------
Also @Andrew, currently, the only user-facing exposure is through jamon. However, the TaskMonitor's getTasks() method is a very straightforward internal exposure mechanism, and it would be relatively trivial to expose show processlist to the shell or any other user interface.
My internship ends on Friday (tomorrow), so I don't have time to do this while I'm at Facebook, but if you think shell exposure is important, I can implement it independently after my internship ends without too much trouble. However, I would really like to get this core functionality in before I leave Facebook.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
> Attachments: HBASE-4057.patch
>
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063659#comment-13063659 ]
Nicolas Spiegelberg commented on HBASE-4057:
--------------------------------------------
@Andrew: took a closer look at the TaskMonitor code after Riley brought this into our branch. Right now, we're logging control path operations. This JIRA is about adding the ability to observe a snapshot of our datapath state. Will the WeakReference route of MonitoredTasks scale to datapath perf?
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062230#comment-13062230 ]
Andrew Purtell commented on HBASE-4057:
---------------------------------------
@Nicolas and Riley: Having a HTTP polling option is not my objection, it is not providing support in the HBase admin API (via RPC somehow) for the shell.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Riley Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062160#comment-13062160 ]
Riley Patterson commented on HBASE-4057:
----------------------------------------
At this point, that's more or less the route I'm taking. I'm currently focusing on exposing the TaskMonitor to the Jetty HTTP server with as many useful metrics as I can expose for each task. Other tools that want to access the information (i.e. the shell or diagnostic tools) can then query the HTTP server for a JSON structure and parse and display it however they like.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063666#comment-13063666 ]
Andrew Purtell commented on HBASE-4057:
---------------------------------------
@Nicolas: Where should I look to dig in? Trunk? A GitHub repo? A reviewboard post?
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100493#comment-13100493 ]
jiraposter@reviews.apache.org commented on HBASE-4057:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1750/#review1817
-----------------------------------------------------------
Ship it!
Took a quick look at the patch -- looks good -- then I tried it. Its lovely. +1 on commit.
- Michael
On 2011-09-08 15:35:10, Riley Patterson wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1750/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-08 15:35:10)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Includes extended TaskMonitor with consideration for high-frequency, short duration changes, a new MonitoredRPCHandler class, integration with the RPC, and filtered exposure both as and HTML table and as JSON via jamon.
bq.
bq.
bq. This addresses bug HBASE-4057.
bq. https://issues.apache.org/jira/browse/HBASE-4057
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandler.java PRE-CREATION
bq. /src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java 1166510
bq. /src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java 1166510
bq. /src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 1166510
bq. /src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon 1166510
bq. /src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1166510
bq. /src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon 1166510
bq. /src/main/jamon/org/apache/hbase/tmpl/common/TaskMonitorTmpl.jamon 1166510
bq. /src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java PRE-CREATION
bq. /src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredTask.java 1166510
bq. /src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredTaskImpl.java 1166510
bq. /src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java 1166510
bq. /src/main/java/org/apache/hadoop/hbase/regionserver/RSStatusServlet.java 1166510
bq. /src/main/resources/hbase-webapps/static/hbase.css 1166510
bq. /src/test/java/org/apache/hadoop/hbase/monitoring/TestTaskMonitor.java 1166510
bq.
bq. Diff: https://reviews.apache.org/r/1750/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. All unit tests passed. All exposure works as expected. Extensive load testing has been done on FB's internal branch.
bq.
bq.
bq. Thanks,
bq.
bq. Riley
bq.
bq.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
> Attachments: HBASE-4057.patch
>
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102108#comment-13102108 ]
stack commented on HBASE-4057:
------------------------------
Added HBASE-4368 to bring this listing up in the shell.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
> Fix For: 0.92.0
>
> Attachments: HBASE-4057-doc.docx, HBASE-4057.patch
>
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102102#comment-13102102 ]
stack commented on HBASE-4057:
------------------------------
I'm going to commit this. We can open new jira to have this show in shell as well as in our RS UI.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
> Attachments: HBASE-4057-doc.docx, HBASE-4057.patch
>
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Riley Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Riley Patterson updated HBASE-4057:
-----------------------------------
Attachment: HBASE-4057-doc.docx
A basic, user-manual type of document describing the feature, its use, its configuration, and grokking its output.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
> Attachments: HBASE-4057-doc.docx, HBASE-4057.patch
>
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicolas Spiegelberg updated HBASE-4057:
---------------------------------------
Description: One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062231#comment-13062231 ]
Andrew Purtell commented on HBASE-4057:
---------------------------------------
bq. One of the complaints that our operations developers had was being forced to use Ruby for scripting if they go through the shell (they are mostly Python), so integrating that isn't a high pri for us.
But by analogy 'show processlist' is a MySQL CLI command, and the HBase CLI is the Ruby shell.
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4057) Implement HBase version of "show
processlist"
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062307#comment-13062307 ]
stack commented on HBASE-4057:
------------------------------
@Andrew Would it make sense to enable the REST server always? We have a container, jetty, running out on each regionserver already. We have the servlets mounted hard at /. How hard would it be to add REST webapp mounting it at /rest? I agree the lads should add an RPC that exposes the regionserver processlist for the likes of the shell to pull on but if the REST server was running always, they could run REST queries from whatever the language de jour to get their process listing (Doesn't the REST server allow you to specify JSON or protocol buffers or base64 already?).
> Implement HBase version of "show processlist"
> ---------------------------------------------
>
> Key: HBASE-4057
> URL: https://issues.apache.org/jira/browse/HBASE-4057
> Project: HBase
> Issue Type: New Feature
> Components: client, regionserver
> Reporter: Nicolas Spiegelberg
> Assignee: Riley Patterson
>
> One of the features that our DBAs use for MySQL analysis is "show processlist", which gives application-level stats about the RPC threads. Right now, we use jstack but that is very core-developer-centric. We need to create a similar tool that DBA/Ops/AppDevs can use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira