You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2011/06/21 22:41:47 UTC
[jira] [Created] (CASSANDRA-2804) expose dropped messages,
exceptions over JMX
expose dropped messages, exceptions over JMX
--------------------------------------------
Key: CASSANDRA-2804
URL: https://issues.apache.org/jira/browse/CASSANDRA-2804
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Priority: Minor
Fix For: 0.8.2
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2804) expose dropped messages,
exceptions over JMX
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2804:
--------------------------------------
Attachment: 2804-v2.txt
v2 adds recently dropped mbean as in Ryan's, and changes logDroppedMessages to log the total counts to avoid interfering with it.
> expose dropped messages, exceptions over JMX
> --------------------------------------------
>
> Key: CASSANDRA-2804
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2804
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.8.2
>
> Attachments: 2804-v2.txt, 2804.txt, twttr-cassandra-0.8-counts-resync-droppedmsg-metric.diff
>
>
> Patch against 0.7.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2804) expose dropped messages,
exceptions over JMX
Posted by "Wojciech Meler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053130#comment-13053130 ]
Wojciech Meler commented on CASSANDRA-2804:
-------------------------------------------
You read in my mind :) - I can't wait to see dropped messages graph in zabbix :)
It would also be nice to have request time stats as we don't know how long requests live in queues so it is hard to say how long clients wait to perform an operation.
> expose dropped messages, exceptions over JMX
> --------------------------------------------
>
> Key: CASSANDRA-2804
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2804
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.8.2
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2804) expose dropped messages,
exceptions over JMX
Posted by "Wojciech Meler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053193#comment-13053193 ]
Wojciech Meler commented on CASSANDRA-2804:
-------------------------------------------
I have TotalReadLatencyMicros graph in my zabbix - but unfortunately time request wait in queue is not included :(.
It would be ok if thread pool was unlimited (so queue len will always be 0), but now I have to check queue lenghts first to trust TotalReadLatencyMicros...
Is there other way to see stats how long client wait for data from given CF ? - because probably I misunderstood TotalReadLatencyMicros meaning :/
> expose dropped messages, exceptions over JMX
> --------------------------------------------
>
> Key: CASSANDRA-2804
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2804
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.8.2
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2804) expose dropped messages,
exceptions over JMX
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2804:
--------------------------------------
Description: Patch against 0.7.
Fix Version/s: 0.7.7
> expose dropped messages, exceptions over JMX
> --------------------------------------------
>
> Key: CASSANDRA-2804
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2804
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.7, 0.8.2
>
> Attachments: 2804.txt
>
>
> Patch against 0.7.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2804) expose dropped messages,
exceptions over JMX
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2804:
--------------------------------------
Attachment: 2804.txt
Added dropped message and exception counts to nodetool tpstats and info, respectively.
To avoid overwhelming users with counts that will always be zero, refactored some code into MessagingService.DROPPABLE_VERBS.
> expose dropped messages, exceptions over JMX
> --------------------------------------------
>
> Key: CASSANDRA-2804
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2804
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.8.2
>
> Attachments: 2804.txt
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2804) expose dropped messages,
exceptions over JMX
Posted by "Ryan King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ryan King updated CASSANDRA-2804:
---------------------------------
Attachment: twttr-cassandra-0.8-counts-resync-droppedmsg-metric.diff
Funny- we have a patch we've been working on for similar things.
Attached patch only does dropped messages, but it also includes a "recent" variant in JMX, which we need for our monitoring.
> expose dropped messages, exceptions over JMX
> --------------------------------------------
>
> Key: CASSANDRA-2804
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2804
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.7, 0.8.2
>
> Attachments: 2804.txt, twttr-cassandra-0.8-counts-resync-droppedmsg-metric.diff
>
>
> Patch against 0.7.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2804) expose dropped messages,
exceptions over JMX
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055470#comment-13055470 ]
Sylvain Lebresne commented on CASSANDRA-2804:
---------------------------------------------
Looks good. That being said, the "recent variant in JMX" is essentially the same as the one the StatusLogger is logging, and I could see a point in preferring to log this in JMX rather than in the log file. Not sure what is the best way to accommodate those two however.
In any case, the patch does improve on the current situation, so +1.
> expose dropped messages, exceptions over JMX
> --------------------------------------------
>
> Key: CASSANDRA-2804
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2804
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.7, 0.8.2
>
> Attachments: 2804.txt, twttr-cassandra-0.8-counts-resync-droppedmsg-metric.diff
>
>
> Patch against 0.7.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2804) expose dropped messages,
exceptions over JMX
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060535#comment-13060535 ]
Sylvain Lebresne commented on CASSANDRA-2804:
---------------------------------------------
In getRecentlyDroppedMessages, the last line of the for loop should be
{noformat}
lastDropped.put(verb, dropped);
{noformat}
Other than that, +1.
> expose dropped messages, exceptions over JMX
> --------------------------------------------
>
> Key: CASSANDRA-2804
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2804
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.8.2
>
> Attachments: 2804-v2.txt, 2804.txt, twttr-cassandra-0.8-counts-resync-droppedmsg-metric.diff
>
>
> Patch against 0.7.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2804) expose dropped messages,
exceptions over JMX
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060628#comment-13060628 ]
Hudson commented on CASSANDRA-2804:
-----------------------------------
Integrated in Cassandra-0.8 #207 (See [https://builds.apache.org/job/Cassandra-0.8/207/])
add MessagingService.get[Recently]DroppedMessages and StorageService.getExceptionCount
patch by jbellis and Ryan King; reviewed by slebresne for CASSANDRA-2804
jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143428
Files :
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/net/MessageDeliveryTask.java
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/tools/NodeProbe.java
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/net/MessagingServiceMBean.java
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageServiceMBean.java
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/tools/NodeCmd.java
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/GCInspector.java
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/AbstractCassandraDaemon.java
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/net/MessagingService.java
> expose dropped messages, exceptions over JMX
> --------------------------------------------
>
> Key: CASSANDRA-2804
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2804
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.8.2
>
> Attachments: 2804-v2.txt, 2804.txt, twttr-cassandra-0.8-counts-resync-droppedmsg-metric.diff
>
>
> Patch against 0.7.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2804) expose dropped messages,
exceptions over JMX
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053230#comment-13053230 ]
Sylvain Lebresne commented on CASSANDRA-2804:
---------------------------------------------
Actually I apologize, I gave you the wrong number. The TotalReadLatencyMicros that corresponds to the full request is the one in org.apache.cassandra.db:StorageProxy. That one does include the time waited in the different queues.
> expose dropped messages, exceptions over JMX
> --------------------------------------------
>
> Key: CASSANDRA-2804
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2804
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.8.2
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2804) expose dropped messages,
exceptions over JMX
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054144#comment-13054144 ]
Jonathan Ellis commented on CASSANDRA-2804:
-------------------------------------------
What to do about logDroppedMessages and its >0 check?
> expose dropped messages, exceptions over JMX
> --------------------------------------------
>
> Key: CASSANDRA-2804
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2804
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.7, 0.8.2
>
> Attachments: 2804.txt, twttr-cassandra-0.8-counts-resync-droppedmsg-metric.diff
>
>
> Patch against 0.7.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2804) expose dropped messages,
exceptions over JMX
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053220#comment-13053220 ]
Jonathan Ellis commented on CASSANDRA-2804:
-------------------------------------------
avg time in queue = (avg read latency * avg pending tasks in queue / reader threads)
> expose dropped messages, exceptions over JMX
> --------------------------------------------
>
> Key: CASSANDRA-2804
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2804
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.8.2
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2804) expose dropped messages,
exceptions over JMX
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053148#comment-13053148 ]
Sylvain Lebresne commented on CASSANDRA-2804:
---------------------------------------------
bq. It would also be nice to have request time stats as we don't know how long requests live in queues so it is hard to say how long clients wait to perform an operation.
That is the goal of the values (RecentReadLatencyMicros, TotalReadLatencyMicros, ...) exposed for each column family in the org.apache.cassandra.db:ColumnFamilies.<keyspaceName>.<columnFamilyName> mbean. It doesn't include any queuing, it is recorded directly in the thrift thread accepting the request (the only thing it doesn't take into account is the small validation we do of the request, but that's a handful of cpu cycles so it doesn't matter).
> expose dropped messages, exceptions over JMX
> --------------------------------------------
>
> Key: CASSANDRA-2804
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2804
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.8.2
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2804) expose dropped messages,
exceptions over JMX
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2804:
--------------------------------------
Fix Version/s: (was: 0.7.7)
Targeting 0.8+ now since we're changing logDroppedMessages behavior.
> expose dropped messages, exceptions over JMX
> --------------------------------------------
>
> Key: CASSANDRA-2804
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2804
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.8.2
>
> Attachments: 2804-v2.txt, 2804.txt, twttr-cassandra-0.8-counts-resync-droppedmsg-metric.diff
>
>
> Patch against 0.7.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira