You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Donald Smith <Do...@audiencescience.com> on 2014/08/27 21:38:15 UTC
How often are JMX Cassandra metrics reset?
I'm using JMX to retrieve Cassandra metrics. I notice that Max and Count are cumulative and aren't reset. How often are the stats for Mean, 99tthPercentile, etc reset back to zero?
For example, 99thPercentile shows as 1.5 mls. Over how many minutes?
ClientRequest/Read/Latency:
LatencyUnit = MICROSECONDS
FiveMinuteRate = 1.12
FifteenMinuteRate = 1.11
RateUnit = SECONDS
MeanRate = 1.65
OneMinuteRate = 1.13
EventType = calls
Max = 237,373.37
Count = 961,312
50thPercentile = 383.2
Mean = 908.46
Min = 95.64
StdDev = 3,034.62
75thPercentile = 626.34
95thPercentile = 954.31
98thPercentile = 1,443.11
99thPercentile = 1,472.4
999thPercentile = 1,858.1
Donald A. Smith | Senior Software Engineer
P: 425.201.3900 x 3866
C: (206) 819-5965
F: (646) 443-2333
donalds@AudienceScience.com<ma...@AudienceScience.com>
[AudienceScience]
RE: How often are JMX Cassandra metrics reset?
Posted by Donald Smith <Do...@audiencescience.com>.
Thanks, Chris.
75thPercentile is clearly NOT lifetime: its value jumps around.
However, I can tell that Max is lifetime; it's been showing the exact same value for days, on various nodes. Hence my doubts.
From: Chris Lohfink [mailto:clohfink@blackbirdit.com]
Sent: Thursday, August 28, 2014 3:56 PM
To: user@cassandra.apache.org
Subject: Re: How often are JMX Cassandra metrics reset?
In the version of metrics used theres a uniform reservoir and a exponentially weighted one. This is used to compute the min, max, mean, std dev and quantiles. For the timers it uses by default it uses the exp. decaying one which is weighted for the last 5 minutes.
http://grepcode.com/file/repo1.maven.org/maven2/com.yammer.metrics/metrics-core/2.2.0/com/yammer/metrics/core/Timer.java?av=f
Chris Lohfink
On Aug 28, 2014, at 5:39 PM, Donald Smith <Do...@audiencescience.com>> wrote:
The metrics OneMinuteRate, FIveMinuteRate, FifteenMinuteRate, and MeanRate are NOT lifetime values but they're all counts of requests, not latency. The latency values (Max, Count, 50thPercentile, Mean, etc) ARE lifetime values, I think, and thus would seem to be kinda useless for me, since our servers have been running for months.
Maybe there's a way to reset lifetime metrics to zero. I connected to a cassandra server remotely via jConsole (port 7199) and I can read various metrics via the MBeans, but I don't see an operation for resetting to zero. But perhaps that's because I'm connecting remotely.
ClientRequest/Read/Latency:
LatencyUnit = MICROSECONDS
FiveMinuteRate = 1.12
FifteenMinuteRate = 1.11
RateUnit = SECONDS
MeanRate = 1.65
OneMinuteRate = 1.13
EventType = calls
Max = 237,373.37
Count = 961,312
50thPercentile = 383.2
Mean = 908.46
Min = 95.64
StdDev = 3,034.62
75thPercentile = 626.34
95thPercentile = 954.31
98thPercentile = 1,443.11
99thPercentile = 1,472.4
999thPercentile = 1,858.1
From: Nick Bailey [mailto:nick@datastax.com]
Sent: Thursday, August 28, 2014 1:50 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: How often are JMX Cassandra metrics reset?
Those percentile values should be for the lifetime of the node yes. Depending on what version of OpsCenter you are using it is either using the 'recent' metrics described by Rob, or it is using the FiveMinuteRate from JMX as well as doing some of it's own aggregation depending on the rollup size.
On Thu, Aug 28, 2014 at 12:36 PM, Robert Coli <rc...@eventbrite.com>> wrote:
On Thu, Aug 28, 2014 at 9:27 AM, Donald Smith <Do...@audiencescience.com>> wrote:
And yet OpsCenter shows graphs with ever-changing metrics that show recent performance. Does OpsCenter not get its stats from JMX?
1) Certain JMX endpoints expose "recent" metrics, or at least used to. These are "recent" as in "since the last time someone polled this endpoint."
2) OpsCenter samples via JMX and then stores metrics in its own columnfamily. I would not be shocked if it does some minor aggregation as it does so.
This all said, OpsCenter is not Apache Cassandra software, so the Apache Cassandra user mailing list may not be the ideal place for it to be discussed or supported...
=Rob
Re: How often are JMX Cassandra metrics reset?
Posted by Chris Lohfink <cl...@blackbirdit.com>.
In the version of metrics used theres a uniform reservoir and a exponentially weighted one. This is used to compute the min, max, mean, std dev and quantiles. For the timers it uses by default it uses the exp. decaying one which is weighted for the last 5 minutes.
http://grepcode.com/file/repo1.maven.org/maven2/com.yammer.metrics/metrics-core/2.2.0/com/yammer/metrics/core/Timer.java?av=f
Chris Lohfink
On Aug 28, 2014, at 5:39 PM, Donald Smith <Do...@audiencescience.com> wrote:
> The metrics OneMinuteRate, FIveMinuteRate, FifteenMinuteRate, and MeanRate are NOT lifetime values but they’re all counts of requests, not latency. The latency values (Max, Count, 50thPercentile, Mean, etc) ARE lifetime values, I think, and thus would seem to be kinda useless for me, since our servers have been running for months.
>
> Maybe there’s a way to reset lifetime metrics to zero. I connected to a cassandra server remotely via jConsole (port 7199) and I can read various metrics via the MBeans, but I don’t see an operation for resetting to zero. But perhaps that’s because I’m connecting remotely.
>
> ClientRequest/Read/Latency:
> LatencyUnit = MICROSECONDS
> FiveMinuteRate = 1.12
> FifteenMinuteRate = 1.11
> RateUnit = SECONDS
> MeanRate = 1.65
> OneMinuteRate = 1.13
> EventType = calls
> Max = 237,373.37
> Count = 961,312
> 50thPercentile = 383.2
> Mean = 908.46
> Min = 95.64
> StdDev = 3,034.62
> 75thPercentile = 626.34
> 95thPercentile = 954.31
> 98thPercentile = 1,443.11
> 99thPercentile = 1,472.4
> 999thPercentile = 1,858.1
>
>
> From: Nick Bailey [mailto:nick@datastax.com]
> Sent: Thursday, August 28, 2014 1:50 PM
> To: user@cassandra.apache.org
> Subject: Re: How often are JMX Cassandra metrics reset?
>
> Those percentile values should be for the lifetime of the node yes. Depending on what version of OpsCenter you are using it is either using the 'recent' metrics described by Rob, or it is using the FiveMinuteRate from JMX as well as doing some of it's own aggregation depending on the rollup size.
>
>
> On Thu, Aug 28, 2014 at 12:36 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Thu, Aug 28, 2014 at 9:27 AM, Donald Smith <Do...@audiencescience.com> wrote:
> And yet OpsCenter shows graphs with ever-changing metrics that show recent performance. Does OpsCenter not get its stats from JMX?
>
> 1) Certain JMX endpoints expose "recent" metrics, or at least used to. These are "recent" as in "since the last time someone polled this endpoint."
> 2) OpsCenter samples via JMX and then stores metrics in its own columnfamily. I would not be shocked if it does some minor aggregation as it does so.
>
> This all said, OpsCenter is not Apache Cassandra software, so the Apache Cassandra user mailing list may not be the ideal place for it to be discussed or supported...
>
> =Rob
Re: How often are JMX Cassandra metrics reset?
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Aug 28, 2014 at 3:39 PM, Donald Smith <
Donald.Smith@audiencescience.com> wrote:
> Maybe there’s a way to reset lifetime metrics to zero.
No. [1]
=Rob
[1] At least, they never have before and neither driftx or I believe they
have been created.
RE: How often are JMX Cassandra metrics reset?
Posted by Donald Smith <Do...@audiencescience.com>.
The metrics OneMinuteRate, FIveMinuteRate, FifteenMinuteRate, and MeanRate are NOT lifetime values but they’re all counts of requests, not latency. The latency values (Max, Count, 50thPercentile, Mean, etc) ARE lifetime values, I think, and thus would seem to be kinda useless for me, since our servers have been running for months.
Maybe there’s a way to reset lifetime metrics to zero. I connected to a cassandra server remotely via jConsole (port 7199) and I can read various metrics via the MBeans, but I don’t see an operation for resetting to zero. But perhaps that’s because I’m connecting remotely.
ClientRequest/Read/Latency:
LatencyUnit = MICROSECONDS
FiveMinuteRate = 1.12
FifteenMinuteRate = 1.11
RateUnit = SECONDS
MeanRate = 1.65
OneMinuteRate = 1.13
EventType = calls
Max = 237,373.37
Count = 961,312
50thPercentile = 383.2
Mean = 908.46
Min = 95.64
StdDev = 3,034.62
75thPercentile = 626.34
95thPercentile = 954.31
98thPercentile = 1,443.11
99thPercentile = 1,472.4
999thPercentile = 1,858.1
From: Nick Bailey [mailto:nick@datastax.com]
Sent: Thursday, August 28, 2014 1:50 PM
To: user@cassandra.apache.org
Subject: Re: How often are JMX Cassandra metrics reset?
Those percentile values should be for the lifetime of the node yes. Depending on what version of OpsCenter you are using it is either using the 'recent' metrics described by Rob, or it is using the FiveMinuteRate from JMX as well as doing some of it's own aggregation depending on the rollup size.
On Thu, Aug 28, 2014 at 12:36 PM, Robert Coli <rc...@eventbrite.com>> wrote:
On Thu, Aug 28, 2014 at 9:27 AM, Donald Smith <Do...@audiencescience.com>> wrote:
And yet OpsCenter shows graphs with ever-changing metrics that show recent performance. Does OpsCenter not get its stats from JMX?
1) Certain JMX endpoints expose "recent" metrics, or at least used to. These are "recent" as in "since the last time someone polled this endpoint."
2) OpsCenter samples via JMX and then stores metrics in its own columnfamily. I would not be shocked if it does some minor aggregation as it does so.
This all said, OpsCenter is not Apache Cassandra software, so the Apache Cassandra user mailing list may not be the ideal place for it to be discussed or supported...
=Rob
Re: How often are JMX Cassandra metrics reset?
Posted by Nick Bailey <ni...@datastax.com>.
Those percentile values should be for the lifetime of the node yes.
Depending on what version of OpsCenter you are using it is either using the
'recent' metrics described by Rob, or it is using the FiveMinuteRate from
JMX as well as doing some of it's own aggregation depending on the rollup
size.
On Thu, Aug 28, 2014 at 12:36 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Thu, Aug 28, 2014 at 9:27 AM, Donald Smith <
> Donald.Smith@audiencescience.com> wrote:
>
>> And yet OpsCenter shows graphs with ever-changing metrics that show
>> recent performance. Does OpsCenter not get its stats from JMX?
>>
>
> 1) Certain JMX endpoints expose "recent" metrics, or at least used to.
> These are "recent" as in "since the last time someone polled this endpoint."
> 2) OpsCenter samples via JMX and then stores metrics in its own
> columnfamily. I would not be shocked if it does some minor aggregation as
> it does so.
>
> This all said, OpsCenter is not Apache Cassandra software, so the Apache
> Cassandra user mailing list may not be the ideal place for it to be
> discussed or supported...
>
> =Rob
>
>
Re: How often are JMX Cassandra metrics reset?
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Aug 28, 2014 at 9:27 AM, Donald Smith <
Donald.Smith@audiencescience.com> wrote:
> And yet OpsCenter shows graphs with ever-changing metrics that show
> recent performance. Does OpsCenter not get its stats from JMX?
>
1) Certain JMX endpoints expose "recent" metrics, or at least used to.
These are "recent" as in "since the last time someone polled this endpoint."
2) OpsCenter samples via JMX and then stores metrics in its own
columnfamily. I would not be shocked if it does some minor aggregation as
it does so.
This all said, OpsCenter is not Apache Cassandra software, so the Apache
Cassandra user mailing list may not be the ideal place for it to be
discussed or supported...
=Rob
RE: How often are JMX Cassandra metrics reset?
Posted by Donald Smith <Do...@audiencescience.com>.
And yet OpsCenter shows graphs with ever-changing metrics that show recent performance. Does OpsCenter not get its stats from JMX?
From: Robert Coli [mailto:rcoli@eventbrite.com]
Sent: Wednesday, August 27, 2014 12:56 PM
To: user@cassandra.apache.org
Subject: Re: How often are JMX Cassandra metrics reset?
On Wed, Aug 27, 2014 at 12:38 PM, Donald Smith <Do...@audiencescience.com>> wrote:
I’m using JMX to retrieve Cassandra metrics. I notice that Max and Count are cumulative and aren’t reset. How often are the stats for Mean, 99tthPercentile, etc reset back to zero?
If they're like the old latency numbers, they are from node startup time and are never reset.
=Rob
Re: How often are JMX Cassandra metrics reset?
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, Aug 27, 2014 at 12:38 PM, Donald Smith <
Donald.Smith@audiencescience.com> wrote:
> I’m using JMX to retrieve Cassandra metrics. I notice that Max and
> Count are cumulative and aren’t reset. How often are the stats for Mean,
> 99tthPercentile, etc reset back to zero?
>
If they're like the old latency numbers, they are from node startup time
and are never reset.
=Rob