You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jai Bheemsen Rao Dhanwada <ja...@gmail.com> on 2018/02/26 07:21:54 UTC
Cassandra Hints monitoring
Hello,
I am trying to evaluate the JMX hints metrics for cassandra(TotalHints).
However, when I am testing the metrics, the total hints accumulated count
shows perfectly fine, but when the hints are replayed and the hints
count(cql> SELECT * from system.hints; ) becomes zero on cql, and the JMX
metrics counter doesn't reset to zero and stays at the last known non-zero
value(eg: 6). when new hints are created the total get added to the last
non-zero count(e.g 6+2). Any idea if there is a way to get the active count
during the given time period?
I am using the JMX metrics on Cassandra nodes and sending the data to
Influx DB and querying it using Grafana with mean using
mean("TotalHints_Count") from storage-metrics with group by 10s.
Any pointers on this is higly appreciated.
Re: Cassandra Hints monitoring
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
This works when the count value is changing/increasing.
In my case if the Hints total value is 10 for 2 hours, the
difference(sum("TotalHints_Count")) is going to be 10 for that 10 seconds
time period and then it will be 0 later on. where as the value should be
constantly 10 until there is a change in number for the count.
On Mon, Feb 26, 2018 at 2:18 AM, Nicolas Guyomar <ni...@gmail.com>
wrote:
> I find DIFFERENCE to be working for dropped mutation, which IMHO works
> the same way as Hint metrics
> *select difference(sum("Dropped_Count")) FROM "cassandraDroppedMessage"
> groupby host* is valid when I check with nodetool over a period of time
>
> Not sure what is not working on your side
>
> On 26 February 2018 at 11:02, Jai Bheemsen Rao Dhanwada <
> jaibheemsen@gmail.com> wrote:
>
>> DIFFERENCE may not work here, if the Hints count is 10 for few hours, the
>> difference is always is zero. which is not the correct value.
>>
>> On Mon, Feb 26, 2018 at 1:18 AM, Jai Bheemsen Rao Dhanwada <
>> jaibheemsen@gmail.com> wrote:
>>
>>> Thanks Alex,
>>>
>>> Let me try it.
>>>
>>>
>>> On Monday, February 26, 2018, Oleksandr Shulgin <
>>> oleksandr.shulgin@zalando.de> wrote:
>>>
>>>> On Mon, Feb 26, 2018 at 10:02 AM, Jai Bheemsen Rao Dhanwada <
>>>> jaibheemsen@gmail.com> wrote:
>>>>
>>>>> Thank you Alex,
>>>>>
>>>>> I tried "TotalHintsInProgress" already, and I don't see it sending
>>>>> the correct metrics. I used mean("TotalHintsInProgress") and I see 0
>>>>> always on grafana.
>>>>> Do you know what is the correct way to do rate or diff for hints using
>>>>> "TotalHints"? I am currently using the below query
>>>>>
>>>>> SELECT mean("TotalHints_Count") FROM "cassandraStorage"
>>>>> WHERE $timeFilter GROUP BY time(10s)
>>>>>
>>>>
>>>> I've never really used InfluxDB, but it seems that DIFFERENCE is what
>>>> you're looking for: https://docs.influxdata.c
>>>> om/influxdb/v1.4/query_language/functions/#difference
>>>>
>>>> --
>>>> Alex
>>>>
>>>>
>>
>
Re: Cassandra Hints monitoring
Posted by Nicolas Guyomar <ni...@gmail.com>.
I find DIFFERENCE to be working for dropped mutation, which IMHO works the
same way as Hint metrics
*select difference(sum("Dropped_Count")) FROM "cassandraDroppedMessage"
groupby host* is valid when I check with nodetool over a period of time
Not sure what is not working on your side
On 26 February 2018 at 11:02, Jai Bheemsen Rao Dhanwada <
jaibheemsen@gmail.com> wrote:
> DIFFERENCE may not work here, if the Hints count is 10 for few hours, the
> difference is always is zero. which is not the correct value.
>
> On Mon, Feb 26, 2018 at 1:18 AM, Jai Bheemsen Rao Dhanwada <
> jaibheemsen@gmail.com> wrote:
>
>> Thanks Alex,
>>
>> Let me try it.
>>
>>
>> On Monday, February 26, 2018, Oleksandr Shulgin <
>> oleksandr.shulgin@zalando.de> wrote:
>>
>>> On Mon, Feb 26, 2018 at 10:02 AM, Jai Bheemsen Rao Dhanwada <
>>> jaibheemsen@gmail.com> wrote:
>>>
>>>> Thank you Alex,
>>>>
>>>> I tried "TotalHintsInProgress" already, and I don't see it sending the
>>>> correct metrics. I used mean("TotalHintsInProgress") and I see 0
>>>> always on grafana.
>>>> Do you know what is the correct way to do rate or diff for hints using "TotalHints"?
>>>> I am currently using the below query
>>>>
>>>> SELECT mean("TotalHints_Count") FROM "cassandraStorage"
>>>> WHERE $timeFilter GROUP BY time(10s)
>>>>
>>>
>>> I've never really used InfluxDB, but it seems that DIFFERENCE is what
>>> you're looking for: https://docs.influxdata.c
>>> om/influxdb/v1.4/query_language/functions/#difference
>>>
>>> --
>>> Alex
>>>
>>>
>
Re: Cassandra Hints monitoring
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
DIFFERENCE may not work here, if the Hints count is 10 for few hours, the
difference is always is zero. which is not the correct value.
On Mon, Feb 26, 2018 at 1:18 AM, Jai Bheemsen Rao Dhanwada <
jaibheemsen@gmail.com> wrote:
> Thanks Alex,
>
> Let me try it.
>
>
> On Monday, February 26, 2018, Oleksandr Shulgin <
> oleksandr.shulgin@zalando.de> wrote:
>
>> On Mon, Feb 26, 2018 at 10:02 AM, Jai Bheemsen Rao Dhanwada <
>> jaibheemsen@gmail.com> wrote:
>>
>>> Thank you Alex,
>>>
>>> I tried "TotalHintsInProgress" already, and I don't see it sending the
>>> correct metrics. I used mean("TotalHintsInProgress") and I see 0 always
>>> on grafana.
>>> Do you know what is the correct way to do rate or diff for hints using "TotalHints"?
>>> I am currently using the below query
>>>
>>> SELECT mean("TotalHints_Count") FROM "cassandraStorage"
>>> WHERE $timeFilter GROUP BY time(10s)
>>>
>>
>> I've never really used InfluxDB, but it seems that DIFFERENCE is what
>> you're looking for: https://docs.influxdata.c
>> om/influxdb/v1.4/query_language/functions/#difference
>>
>> --
>> Alex
>>
>>
Re: Cassandra Hints monitoring
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
Thanks Alex,
Let me try it.
On Monday, February 26, 2018, Oleksandr Shulgin <
oleksandr.shulgin@zalando.de> wrote:
> On Mon, Feb 26, 2018 at 10:02 AM, Jai Bheemsen Rao Dhanwada <
> jaibheemsen@gmail.com> wrote:
>
>> Thank you Alex,
>>
>> I tried "TotalHintsInProgress" already, and I don't see it sending the
>> correct metrics. I used mean("TotalHintsInProgress") and I see 0 always
>> on grafana.
>> Do you know what is the correct way to do rate or diff for hints using "TotalHints"?
>> I am currently using the below query
>>
>> SELECT mean("TotalHints_Count") FROM "cassandraStorage" WHERE $timeFilter
>> GROUP BY time(10s)
>>
>
> I've never really used InfluxDB, but it seems that DIFFERENCE is what
> you're looking for: https://docs.influxdata.com/influxdb/v1.4/query_
> language/functions/#difference
>
> --
> Alex
>
>
Re: Cassandra Hints monitoring
Posted by Oleksandr Shulgin <ol...@zalando.de>.
On Mon, Feb 26, 2018 at 10:02 AM, Jai Bheemsen Rao Dhanwada <
jaibheemsen@gmail.com> wrote:
> Thank you Alex,
>
> I tried "TotalHintsInProgress" already, and I don't see it sending the
> correct metrics. I used mean("TotalHintsInProgress") and I see 0 always
> on grafana.
> Do you know what is the correct way to do rate or diff for hints using "TotalHints"?
> I am currently using the below query
>
> SELECT mean("TotalHints_Count") FROM "cassandraStorage" WHERE $timeFilter
> GROUP BY time(10s)
>
I've never really used InfluxDB, but it seems that DIFFERENCE is what
you're looking for:
https://docs.influxdata.com/influxdb/v1.4/query_language/functions/#difference
--
Alex
Re: Cassandra Hints monitoring
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
Thank you Alex,
I tried "TotalHintsInProgress" already, and I don't see it sending the
correct metrics. I used mean("TotalHintsInProgress") and I see 0 always on
grafana.
Do you know what is the correct way to do rate or diff for hints using
"TotalHints"?
I am currently using the below query
SELECT mean("TotalHints_Count") FROM "cassandraStorage" WHERE $timeFilter
GROUP BY time(10s)
On Mon, Feb 26, 2018 at 12:26 AM, Oleksandr Shulgin <
oleksandr.shulgin@zalando.de> wrote:
> On Mon, Feb 26, 2018 at 8:21 AM, Jai Bheemsen Rao Dhanwada <
> jaibheemsen@gmail.com> wrote:
>
>>
>> I am trying to evaluate the JMX hints metrics for cassandra(TotalHints).
>> However, when I am testing the metrics, the total hints accumulated count
>> shows perfectly fine, but when the hints are replayed and the hints
>> count(cql> SELECT * from system.hints; ) becomes zero on cql, and the JMX
>> metrics counter doesn't reset to zero and stays at the last known non-zero
>> value(eg: 6). when new hints are created the total get added to the last
>> non-zero count(e.g 6+2). Any idea if there is a way to get the active count
>> during the given time period?
>>
>
> That's correct. The "TotalHints" metric gives you a running total since
> the last startup of the node. In Grafana you can use rate() or diff()
> aggregation on top of that, with a caveat that it produces a huge negative
> when the node is actually restarted and the next reading is less than the
> previous.
>
> There is also "TotalHintsInProgress" which might be what you're looking
> for, this one is reset back to 0 by the node once the hints are processed.
>
> Cheers,
> --
> Alex
>
>
Re: Cassandra Hints monitoring
Posted by Oleksandr Shulgin <ol...@zalando.de>.
On Mon, Feb 26, 2018 at 8:21 AM, Jai Bheemsen Rao Dhanwada <
jaibheemsen@gmail.com> wrote:
>
> I am trying to evaluate the JMX hints metrics for cassandra(TotalHints).
> However, when I am testing the metrics, the total hints accumulated count
> shows perfectly fine, but when the hints are replayed and the hints
> count(cql> SELECT * from system.hints; ) becomes zero on cql, and the JMX
> metrics counter doesn't reset to zero and stays at the last known non-zero
> value(eg: 6). when new hints are created the total get added to the last
> non-zero count(e.g 6+2). Any idea if there is a way to get the active count
> during the given time period?
>
That's correct. The "TotalHints" metric gives you a running total since
the last startup of the node. In Grafana you can use rate() or diff()
aggregation on top of that, with a caveat that it produces a huge negative
when the node is actually restarted and the next reading is less than the
previous.
There is also "TotalHintsInProgress" which might be what you're looking
for, this one is reset back to 0 by the node once the hints are processed.
Cheers,
--
Alex