You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Clément Lardeur (JIRA)" <ji...@apache.org> on 2016/08/22 08:06:21 UTC
[jira] [Comment Edited] (CASSANDRA-11117)
ColUpdateTimeDeltaHistogram histogram overflow
[ https://issues.apache.org/jira/browse/CASSANDRA-11117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15430271#comment-15430271 ]
Clément Lardeur edited comment on CASSANDRA-11117 at 8/22/16 8:06 AM:
----------------------------------------------------------------------
Hi we have the same problem with the version 3.0.8, the graphite reporter stop working like CASSANDRA-12185.
{noformat}
ERROR [metrics-graphite-reporter-1-thread-1] 2016-08-22 09:40:42,618 ScheduledReporter.java:119 - RuntimeException thrown from GraphiteReporter#report. Exception was suppressed.
java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed
at org.apache.cassandra.utils.EstimatedHistogram.rawMean(EstimatedHistogram.java:231) ~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.metrics.EstimatedHistogramReservoir$HistogramSnapshot.getMean(EstimatedHistogramReservoir.java:103) ~[apache-cassandra-3.0.8.jar:3.0.8]
at com.codahale.metrics.graphite.GraphiteReporter.reportHistogram(GraphiteReporter.java:265) ~[metrics-graphite-3.1.2.jar:3.1.2]
at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:179) ~[metrics-graphite-3.1.2.jar:3.1.2]
at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) ~[metrics-core-3.1.0.jar:3.1.0]
at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) ~[metrics-core-3.1.0.jar:3.1.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
{noformat}
I don't think that ticket should be flagged as Minor because it's impacting our production monitoring of the cluster. Since the version 3.x with the incompatibility of OpsCenter, many users switch to another monitoring solution so the graphite reporter is now more critical for the Open Source community than previously.
If I could help I can give you more details with our cluster settings.
was (Author: clardeur):
Hi we have the same problem with the version 3.0.8, the graphite reporter stop working like CASSANDRA-12185.
{noformat}
ERROR [metrics-graphite-reporter-1-thread-1] 2016-08-22 09:40:42,618 ScheduledReporter.java:119 - RuntimeException thrown from GraphiteReporter#report. Exception was suppressed.
java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed
at org.apache.cassandra.utils.EstimatedHistogram.rawMean(EstimatedHistogram.java:231) ~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.metrics.EstimatedHistogramReservoir$HistogramSnapshot.getMean(EstimatedHistogramReservoir.java:103) ~[apache-cassandra-3.0.8.jar:3.0.8]
at com.codahale.metrics.graphite.GraphiteReporter.reportHistogram(GraphiteReporter.java:265) ~[metrics-graphite-3.1.2.jar:3.1.2]
at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:179) ~[metrics-graphite-3.1.2.jar:3.1.2]
at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) ~[metrics-core-3.1.0.jar:3.1.0]
at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) ~[metrics-core-3.1.0.jar:3.1.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
{noformat}
I don't think that ticket should be flagged as Minor because it's impacting our production monitoring of the cluster. Since the version 3.x with the incompatibility of OpsCenter, many users switch to another monitoring solution so the graphite reporter is now more critical for the OOS community than previously.
If I could help I can give you more details with our cluster settings.
> ColUpdateTimeDeltaHistogram histogram overflow
> ----------------------------------------------
>
> Key: CASSANDRA-11117
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11117
> Project: Cassandra
> Issue Type: Bug
> Reporter: Chris Lohfink
> Assignee: Joel Knighton
> Priority: Minor
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> {code}
> getting attribute Mean of org.apache.cassandra.metrics:type=ColumnFamily,name=ColUpdateTimeDeltaHistogram threw an exceptionjavax.management.RuntimeMBeanException: java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed
> {code}
> Although the fact that this histogram has 164 buckets already, I wonder if there is something weird with the computation thats causing this to be so large? It appears to be coming from updates to system.local
> {code}
> org.apache.cassandra.metrics:type=Table,keyspace=system,scope=local,name=ColUpdateTimeDeltaHistogram
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)