You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Mike (JIRA)" <ji...@apache.org> on 2016/07/13 06:47:20 UTC
[jira] [Updated] (CASSANDRA-12185) Exception during metrics
calculation
[ https://issues.apache.org/jira/browse/CASSANDRA-12185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike updated CASSANDRA-12185:
-----------------------------
Description:
I am trying to report Cassandra metrics to Graphite server using metrics-graphite. When there is no load on the cluster everything works fine and all metrics are reported properly. But if some load occurs, I receive following exception in system.log:
{noformat}
ERROR [metrics-graphite-reporter-1-thread-1] 2016-07-13 08:21:23,580 ScheduledReporter.java:119 - RuntimeException thrown from GraphiteReporter#report. Exception was suppressed.
java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed
at org.apache.cassandra.utils.EstimatedHistogram.rawMean(EstimatedHistogram.java:231) ~[apache-cassandra-3.0.7.jar:3.0.7]
at org.apache.cassandra.metrics.EstimatedHistogramReservoir$HistogramSnapshot.getMean(EstimatedHistogramReservoir.java:103) ~[apache-cassandra-3.0.7.jar:3.0.7]
at com.codahale.metrics.graphite.GraphiteReporter.reportHistogram(GraphiteReporter.java:265) ~[metrics-graphite-3.1.2.jar:3.1.2]
at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:179) ~[metrics-graphite-3.1.2.jar:3.1.2]
at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) ~[metrics-core-3.1.0.jar:3.1.0]
at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) ~[metrics-core-3.1.0.jar:3.1.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
{noformat}
This message is repeated every second on every Cassandra node and some metrics become unavailable. In order to receive the metrics again, I have to restart all Cassandra nodes. I tried different metrics-graphite versions from 3.1.0 to 3.1.2 with the same issue.
was:
I am trying to report Cassandra metrics to Graphite server using metrics-graphite. When there is no load on the cluster everything works fine and all metrics are reported properly. But if some load occurs, I receive following exception in system.log:
{noformat}
ERROR [metrics-graphite-reporter-1-thread-1] 2016-07-13 08:21:23,580 ScheduledReporter.java:119 - RuntimeException thrown from GraphiteReporter#report. Exception was suppressed.
java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed
at org.apache.cassandra.utils.EstimatedHistogram.rawMean(EstimatedHistogram.java:231) ~[apache-cassandra-3.0.7.jar:3.0.7]
at org.apache.cassandra.metrics.EstimatedHistogramReservoir$HistogramSnapshot.getMean(EstimatedHistogramReservoir.java:103) ~[apache-cassandra-3.0.7.jar:3.0.7]
at com.codahale.metrics.graphite.GraphiteReporter.reportHistogram(GraphiteReporter.java:265) ~[metrics-graphite-3.1.2.jar:3.1.2]
at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:179) ~[metrics-graphite-3.1.2.jar:3.1.2]
at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) ~[metrics-core-3.1.0.jar:3.1.0]
at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) ~[metrics-core-3.1.0.jar:3.1.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
{noformat}
This message is repeated every second on every Cassandra node and some metrics become unavailable. In order to receive the metrics again, I have to restart all Cassandra nodes.
> Exception during metrics calculation
> ------------------------------------
>
> Key: CASSANDRA-12185
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12185
> Project: Cassandra
> Issue Type: Bug
> Components: Observability
> Environment: Ubuntu 14.04, Java 1.8.0_91, metrics-graphite-3.1.2
> Reporter: Mike
>
> I am trying to report Cassandra metrics to Graphite server using metrics-graphite. When there is no load on the cluster everything works fine and all metrics are reported properly. But if some load occurs, I receive following exception in system.log:
> {noformat}
> ERROR [metrics-graphite-reporter-1-thread-1] 2016-07-13 08:21:23,580 ScheduledReporter.java:119 - RuntimeException thrown from GraphiteReporter#report. Exception was suppressed.
> java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed
> at org.apache.cassandra.utils.EstimatedHistogram.rawMean(EstimatedHistogram.java:231) ~[apache-cassandra-3.0.7.jar:3.0.7]
> at org.apache.cassandra.metrics.EstimatedHistogramReservoir$HistogramSnapshot.getMean(EstimatedHistogramReservoir.java:103) ~[apache-cassandra-3.0.7.jar:3.0.7]
> at com.codahale.metrics.graphite.GraphiteReporter.reportHistogram(GraphiteReporter.java:265) ~[metrics-graphite-3.1.2.jar:3.1.2]
> at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:179) ~[metrics-graphite-3.1.2.jar:3.1.2]
> at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) ~[metrics-core-3.1.0.jar:3.1.0]
> at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) ~[metrics-core-3.1.0.jar:3.1.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_91]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_91]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_91]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> {noformat}
> This message is repeated every second on every Cassandra node and some metrics become unavailable. In order to receive the metrics again, I have to restart all Cassandra nodes. I tried different metrics-graphite versions from 3.1.0 to 3.1.2 with the same issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)