[jira] [Created] (CASSANDRA-9973) java.lang.IllegalStateException: Unable to compute when histogram overflowed

["Mark Manley (JIRA)" <ji...@apache.org>]

Mark Manley created CASSANDRA-9973:
--------------------------------------

             Summary: java.lang.IllegalStateException: Unable to compute when histogram overflowed
                 Key: CASSANDRA-9973
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9973
             Project: Cassandra
          Issue Type: Bug
          Components: Core
            Reporter: Mark Manley
             Fix For: 2.2.x


I recently, and probably mistakenly, upgraded one of my production C* clusters to 2.2.0.  I am seeing these errors in the logs, followed by an intense period of garbage collection until the node, then the ring, becomes crippled:

{code}
ERROR [OptionalTasks:1] 2015-08-04 03:24:56,057 CassandraDaemon.java:182 - Exception in thread Thread[OptionalTasks:1,5,main]
java.lang.IllegalStateException: Unable to compute when histogram overflowed
        at org.apache.cassandra.utils.EstimatedHistogram.percentile(EstimatedHistogram.java:179) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.metrics.EstimatedHistogramReservoir$HistogramSnapshot.getValue(EstimatedHistogramReservoir.java:84) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.db.ColumnFamilyStore$3.run(ColumnFamilyStore.java:405) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
{code}

I am not sure if the GC instability is this or something else, but I though this histogram overflow issue was fixed in 2.1.3?  Anyway, reporting now as a possible regression.  Please let me know what I can provide in terms of information to help with this.  Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)