[jira] [Created] (CASSANDRA-12978) mx4j -> HTTP 500 -> ConcurrentModificationException

["Rob Emery (JIRA)" <ji...@apache.org>]

Rob Emery created CASSANDRA-12978:
-------------------------------------

             Summary: mx4j -> HTTP 500 -> ConcurrentModificationException
                 Key: CASSANDRA-12978
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12978
             Project: Cassandra
          Issue Type: Bug
          Components: Tools
         Environment: Debian, Single cluster, 2 data centres, E5-2620 v3, 16GB, RAID1 SSD Commit log, RAID10 15k HDD data
            Reporter: Rob Emery
             Fix For: 2.1.6


We run some checks from our Monitoring software that rely on mx4j.

The checks typically grab some xml via HTTP request and parse it. For 
example, CF Stats on 'MyKeySpace' and 'MyColumnFamily' are retrieved 
using:

http://cassandra001:8081/mbean?template=identity&objectname=org.apache.cassandra.db%3Atype%3DColumnFamilies%2Ckeyspace%3DMyKeySpace%2Ccolumnfamily%3DMyColumnFamily

The checks run each minute. Periodically they result in a "HTTP 500 internal server error". The HTML body returned is empty.

Experimentally we ran Cassandra in the foreground on one node and reproduced the problem. this elicited the following stack trace:

javax.management.RuntimeMBeanException: java.util.ConcurrentModificationException
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
        at mx4j.tools.adaptor.http.MBeanCommandProcessor.createMBeanElement(MBeanCommandProcessor.java:119)
        at mx4j.tools.adaptor.http.MBeanCommandProcessor.executeRequest(MBeanCommandProcessor.java:56)
        at mx4j.tools.adaptor.http.HttpAdaptor$HttpClient.run(HttpAdaptor.java:980)
Caused by: java.util.ConcurrentModificationException
        at java.util.TreeMap$NavigableSubMap$SubMapIterator.nextEntry(TreeMap.java:1594)
        at java.util.TreeMap$NavigableSubMap$SubMapEntryIterator.next(TreeMap.java:1642)
        at java.util.TreeMap$NavigableSubMap$SubMapEntryIterator.next(TreeMap.java:1636)
        at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
        at org.apache.cassandra.utils.StreamingHistogram.sum(StreamingHistogram.java:160)
        at org.apache.cassandra.io.sstable.metadata.StatsMetadata.getDroppableTombstonesBefore(StatsMetadata.java:113)
        at org.apache.cassandra.io.sstable.SSTableReader.getDroppableTombstonesBefore(SSTableReader.java:2004)
        at org.apache.cassandra.db.DataTracker.getDroppableTombstoneRatio(DataTracker.java:507)
        at org.apache.cassandra.db.ColumnFamilyStore.getDroppableTombstoneRatio(ColumnFamilyStore.java:3089)
        at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
        at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
        at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
        ... 4 more




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)