You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Jeffrey Wang <jw...@palantir.com> on 2011/03/17 05:55:42 UTC

insert during forced compaction

Hey all,

I'm running 0.7.0 on a cluster of 5 machines. When I create a new column family after I run nodetool compact (but before it finishes), I see the error below. Seems like StorageService.getValidColumnFamilies() should make a copy of the set of column families in the case where cfNames.length == 0. Incidentally, the interaction of schema changes with forcing compaction/flush/etc seems like it could be the source of many race conditions (e.g. CF is deleted before the compaction gets to it). Any thoughts?

-Jeffrey

java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
        at java.util.HashMap$ValueIterator.next(Unknown Source)
        at java.util.Collections$UnmodifiableCollection$1.next(Unknown Source)
        at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1140)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
        at sun.rmi.transport.Transport$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

Re: insert during forced compaction

Posted by Jonathan Ellis <jb...@gmail.com>.

We're aware of the potential for races during schema change but it
looks like we missed this one.  Can you create a ticket?

On Wed, Mar 16, 2011 at 11:55 PM, Jeffrey Wang <jw...@palantir.com> wrote:
> Hey all,
>
>
>
> I’m running 0.7.0 on a cluster of 5 machines. When I create a new column
> family after I run nodetool compact (but before it finishes), I see the
> error below. Seems like StorageService.getValidColumnFamilies() should make
> a copy of the set of column families in the case where cfNames.length == 0.
> Incidentally, the interaction of schema changes with forcing
> compaction/flush/etc seems like it could be the source of many race
> conditions (e.g. CF is deleted before the compaction gets to it). Any
> thoughts?
>
>
>
> -Jeffrey
>
>
>
> java.util.ConcurrentModificationException
>
>         at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
>
>         at java.util.HashMap$ValueIterator.next(Unknown Source)
>
>         at java.util.Collections$UnmodifiableCollection$1.next(Unknown
> Source)
>
>         at
> org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1140)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>
>         at java.lang.reflect.Method.invoke(Unknown Source)
>
>         at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
>
>         at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
>
>         at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
>
>         at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
>
>         at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
>
>         at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
>
>         at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
>
>         at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown
> Source)
>
>         at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown
> Source)
>
>         at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown
> Source)
>
>         at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown
> Source)
>
>         at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown
> Source)
>
>         at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
>
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>
>         at java.lang.reflect.Method.invoke(Unknown Source)
>
>         at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
>
>         at sun.rmi.transport.Transport$1.run(Unknown Source)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at sun.rmi.transport.Transport.serviceCall(Unknown Source)
>
>         at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
>
>         at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown
> Source)
>
>         at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown
> Source)
>
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>         at java.lang.Thread.run(Unknown Source)



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com