You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by David King <dk...@ketralnis.com> on 2011/01/19 08:10:19 UTC

Keys must be written in ascending order

I'm upgrading an 0.6 cluster to 0.7 in a testing environment. In cleaning up one of the nodes I get the exception below. Googling around seems to reveal people having trouble with it caused by too-small heap sizes but that doesn't look to be what's going on here. Am I missing something obvious?

$ time ./cassandra-0.7/bin/nodetool -h cassa7test01 cleanup
Error occured while cleaning up keyspace keyspace
java.util.concurrent.ExecutionException: java.io.IOException: Keys must be written in ascending order.
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
        at org.apache.cassandra.db.CompactionManager.performCleanup(CompactionManager.java:180)
        at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:909)
        at org.apache.cassandra.service.StorageService.forceTableCleanup(StorageService.java:1127)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1449)
        at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1284)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1382)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
        at sun.rmi.transport.Transport$1.run(Transport.java:177)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.IOException: Keys must be written in ascending order.
        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:107)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:124)
        at org.apache.cassandra.db.CompactionManager.doCleanupCompaction(CompactionManager.java:411)
        at org.apache.cassandra.db.CompactionManager.access$400(CompactionManager.java:54)
        at org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:171)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        ... 3 more

real    14m27.895s
user    0m0.670s
sys     0m0.200s

Re: Keys must be written in ascending order

Posted by Jonathan Ellis <jb...@gmail.com>.
On Wed, Jan 19, 2011 at 2:26 AM, aaron morton <aa...@thelastpickle.com> wrote:
> David,
>        There was an issue similar to this reported during the 0.7 development https://issues.apache.org/jira/browse/CASSANDRA-1744 which was marked as duplicate with https://issues.apache.org/jira/browse/CASSANDRA-1743 It looks like the fix for 1743 did not make it into the 0.7 release, but is in the 0.7 branch. I think your's is a bit different but it might pay to test the fix.

What was going on there is, the initial version of 1743 caused sstable
corruption (1744) and was not committed in 0.7.0.  After fixing 1743,
it was committed for 0.7.1.

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: Keys must be written in ascending order

Posted by aaron morton <aa...@thelastpickle.com>.
David, 
	There was an issue similar to this reported during the 0.7 development https://issues.apache.org/jira/browse/CASSANDRA-1744 which was marked as duplicate with https://issues.apache.org/jira/browse/CASSANDRA-1743 It looks like the fix for 1743 did not make it into the 0.7 release, but is in the 0.7 branch. I think your's is a bit different but it might pay to test the fix.

I cannot find hudson builds for the 0.7 branch (only trunk builds https://hudson.apache.org/hudson/job/Cassandra/), are you able to grab the 0.7 branch from svn or git and build it yourself and test it ? 

The other possibility is a change in the partitioner http://www.mail-archive.com/user@cassandra.apache.org/msg02587.html

Cheers
Aaron



	
On 19 Jan 2011, at 20:10, David King wrote:

> I'm upgrading an 0.6 cluster to 0.7 in a testing environment. In cleaning up one of the nodes I get the exception below. Googling around seems to reveal people having trouble with it caused by too-small heap sizes but that doesn't look to be what's going on here. Am I missing something obvious?
> 
> $ time ./cassandra-0.7/bin/nodetool -h cassa7test01 cleanup
> Error occured while cleaning up keyspace keyspace
> java.util.concurrent.ExecutionException: java.io.IOException: Keys must be written in ascending order.
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>        at org.apache.cassandra.db.CompactionManager.performCleanup(CompactionManager.java:180)
>        at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:909)
>        at org.apache.cassandra.service.StorageService.forceTableCleanup(StorageService.java:1127)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:616)
>        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
>        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
>        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
>        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
>        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
>        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
>        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
>        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1449)
>        at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
>        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1284)
>        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1382)
>        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:616)
>        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
>        at sun.rmi.transport.Transport$1.run(Transport.java:177)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
>        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
>        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
>        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> Caused by: java.io.IOException: Keys must be written in ascending order.
>        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:107)
>        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:124)
>        at org.apache.cassandra.db.CompactionManager.doCleanupCompaction(CompactionManager.java:411)
>        at org.apache.cassandra.db.CompactionManager.access$400(CompactionManager.java:54)
>        at org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:171)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>        ... 3 more
> 
> real    14m27.895s
> user    0m0.670s
> sys     0m0.200s
> 
>

Re: Keys must be written in ascending order

Posted by Jonathan Ellis <jb...@gmail.com>.
If you run sstablekeys it will tell you which key is out of orde --
assuming your index and data files contain the same keys, which if you
have a corrupt sstable may or may not be the case.

On Wed, Jan 19, 2011 at 1:10 AM, David King <dk...@ketralnis.com> wrote:
> I'm upgrading an 0.6 cluster to 0.7 in a testing environment. In cleaning up one of the nodes I get the exception below. Googling around seems to reveal people having trouble with it caused by too-small heap sizes but that doesn't look to be what's going on here. Am I missing something obvious?
>
> $ time ./cassandra-0.7/bin/nodetool -h cassa7test01 cleanup
> Error occured while cleaning up keyspace keyspace
> java.util.concurrent.ExecutionException: java.io.IOException: Keys must be written in ascending order.
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>        at org.apache.cassandra.db.CompactionManager.performCleanup(CompactionManager.java:180)
>        at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:909)
>        at org.apache.cassandra.service.StorageService.forceTableCleanup(StorageService.java:1127)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:616)
>        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
>        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
>        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
>        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
>        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
>        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
>        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
>        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1449)
>        at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
>        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1284)
>        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1382)
>        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:616)
>        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
>        at sun.rmi.transport.Transport$1.run(Transport.java:177)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
>        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
>        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
>        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> Caused by: java.io.IOException: Keys must be written in ascending order.
>        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:107)
>        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:124)
>        at org.apache.cassandra.db.CompactionManager.doCleanupCompaction(CompactionManager.java:411)
>        at org.apache.cassandra.db.CompactionManager.access$400(CompactionManager.java:54)
>        at org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:171)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>        ... 3 more
>
> real    14m27.895s
> user    0m0.670s
> sys     0m0.200s
>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com