nodetool scrub throws exception FileAlreadyExistsException

[Donald Smith <Do...@audiencescience.com>]

% time nodetool scrub -s as_reports data_report_info_2011
xss =  -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M -Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
Exception in thread "main" FSWriteError in /mnt/cassandra-storage/data/as_reports/data_report_info_2011/snapshots/pre-scrub-1395848747073/as_reports-data_report_info_2011-jb-3-Data.db
        at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:84)
        at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:1215)
        at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1817)
        at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:1123)
        at org.apache.cassandra.service.StorageService.scrub(StorageService.java:2197)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at sun.reflect.misc.Trampoline.invoke(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at sun.reflect.misc.MethodUtil.invoke(Unknown Source)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
        at sun.rmi.transport.Transport$1.run(Unknown Source)
        at sun.rmi.transport.Transport$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.nio.file.FileAlreadyExistsException: /mnt/cassandra-storage/data/as_reports/data_report_info_2011/snapshots/pre-scrub-1395848747073/as_reports-data_report_info_2011-jb-3-Data.db -> /mnt/cassandra-storage/data/as_reports/data_report_info_2011/as_reports-data_report_info_2011-jb-3-Data.db
        at sun.nio.fs.UnixException.translateToIOException(Unknown Source)
        at sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
        at sun.nio.fs.UnixFileSystemProvider.createLink(Unknown Source)
        at java.nio.file.Files.createLink(Unknown Source)
        at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:80)
        ... 39 more
1.112u 0.122s 3:38.36 0.5%      0+0k 0+328io 0pf+0w


That table is new and very unlikely to be corrupted.  I retried the command without "-s" and it succeeded right away. I tried again WITH "-s" and it succeeded again too.

Donald A. Smith | Senior Software Engineer
P: 425.201.3900 x 3866
C: (206) 819-5965
F: (646) 443-2333
donalds@AudienceScience.com<ma...@AudienceScience.com>

[AudienceScience]