You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Tyler Patterson (JIRA)" <ji...@apache.org> on 2012/05/04 15:24:47 UTC
[jira] [Created] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Tyler Patterson created CASSANDRA-4221:
------------------------------------------
Summary: Error while deleting a columnfamily that is being compacted.
Key: CASSANDRA-4221
URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
Reporter: Tyler Patterson
The following dtest command produces an error:
{code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
Here is the error:
{code}
Error occured during compaction
java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
... 3 more
Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
... 13 more
{code}
For reference, here is the dtest function that causes the failure:
{code}
def load_test(self):
"""
apply schema changes while the cluster is under load.
"""
debug("load_test()")
cluster = self.cluster
cluster.populate(1).start()
node1 = cluster.nodelist()[0]
wait(2)
cursor = self.cql_connection(node1).cursor()
def stress(args=[]):
debug("Stressing")
node1.stress(args)
debug("Done Stressing")
def compact():
debug("Compacting...")
node1.nodetool('compact')
debug("Done Compacting.")
# put some data into the cluster
stress(['--num-keys=1000000'])
# now start compacting...
tcompact = Thread(target=compact)
tcompact.start()
wait(1)
# now the cluster is under a lot of load. Make some schema changes.
cursor.execute("USE Keyspace1")
wait(1)
cursor.execute("DROP COLUMNFAMILY Standard1")
wait(3)
cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
tcompact.join()
{code}
Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281982#comment-13281982 ]
Pavel Yaskevich commented on CASSANDRA-4221:
--------------------------------------------
bq. stopCompactionFor should take CFS parameters instead of String
I don't really follow here, if you want it to have list of CFMetaData instead of String? String is better suited because CompactionInfo.getColumnFamily() returns a String (CF name).
bq. I don't see any reason to not wait indefinitely here; in fact, if we make sure to wait until compaction finishes, the odds are much better that when we tell the client "all done" he won't be able to send a "create" quickly enough to hit the bug
We don't really try to wait indefinitely here, just for 30 seconds (worst case), if compactions don't finish until then we just move on with delete. do you want it to wait until all compactions to finish?
bq. Need to call stopCompactionFor on every replica, not just CompactionServer – move this to DefsTable.dropColumnFamily?
I agree, I'm going to move that into dropColumnFamily call so it gets called on the replicas too.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch, system.log
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Tyler Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tyler Patterson updated CASSANDRA-4221:
---------------------------------------
Attachment: system.log
This was after applying both patches to the cassandra-1.1 branch, and setting logging to DEBUG.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch, system.log
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13282007#comment-13282007 ]
Pavel Yaskevich commented on CASSANDRA-4221:
--------------------------------------------
bq. It throws IOException. Remember that we check for ability to abort compaction every row; if we're compacting a wide row, it could easily take over 30s w/ throttling.
Oh yes, sorry. I think we can just remove that exception and move on with drop, or do you want it to until all compactions finish?
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch, system.log
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4221:
---------------------------------------
Attachment: CASSANDRA-4221-logging.patch
Interesting, I can't reproduce it myself. Can you please run it with logging patch attached (and enabled DEBUG logging) and attach debug log from your C* node to this task, so I can check that is happening inside of DataTracker in your case?...
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4221:
---------------------------------------
Attachment: (was: CASSANDRA-4230.patch)
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271724#comment-13271724 ]
Pavel Yaskevich commented on CASSANDRA-4221:
--------------------------------------------
For the KS or CF drop this seems necessary to try to wait until all running compactions finish otherwise it would end up in errors like one in the description, also other operations - create, update - are not affected by this.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279130#comment-13279130 ]
Pavel Yaskevich commented on CASSANDRA-4221:
--------------------------------------------
Hah, now I know what is causing it - it's not a drop problem, the situation is triggered when you re-create ColumnFamily right after drop +(before all SSTables were actually deleted by background task)+ so it reads up all SSTables in the directory back to system and tries to compact them simultaneously with them being deleted in the background. That is why we warn people to *avoid* making any modifications to the active CFs otherwise it could lead to the strange situations like this one.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch, system.log
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4221:
---------------------------------------
Attachment: CASSANDRA-4221-v2.patch
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221-v2.patch, CASSANDRA-4221.patch, system.log
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Tyler Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13278043#comment-13278043 ]
Tyler Patterson commented on CASSANDRA-4221:
--------------------------------------------
Somehow that server.log did not have the debug info. Looking into it now.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276866#comment-13276866 ]
Jonathan Ellis commented on CASSANDRA-4221:
-------------------------------------------
Tyler, can you still reproduce after the recent schema fixes on the 1.1 branch?
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279580#comment-13279580 ]
Jonathan Ellis commented on CASSANDRA-4221:
-------------------------------------------
Would this be fixed by CASSANDRA-3794 then, since old and new CF will have different IDs?
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch, system.log
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13278219#comment-13278219 ]
Pavel Yaskevich commented on CASSANDRA-4221:
--------------------------------------------
I see debug information I added right now, but there is no IOError in that log described in this task...
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch, system_log.zip
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271649#comment-13271649 ]
Pavel Yaskevich commented on CASSANDRA-4221:
--------------------------------------------
This one seems to be caused by the same problem as CASSANDRA-4230.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Tyler Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277537#comment-13277537 ]
Tyler Patterson commented on CASSANDRA-4221:
--------------------------------------------
Yes, the error just happened again for me. I did a fresh pull on branch branch cassandra-1.1.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Tyler Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tyler Patterson updated CASSANDRA-4221:
---------------------------------------
Attachment: system_log.zip
Debug is enabled now; It looks like CCM overwrites the log level. Only the logging patch was applied in this run.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch, system_log.zip
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4221:
---------------------------------------
Attachment: CASSANDRA-4230.patch
Patch adds a try to stop all running compactions on given Keyspace or ColumnFamily before running a drop command. I have tried the test you have in the description and it ran without failures.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Attachments: CASSANDRA-4230.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Tyler Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13278946#comment-13278946 ]
Tyler Patterson edited comment on CASSANDRA-4221 at 5/18/12 4:53 PM:
---------------------------------------------------------------------
So after some experimentation, the problem is only happening when the log level is set to INFO, but it doesn't happen at DEBUG. Gotta love these ones! I modified the logging patch to do logging.info() rather then logging.debug(), and the problem still happens, so at least you can see those debug messages. I hope this is enough to go on.
was (Author: tpatterson):
So after some experimentation, the problem is only happening when log_level is set at info, NOT debug. Gotta love these ones! I modified the logging patch to do logging.info() rather then logging.debug(), and the problem still happens, so at least you can see those debug messages. I hope this is enough to go on.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch, system.log
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Tyler Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tyler Patterson updated CASSANDRA-4221:
---------------------------------------
Description:
The following dtest command produces an error:
{code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
Here is the error:
{code}
Error occured during compaction
java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
... 3 more
Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
... 13 more
{code}
For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
{code}
def load_test(self):
"""
apply schema changes while the cluster is under load.
"""
debug("load_test()")
cluster = self.cluster
cluster.populate(1).start()
node1 = cluster.nodelist()[0]
wait(2)
cursor = self.cql_connection(node1).cursor()
def stress(args=[]):
debug("Stressing")
node1.stress(args)
debug("Done Stressing")
def compact():
debug("Compacting...")
node1.nodetool('compact')
debug("Done Compacting.")
# put some data into the cluster
stress(['--num-keys=1000000'])
# now start compacting...
tcompact = Thread(target=compact)
tcompact.start()
wait(1)
# now the cluster is under a lot of load. Make some schema changes.
cursor.execute("USE Keyspace1")
wait(1)
cursor.execute("DROP COLUMNFAMILY Standard1")
wait(3)
cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
tcompact.join()
{code}
Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
was:
The following dtest command produces an error:
{code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
Here is the error:
{code}
Error occured during compaction
java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
... 3 more
Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
... 13 more
{code}
For reference, here is the dtest function that causes the failure:
{code}
def load_test(self):
"""
apply schema changes while the cluster is under load.
"""
debug("load_test()")
cluster = self.cluster
cluster.populate(1).start()
node1 = cluster.nodelist()[0]
wait(2)
cursor = self.cql_connection(node1).cursor()
def stress(args=[]):
debug("Stressing")
node1.stress(args)
debug("Done Stressing")
def compact():
debug("Compacting...")
node1.nodetool('compact')
debug("Done Compacting.")
# put some data into the cluster
stress(['--num-keys=1000000'])
# now start compacting...
tcompact = Thread(target=compact)
tcompact.start()
wait(1)
# now the cluster is under a lot of load. Make some schema changes.
cursor.execute("USE Keyspace1")
wait(1)
cursor.execute("DROP COLUMNFAMILY Standard1")
wait(3)
cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
tcompact.join()
{code}
Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Tyler Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tyler Patterson updated CASSANDRA-4221:
---------------------------------------
Attachment: (was: system_log.zip)
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272451#comment-13272451 ]
Pavel Yaskevich commented on CASSANDRA-4221:
--------------------------------------------
The problem I see that that is we need to do a snapshot before start dropping or deleting any CF files so it's probably better to make that drop option 'deferred' until running compactions are stopped so we have a persistent view of the files we would have to operate upon.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271922#comment-13271922 ]
Pavel Yaskevich commented on CASSANDRA-4221:
--------------------------------------------
I don't know which one is better tho because if compaction fails for some reason which that scenario, wouldn't that mean that all SSTables that were left behind are staying until somebody manually deletes them (or restart would drop them)? We would have to add complexity to the schema merge just to handle that case as well as on the local side...
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Tyler Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tyler Patterson updated CASSANDRA-4221:
---------------------------------------
Attachment: (was: system.log)
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279622#comment-13279622 ]
Jonathan Ellis commented on CASSANDRA-4221:
-------------------------------------------
Should we just add a call to abort in-progress compactions at drop time (which will help cleanup happen faster) and call that "as close as we're going to get?"
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch, system.log
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271710#comment-13271710 ]
Jonathan Ellis commented on CASSANDRA-4221:
-------------------------------------------
That takes us back to the Bad Old Days pre-CASSANDRA-3116, though. We should be able to fix w/o resorting to A Big Lock.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4230.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281986#comment-13281986 ]
Jonathan Ellis commented on CASSANDRA-4221:
-------------------------------------------
bq. String is better suited because CompactionInfo.getColumnFamily() returns a String
Feel free to fix that. :)
bq. if compactions don't finish until then we just move on with delete
It throws IOException.
Remember that we check for ability to abort compaction every row; if we're compacting a wide row, it could easily take over 30s w/ throttling.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch, system.log
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279624#comment-13279624 ]
Pavel Yaskevich commented on CASSANDRA-4221:
--------------------------------------------
This is what I did in my patch :)
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch, system.log
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271905#comment-13271905 ]
Jonathan Ellis commented on CASSANDRA-4221:
-------------------------------------------
The idea from 3116 was:
- Drop will only delete sstables not actively being compacted
- post-compaction, we check if the CF was dropped, and if so we delete the sstables then
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271653#comment-13271653 ]
Jonathan Ellis commented on CASSANDRA-4221:
-------------------------------------------
Maybe, but I'm skeptical -- 4230 is complaining about a file existing when it shouldn't, while this one says a file doesn't exist that should :)
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4221:
--------------------------------------
Attachment: 4221-v3.txt
v3 attached. removes CompactionInfo fields that are redundant w/ the introduction of CFM, and removes the wait from the stop method (it doesn't help clean up the sstables involved any faster, so there is no point in slowing down the drop for it).
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: 4221-v3.txt, CASSANDRA-4221-logging.patch, CASSANDRA-4221-v2.patch, CASSANDRA-4221.patch, system.log
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272038#comment-13272038 ]
Jonathan Ellis commented on CASSANDRA-4221:
-------------------------------------------
bq. wouldn't that mean that all SSTables that were left behind are staying until somebody manually deletes them (or restart would drop them)?
We already clean up partially-written sstables after compaction failure, I don't see why we couldn't use similar logic here.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279618#comment-13279618 ]
Pavel Yaskevich commented on CASSANDRA-4221:
--------------------------------------------
Not really because it generates UUID from ksName + cfName to be able make it the same across all machines independent of their state.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch, system.log
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271927#comment-13271927 ]
Pavel Yaskevich commented on CASSANDRA-4221:
--------------------------------------------
The other way would be to 'mark a CF for delete' and return to the user right way (making CF invisible to users), sending the drop request to the others where they would apply the same thing (try to stop all compactions running, wait until they are done) and drop.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272457#comment-13272457 ]
Jonathan Ellis commented on CASSANDRA-4221:
-------------------------------------------
DataTracker already makes sstable changes atomic, though. At any time you can snapshot with that and get a consistent view.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281955#comment-13281955 ]
Jonathan Ellis commented on CASSANDRA-4221:
-------------------------------------------
- stopCompactionFor should take CFS parameters instead of String
- I don't see any reason to not wait indefinitely here; in fact, if we make sure to wait until compaction finishes, the odds are much better that when we tell the client "all done" he won't be able to send a "create" quickly enough to hit the bug
- Need to call stopCompactionFor on every replica, not just CompactionServer -- move this to DefsTable.dropColumnFamily?
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch, system.log
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-4221:
-----------------------------------------
Assignee: Pavel Yaskevich
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Tyler Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tyler Patterson updated CASSANDRA-4221:
---------------------------------------
Attachment: system.log
So after some experimentation, the problem is only happening when log_level is set at info, NOT debug. Gotta love these ones! I modified the logging patch to do logging.info() rather then logging.debug(), and the problem still happens, so at least you can see those debug messages. I hope this is enough to go on.
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221-logging.patch, CASSANDRA-4221.patch, system.log
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4221) Error while deleting a
columnfamily that is being compacted.
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-4221:
---------------------------------------
Attachment: CASSANDRA-4221.patch
> Error while deleting a columnfamily that is being compacted.
> ------------------------------------------------------------
>
> Key: CASSANDRA-4221
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4221
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: ccm, dtest, cassandra-1.1. The error does not happen in cassandra-1.0.
> Reporter: Tyler Patterson
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-4221.patch
>
>
> The following dtest command produces an error:
> {code}export CASSANDRA_VERSION=git:cassandra-1.1; nosetests --nocapture --nologcapture concurrent_schema_changes_test.py:TestConcurrentSchemaChanges.load_test{code}
> Here is the error:
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:239)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1580)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1770)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:839)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:851)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:142)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:148)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:121)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:264)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /tmp/dtest-6ECMgy/test/node1/data/Keyspace1/Standard1/Keyspace1-Standard1-hc-47-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:102)
> at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:87)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:985)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:56)
> ... 13 more
> {code}
> For reference, here is the dtest function that causes the failure. The error happens on the line near the bottom that drops the columnfamily:
> {code}
> def load_test(self):
> """
> apply schema changes while the cluster is under load.
> """
> debug("load_test()")
>
> cluster = self.cluster
> cluster.populate(1).start()
> node1 = cluster.nodelist()[0]
> wait(2)
> cursor = self.cql_connection(node1).cursor()
>
> def stress(args=[]):
> debug("Stressing")
> node1.stress(args)
> debug("Done Stressing")
>
> def compact():
> debug("Compacting...")
> node1.nodetool('compact')
> debug("Done Compacting.")
>
> # put some data into the cluster
> stress(['--num-keys=1000000'])
>
> # now start compacting...
> tcompact = Thread(target=compact)
> tcompact.start()
> wait(1)
>
> # now the cluster is under a lot of load. Make some schema changes.
> cursor.execute("USE Keyspace1")
> wait(1)
> cursor.execute("DROP COLUMNFAMILY Standard1")
>
> wait(3)
>
> cursor.execute("CREATE COLUMNFAMILY Standard1 (KEY text PRIMARY KEY)")
>
> tcompact.join()
>
> {code}
> Again, the error happens on cassandra-1.1, but not on cassandra-1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira