You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Lewis John Mcgibbney <le...@gmail.com> on 2012/04/04 15:57:13 UTC
Attempting to avoid fatal flush due to disk space
Hi,
When writing some test code (to test our Gora-Cassandra module) for Apache
Gora, I'm experiencing a major problem when trying to tearDown, flush and
basically clean everything up. The tests consist of running our test suite
against an embedded Cassandra instance using the Gora API to do all sorts
of tasks which may highkight where there are bugs in the code.
Now regardless of whether the tests are failing or not, I get the log and
subsequent stack trace as below which quite clearly indicates that there is
insufficient disk to flush. However as I don't have the luxury of making
changes @ runtime I'm slightly stumped as to how to solve it as all I'm
working with is the Cassandra server which is initiated in setUpClass.
My yaml is ripped directly from the Cassandra 1.0.2 distribution so there
is nothing out of place in there. However maybe there are some settings
which I have not configured?
Please indicate if I need to provide some more information to paint a
clearer picture of the situation.
When calling tearDownClass I have
if (cassandraThread != null) {
cassandraDaemon.stop();
cassandraDaemon.destroy();
cassandraThread.interrupt();
cassandraThread = null;
}
cleanupDirectoriesFailover(); // this recursively deletes the
directories Cassandra works with whilst the server is running.
}
Thank you very much in advance for any help coming this way.
Lewis
12/04/04 13:24:11 INFO gora.GoraTestDriver: tearing down test
12/04/04 13:24:11 INFO migration.Migration: Applying migration
15541bf0-7e51-11e1-0000-242d50cf1fff Drop keyspace: Employee
12/04/04 13:24:11 INFO db.ColumnFamilyStore: Enqueuing flush of
Memtable-Migrations@9190262(6586/82325 serialized/live bytes, 1 ops)
12/04/04 13:24:11 INFO db.Memtable: Writing
Memtable-Migrations@9190262(6586/82325
serialized/live bytes, 1 ops)
12/04/04 13:24:11 INFO db.ColumnFamilyStore: Enqueuing flush of
Memtable-Schema@9695615(2796/34950 serialized/live bytes, 2 ops)
12/04/04 13:24:11 INFO db.Memtable: Completed flushing
target/test/var/lib/cassandra/data/system/Migrations-h-2764-Data.db (6650
bytes)
12/04/04 13:24:11 INFO db.Memtable: Writing Memtable-Schema@9695615(2796/34950
serialized/live bytes, 2 ops)
12/04/04 13:24:11 INFO db.Memtable: Completed flushing
target/test/var/lib/cassandra/data/system/Schema-h-2764-Data.db (2946 bytes)
12/04/04 13:24:12 INFO compaction.CompactionTask: Compacted to
[target/test/var/lib/cassandra/data/system/Migrations-h-2765-Data.db,].
12,979,714 to 12,979,522 (~99% of original) bytes for 1 keys at
12.341213MB/s. Time: 1,003ms.
12/04/04 13:24:12 INFO compaction.CompactionTask: Compacting
[SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2764-Data.db'),
SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2760-Data.db'),
SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2761-Data.db'),
SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2763-Data.db')]
12/04/04 13:24:12 INFO store.DataStoreTestBase: tearing down class
12/04/04 13:24:12 INFO service.AbstractCassandraDaemon: Cassandra shutting
down...
12/04/04 13:24:12 INFO thrift.CassandraDaemon: Stop listening to thrift
clients
12/04/04 13:24:12 INFO compaction.CompactionTask: Compacted to
[target/test/var/lib/cassandra/data/system/Schema-h-2765-Data.db,].
5,735,956 to 5,735,629 (~99% of original) bytes for 1,681 keys at
9.011404MB/s. Time: 607ms.
Tests run: 51, Failures: 1, Errors: 50, Skipped: 0, Time elapsed: 913.842
sec <<< FAILURE!
12/04/04 13:25:12 INFO net.MessagingService: Shutting down MessageService...
12/04/04 13:25:12 INFO net.MessagingService: MessagingService shutting down
server thread.
12/04/04 13:25:12 INFO net.MessagingService: Waiting for in-progress
requests to complete
12/04/04 13:25:12 INFO db.ColumnFamilyStore: Enqueuing flush of
Memtable-Versions@6081747(83/103 serialized/live bytes, 3 ops)
12/04/04 13:25:12 INFO db.Memtable: Writing Memtable-Versions@6081747(83/103
serialized/live bytes, 3 ops)
12/04/04 13:25:12 ERROR service.AbstractCassandraDaemon: Fatal exception in
thread Thread[FlushWriter:2,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk
space to flush 133 bytes
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Insufficient disk space to flush 133
bytes
at
org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:621)
at
org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:1866)
at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:248)
at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)
at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:289)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
12/04/04 13:25:12 ERROR service.AbstractCassandraDaemon: Fatal exception in
thread Thread[FlushWriter:2,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk
space to flush 133 bytes
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Insufficient disk space to flush 133
bytes
at
org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:621)
at
org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:1866)
at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:248)
at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)
at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:289)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
--
*Lewis*
Re: Attempting to avoid fatal flush due to disk space
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Aaron,
On Wed, Apr 4, 2012 at 7:57 PM, aaron morton <aa...@thelastpickle.com>wrote:
> Is cleanupDirectoriesFailover able to delete the files ?
>
Yes, around the time that the stack traces appear (after final
teadDownClass()) you'll see my log.INFO statement and subsequent service
log indicating that CassandraDaemon.stop and CassandraDaemon.destroy have
both been called. This is backed up by my manual checks which confirm that
the test directory and files within are sucessfully deleted. So I know that
the cleanupDirectoriesFailover code is executing as planned and that things
are working as they should up until my description as above. I've pasted
the code below with some additional comments below that.
12/04/04 21:59:21 INFO store.DataStoreTestBase: tearing down class
12/04/04 21:59:21 INFO cassandra.GoraCassandraTestDriver: Shutting down
Embedded Cassandra server...
12/04/04 21:59:21 INFO service.AbstractCassandraDaemon: Cassandra shutting
down...
12/04/04 21:59:21 INFO thrift.CassandraDaemon: Stop listening to thrift
clients
Tests run: 28, Failures: 0, Errors: 28, Skipped: 0, Time elapsed: 69.043
sec <<< FAILURE!
12/04/04 22:00:21 INFO net.MessagingService: Shutting down MessageService...
12/04/04 22:00:21 INFO net.MessagingService: MessagingService shutting down
server thread.
12/04/04 22:00:21 INFO net.MessagingService: Waiting for in-progress
requests to complete
12/04/04 22:00:21 INFO db.ColumnFamilyStore: Enqueuing flush of
Memtable-Versions@6196605(83/103 serialized/live bytes, 3 ops)
12/04/04 22:00:21 INFO db.Memtable: Writing Memtable-Versions@6196605(83/103
serialized/live bytes, 3 ops)
12/04/04 22:00:21 ERROR service.AbstractCassandraDaemon: Fatal exception in
thread Thread[FlushWriter:2,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk
space to flush 133 bytes
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.RuntimeException: Insufficient disk space to flush 133
bytes
at
org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:623)
at
org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:1850)
at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:250)
at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)
at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:291)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
12/04/04 22:00:21 ERROR service.AbstractCassandraDaemon: Fatal exception in
thread Thread[FlushWriter:2,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk
space to flush 133 bytes
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.RuntimeException: Insufficient disk space to flush 133
bytes
at
org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:623)
at
org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:1850)
at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:250)
at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)
at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:291)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
>
> When you get the error is the disk actually full ?
>
I checked the memory on disk it is certainly not full. I have well over
20GB still available!
>
> Can you narrow this down to "i cannot delete the sstables"? (and what
> platform are you on).
>
Well for each failed test I'm currently getting
12/04/04 21:59:21 ERROR thrift.Cassandra$Processor: Internal error
processing system_add_keyspace
java.lang.IncompatibleClassChangeError: Class
org.apache.avro.generic.GenericData$Array does not implement the requested
interface java.util.List
at org.apache.cassandra.config.KSMetaData.toAvro(KSMetaData.java:136)
at
org.apache.cassandra.db.migration.Migration.makeDefinitionMutation(Migration.java:223)
at
org.apache.cassandra.db.migration.AddKeyspace.<init>(AddKeyspace.java:49)
at
org.apache.cassandra.thrift.CassandraServer.system_add_keyspace(CassandraServer.java:986)
at
org.apache.cassandra.thrift.Cassandra$Processor$system_add_keyspace.process(Cassandra.java:3912)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
This is not merely a failed assertion so I'm currently investigating this
as well.
I'm using single core Ubuntu release 11.10, Kernel Linux 3.0.0-17-generic
GNOME 3.2.1, AMD Athlon 64 processor 3800+
Looking back through my full log they only WARN I have is early on
12/04/04 15:30:09 WARN service.StorageService: Generated random token
122116503163726529681704205068506497955. Random tokens will result in an
unbalanced ring; see http://wiki.apache.org/cassandra/Operations
Although I don't think this leads to the problem I'm encountering. I'm
looking into the 2nd snippet of code I pasted above and hopefully this will
produce some results.
Thanks for your comments.
Lewis
Re: Attempting to avoid fatal flush due to disk space
Posted by aaron morton <aa...@thelastpickle.com>.
Is cleanupDirectoriesFailover able to delete the files ?
When you get the error is the disk actually full ?
Can you narrow this down to "i cannot delete the sstables"? (and what platform are you on).
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 5/04/2012, at 1:57 AM, Lewis John Mcgibbney wrote:
> Hi,
>
> When writing some test code (to test our Gora-Cassandra module) for Apache Gora, I'm experiencing a major problem when trying to tearDown, flush and basically clean everything up. The tests consist of running our test suite against an embedded Cassandra instance using the Gora API to do all sorts of tasks which may highkight where there are bugs in the code.
>
> Now regardless of whether the tests are failing or not, I get the log and subsequent stack trace as below which quite clearly indicates that there is insufficient disk to flush. However as I don't have the luxury of making changes @ runtime I'm slightly stumped as to how to solve it as all I'm working with is the Cassandra server which is initiated in setUpClass.
>
> My yaml is ripped directly from the Cassandra 1.0.2 distribution so there is nothing out of place in there. However maybe there are some settings which I have not configured?
>
> Please indicate if I need to provide some more information to paint a clearer picture of the situation.
>
> When calling tearDownClass I have
>
> if (cassandraThread != null) {
> cassandraDaemon.stop();
> cassandraDaemon.destroy();
> cassandraThread.interrupt();
> cassandraThread = null;
> }
> cleanupDirectoriesFailover(); // this recursively deletes the directories Cassandra works with whilst the server is running.
> }
>
> Thank you very much in advance for any help coming this way.
>
> Lewis
>
> 12/04/04 13:24:11 INFO gora.GoraTestDriver: tearing down test
> 12/04/04 13:24:11 INFO migration.Migration: Applying migration 15541bf0-7e51-11e1-0000-242d50cf1fff Drop keyspace: Employee
> 12/04/04 13:24:11 INFO db.ColumnFamilyStore: Enqueuing flush of Memtable-Migrations@9190262(6586/82325 serialized/live bytes, 1 ops)
> 12/04/04 13:24:11 INFO db.Memtable: Writing Memtable-Migrations@9190262(6586/82325 serialized/live bytes, 1 ops)
> 12/04/04 13:24:11 INFO db.ColumnFamilyStore: Enqueuing flush of Memtable-Schema@9695615(2796/34950 serialized/live bytes, 2 ops)
> 12/04/04 13:24:11 INFO db.Memtable: Completed flushing target/test/var/lib/cassandra/data/system/Migrations-h-2764-Data.db (6650 bytes)
> 12/04/04 13:24:11 INFO db.Memtable: Writing Memtable-Schema@9695615(2796/34950 serialized/live bytes, 2 ops)
> 12/04/04 13:24:11 INFO db.Memtable: Completed flushing target/test/var/lib/cassandra/data/system/Schema-h-2764-Data.db (2946 bytes)
> 12/04/04 13:24:12 INFO compaction.CompactionTask: Compacted to [target/test/var/lib/cassandra/data/system/Migrations-h-2765-Data.db,]. 12,979,714 to 12,979,522 (~99% of original) bytes for 1 keys at 12.341213MB/s. Time: 1,003ms.
> 12/04/04 13:24:12 INFO compaction.CompactionTask: Compacting [SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2764-Data.db'), SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2760-Data.db'), SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2761-Data.db'), SSTableReader(path='target/test/var/lib/cassandra/data/system/Schema-h-2763-Data.db')]
> 12/04/04 13:24:12 INFO store.DataStoreTestBase: tearing down class
> 12/04/04 13:24:12 INFO service.AbstractCassandraDaemon: Cassandra shutting down...
> 12/04/04 13:24:12 INFO thrift.CassandraDaemon: Stop listening to thrift clients
> 12/04/04 13:24:12 INFO compaction.CompactionTask: Compacted to [target/test/var/lib/cassandra/data/system/Schema-h-2765-Data.db,]. 5,735,956 to 5,735,629 (~99% of original) bytes for 1,681 keys at 9.011404MB/s. Time: 607ms.
> Tests run: 51, Failures: 1, Errors: 50, Skipped: 0, Time elapsed: 913.842 sec <<< FAILURE!
> 12/04/04 13:25:12 INFO net.MessagingService: Shutting down MessageService...
> 12/04/04 13:25:12 INFO net.MessagingService: MessagingService shutting down server thread.
> 12/04/04 13:25:12 INFO net.MessagingService: Waiting for in-progress requests to complete
> 12/04/04 13:25:12 INFO db.ColumnFamilyStore: Enqueuing flush of Memtable-Versions@6081747(83/103 serialized/live bytes, 3 ops)
> 12/04/04 13:25:12 INFO db.Memtable: Writing Memtable-Versions@6081747(83/103 serialized/live bytes, 3 ops)
> 12/04/04 13:25:12 ERROR service.AbstractCassandraDaemon: Fatal exception in thread Thread[FlushWriter:2,5,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes
> at org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:621)
> at org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:1866)
> at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:248)
> at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)
> at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:289)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> 12/04/04 13:25:12 ERROR service.AbstractCassandraDaemon: Fatal exception in thread Thread[FlushWriter:2,5,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException: Insufficient disk space to flush 133 bytes
> at org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:621)
> at org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:1866)
> at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:248)
> at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47)
> at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:289)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
>
> --
> Lewis
>