You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Arya Goudarzi <ag...@gaiaonline.com> on 2010/08/09 21:30:54 UTC

COMMIT-LOG_WRITER Assertion Error

Just throwing this out there as it could be a concern. I had a cluster of 3 nodes running. Over the weekend I updated to trunc (Aug 9th @ 2pm). Today, I came to run my daily tests and my client kept giving me TSocket timeouts. Checking the error log of Cassandra servers, all 3 nodes had this and they all became unresponsive! Not sure how to reproduce this but a restart of all 3 nodes fixed the issue:

ERROR [COMMIT-LOG-WRITER] 2010-08-09 11:30:27,722 CassandraDaemon.java (line 82) Uncaught exception in thread Thread[COMMIT-LOG-WRITER,5,main]
java.lang.AssertionError
        at org.apache.cassandra.db.commitlog.CommitLogHeader$CommitLogHeaderSerializer.serialize(CommitLogHeader.java:157)
        at org.apache.cassandra.db.commitlog.CommitLogHeader.writeCommitLogHeader(CommitLogHeader.java:124)
        at org.apache.cassandra.db.commitlog.CommitLogSegment.writeHeader(CommitLogSegment.java:70)
        at org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:103)
        at org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:521)
        at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        at java.lang.Thread.run(Thread.java:636)


-Arya

Re: COMMIT-LOG_WRITER Assertion Error

Posted by Arya Goudarzi <ag...@gaiaonline.com>.
Sure. https://issues.apache.org/jira/browse/CASSANDRA-1376

----- Original Message -----
From: "Jonathan Ellis" <jb...@gmail.com>
To: user@cassandra.apache.org
Sent: Tuesday, August 10, 2010 7:05:31 AM
Subject: Re: COMMIT-LOG_WRITER Assertion Error

Can you create a ticket for this?

On Mon, Aug 9, 2010 at 8:42 PM, Arya Goudarzi <ag...@gaiaonline.com> wrote:
> I've never run 0.6. I have been running of trunc with automatic svn update and build everyday at 2pm. One of my nodes got this error which lead to the same last error prior to build and restart today. Hope this helps better:
>
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
>        at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:549)
>        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:339)
>        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:174)
>        at org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:120)
>        at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:90)
>        at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:224)
> Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>        at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:545)
>        ... 5 more
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
>        at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegments(CommitLog.java:408)
>        at org.apache.cassandra.db.ColumnFamilyStore$2.runMayThrow(ColumnFamilyStore.java:445)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        ... 6 more
> Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>        at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegments(CommitLog.java:400)
>        ... 8 more
> Caused by: java.lang.AssertionError
>        at org.apache.cassandra.db.commitlog.CommitLogHeader$CommitLogHeaderSerializer.serialize(CommitLogHeader.java:157)
>        at org.apache.cassandra.db.commitlog.CommitLogHeader.writeCommitLogHeader(CommitLogHeader.java:124)
>        at org.apache.cassandra.db.commitlog.CommitLogSegment.writeHeader(CommitLogSegment.java:70)
>        at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegmentsInternal(CommitLog.java:450)
>        at org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog.java:75)
>        at org.apache.cassandra.db.commitlog.CommitLog$6.call(CommitLog.java:394)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>        at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        ... 1 more
>
> ----- Original Message -----
> From: "Jonathan Ellis" <jb...@gmail.com>
> To: user@cassandra.apache.org
> Sent: Monday, August 9, 2010 5:18:35 PM
> Subject: Re: COMMIT-LOG_WRITER Assertion Error
>
> Sounds like you upgraded to trunk from 0.6 without draining your
> commitlog first?
>
> On Mon, Aug 9, 2010 at 3:30 PM, Arya Goudarzi <ag...@gaiaonline.com> wrote:
>> Just throwing this out there as it could be a concern. I had a cluster of 3 nodes running. Over the weekend I updated to trunc (Aug 9th @ 2pm). Today, I came to run my daily tests and my client kept giving me TSocket timeouts. Checking the error log of Cassandra servers, all 3 nodes had this and they all became unresponsive! Not sure how to reproduce this but a restart of all 3 nodes fixed the issue:
>>
>> ERROR [COMMIT-LOG-WRITER] 2010-08-09 11:30:27,722 CassandraDaemon.java (line 82) Uncaught exception in thread Thread[COMMIT-LOG-WRITER,5,main]
>> java.lang.AssertionError
>>        at org.apache.cassandra.db.commitlog.CommitLogHeader$CommitLogHeaderSerializer.serialize(CommitLogHeader.java:157)
>>        at org.apache.cassandra.db.commitlog.CommitLogHeader.writeCommitLogHeader(CommitLogHeader.java:124)
>>        at org.apache.cassandra.db.commitlog.CommitLogSegment.writeHeader(CommitLogSegment.java:70)
>>        at org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:103)
>>        at org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:521)
>>        at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52)
>>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>        at java.lang.Thread.run(Thread.java:636)
>>
>>
>> -Arya
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: COMMIT-LOG_WRITER Assertion Error

Posted by Jonathan Ellis <jb...@gmail.com>.
Can you create a ticket for this?

On Mon, Aug 9, 2010 at 8:42 PM, Arya Goudarzi <ag...@gaiaonline.com> wrote:
> I've never run 0.6. I have been running of trunc with automatic svn update and build everyday at 2pm. One of my nodes got this error which lead to the same last error prior to build and restart today. Hope this helps better:
>
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
>        at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:549)
>        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:339)
>        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:174)
>        at org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:120)
>        at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:90)
>        at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:224)
> Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>        at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:545)
>        ... 5 more
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
>        at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegments(CommitLog.java:408)
>        at org.apache.cassandra.db.ColumnFamilyStore$2.runMayThrow(ColumnFamilyStore.java:445)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        ... 6 more
> Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>        at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegments(CommitLog.java:400)
>        ... 8 more
> Caused by: java.lang.AssertionError
>        at org.apache.cassandra.db.commitlog.CommitLogHeader$CommitLogHeaderSerializer.serialize(CommitLogHeader.java:157)
>        at org.apache.cassandra.db.commitlog.CommitLogHeader.writeCommitLogHeader(CommitLogHeader.java:124)
>        at org.apache.cassandra.db.commitlog.CommitLogSegment.writeHeader(CommitLogSegment.java:70)
>        at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegmentsInternal(CommitLog.java:450)
>        at org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog.java:75)
>        at org.apache.cassandra.db.commitlog.CommitLog$6.call(CommitLog.java:394)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>        at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        ... 1 more
>
> ----- Original Message -----
> From: "Jonathan Ellis" <jb...@gmail.com>
> To: user@cassandra.apache.org
> Sent: Monday, August 9, 2010 5:18:35 PM
> Subject: Re: COMMIT-LOG_WRITER Assertion Error
>
> Sounds like you upgraded to trunk from 0.6 without draining your
> commitlog first?
>
> On Mon, Aug 9, 2010 at 3:30 PM, Arya Goudarzi <ag...@gaiaonline.com> wrote:
>> Just throwing this out there as it could be a concern. I had a cluster of 3 nodes running. Over the weekend I updated to trunc (Aug 9th @ 2pm). Today, I came to run my daily tests and my client kept giving me TSocket timeouts. Checking the error log of Cassandra servers, all 3 nodes had this and they all became unresponsive! Not sure how to reproduce this but a restart of all 3 nodes fixed the issue:
>>
>> ERROR [COMMIT-LOG-WRITER] 2010-08-09 11:30:27,722 CassandraDaemon.java (line 82) Uncaught exception in thread Thread[COMMIT-LOG-WRITER,5,main]
>> java.lang.AssertionError
>>        at org.apache.cassandra.db.commitlog.CommitLogHeader$CommitLogHeaderSerializer.serialize(CommitLogHeader.java:157)
>>        at org.apache.cassandra.db.commitlog.CommitLogHeader.writeCommitLogHeader(CommitLogHeader.java:124)
>>        at org.apache.cassandra.db.commitlog.CommitLogSegment.writeHeader(CommitLogSegment.java:70)
>>        at org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:103)
>>        at org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:521)
>>        at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52)
>>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>        at java.lang.Thread.run(Thread.java:636)
>>
>>
>> -Arya
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: COMMIT-LOG_WRITER Assertion Error

Posted by Arya Goudarzi <ag...@gaiaonline.com>.
I've never run 0.6. I have been running of trunc with automatic svn update and build everyday at 2pm. One of my nodes got this error which lead to the same last error prior to build and restart today. Hope this helps better:

java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
        at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:549)
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:339)
        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:174)
        at org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:120)
        at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:90)
        at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:224)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
        at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:545)
        ... 5 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError
        at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegments(CommitLog.java:408)
        at org.apache.cassandra.db.ColumnFamilyStore$2.runMayThrow(ColumnFamilyStore.java:445)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        ... 6 more
Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
        at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegments(CommitLog.java:400)
        ... 8 more
Caused by: java.lang.AssertionError
        at org.apache.cassandra.db.commitlog.CommitLogHeader$CommitLogHeaderSerializer.serialize(CommitLogHeader.java:157)
        at org.apache.cassandra.db.commitlog.CommitLogHeader.writeCommitLogHeader(CommitLogHeader.java:124)
        at org.apache.cassandra.db.commitlog.CommitLogSegment.writeHeader(CommitLogSegment.java:70)
        at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegmentsInternal(CommitLog.java:450)
        at org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog.java:75)
        at org.apache.cassandra.db.commitlog.CommitLog$6.call(CommitLog.java:394)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        ... 1 more 

----- Original Message -----
From: "Jonathan Ellis" <jb...@gmail.com>
To: user@cassandra.apache.org
Sent: Monday, August 9, 2010 5:18:35 PM
Subject: Re: COMMIT-LOG_WRITER Assertion Error

Sounds like you upgraded to trunk from 0.6 without draining your
commitlog first?

On Mon, Aug 9, 2010 at 3:30 PM, Arya Goudarzi <ag...@gaiaonline.com> wrote:
> Just throwing this out there as it could be a concern. I had a cluster of 3 nodes running. Over the weekend I updated to trunc (Aug 9th @ 2pm). Today, I came to run my daily tests and my client kept giving me TSocket timeouts. Checking the error log of Cassandra servers, all 3 nodes had this and they all became unresponsive! Not sure how to reproduce this but a restart of all 3 nodes fixed the issue:
>
> ERROR [COMMIT-LOG-WRITER] 2010-08-09 11:30:27,722 CassandraDaemon.java (line 82) Uncaught exception in thread Thread[COMMIT-LOG-WRITER,5,main]
> java.lang.AssertionError
>        at org.apache.cassandra.db.commitlog.CommitLogHeader$CommitLogHeaderSerializer.serialize(CommitLogHeader.java:157)
>        at org.apache.cassandra.db.commitlog.CommitLogHeader.writeCommitLogHeader(CommitLogHeader.java:124)
>        at org.apache.cassandra.db.commitlog.CommitLogSegment.writeHeader(CommitLogSegment.java:70)
>        at org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:103)
>        at org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:521)
>        at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        at java.lang.Thread.run(Thread.java:636)
>
>
> -Arya
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: COMMIT-LOG_WRITER Assertion Error

Posted by Jonathan Ellis <jb...@gmail.com>.
Sounds like you upgraded to trunk from 0.6 without draining your
commitlog first?

On Mon, Aug 9, 2010 at 3:30 PM, Arya Goudarzi <ag...@gaiaonline.com> wrote:
> Just throwing this out there as it could be a concern. I had a cluster of 3 nodes running. Over the weekend I updated to trunc (Aug 9th @ 2pm). Today, I came to run my daily tests and my client kept giving me TSocket timeouts. Checking the error log of Cassandra servers, all 3 nodes had this and they all became unresponsive! Not sure how to reproduce this but a restart of all 3 nodes fixed the issue:
>
> ERROR [COMMIT-LOG-WRITER] 2010-08-09 11:30:27,722 CassandraDaemon.java (line 82) Uncaught exception in thread Thread[COMMIT-LOG-WRITER,5,main]
> java.lang.AssertionError
>        at org.apache.cassandra.db.commitlog.CommitLogHeader$CommitLogHeaderSerializer.serialize(CommitLogHeader.java:157)
>        at org.apache.cassandra.db.commitlog.CommitLogHeader.writeCommitLogHeader(CommitLogHeader.java:124)
>        at org.apache.cassandra.db.commitlog.CommitLogSegment.writeHeader(CommitLogSegment.java:70)
>        at org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:103)
>        at org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:521)
>        at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52)
>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        at java.lang.Thread.run(Thread.java:636)
>
>
> -Arya
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com