0.8.6 - Cannot Restart Node, Commit Log Exception

[Dan Hendry <da...@gmail.com>]

Last night one of my nodes died inexplicably, with no log entries anywhere
indicating the reason (cassandras log, /var/log/messages, etc). I tried to
restart it but the node will not restart as it fails with the errors below
when replaying the commit log. I should point out that the cluster is
currently composed of 5 nodes running 0.8.6, and one running 1.0.2. The node
the below exception occurred on is adjacent to the 1.0.2 node. (I have not
fully upgraded the cluster since there have been a number of issues with
1.0.2). I should also mention that the commit log for which playback
ostensibly failed on (CommitLog-1321354914026.log) should have been rotated
out *well* before the node crashed so I don't think it was corrupted by the
crash.

 

Known issue or new bug? Presumably a new one since I don't see any bugs
referenced in the changelogs between 0.8.6 and 1.0.3 which look related.

 

INFO [main] 2011-11-15 09:10:17,682 CommitLog.java (line 341) Finished
reading /var/lib/cassandra/commitlog/CommitLog-1321354340572.log

ERROR [MutationStage:5] 2011-11-15 09:10:19,753 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[MutationStage:5,5,main]

java.lang.RuntimeException: java.lang.NullPointerException

        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)

        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

        at java.util.concurrent.FutureTask.run(FutureTask.java:138)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

        at java.lang.Thread.run(Thread.java:662)

Caused by: java.lang.NullPointerException

        at
org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:309
)

        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)

        ... 6 more

INFO [main] 2011-11-15 09:10:19,763 CommitLog.java (line 341) Finished
reading /var/lib/cassandra/commitlog/CommitLog-1321354914026.log

ERROR [main] 2011-11-15 09:10:19,763 AbstractCassandraDaemon.java (line 358)
Exception encountered during startup.

java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.lang.NullPointerException

        at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:576)

        at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:333)

        at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)

        at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandra
Daemon.java:200)

        at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassan
draDaemon.java:341)

        at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:97)

Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.lang.NullPointerException

        at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)

        at java.util.concurrent.FutureTask.get(FutureTask.java:83)

        at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:572)

        ... 5 more

Caused by: java.lang.RuntimeException: java.lang.NullPointerException

        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)

        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

        at java.util.concurrent.FutureTask.run(FutureTask.java:138)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

        at java.lang.Thread.run(Thread.java:662)

Caused by: java.lang.NullPointerException

        at
org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:309
)

        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)

        ... 6 more