You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "David Loegering (JIRA)" <ji...@apache.org> on 2015/09/14 16:49:46 UTC
[jira] [Commented] (CASSANDRA-10315) Cassandra nodes shutting down
on COMMIT-LOG-ALLOCATOR error
[ https://issues.apache.org/jira/browse/CASSANDRA-10315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743623#comment-14743623 ]
David Loegering commented on CASSANDRA-10315:
---------------------------------------------
This is very easy to reproduce and is happening on multiple clusters on all nodes at random times. Just starting thrift and gossip without restarting Cassandra seems to resolve the issue. I have very complete trace data.
> Cassandra nodes shutting down on COMMIT-LOG-ALLOCATOR error
> -----------------------------------------------------------
>
> Key: CASSANDRA-10315
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10315
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: 16 GB, Cassandra 2.0.15,
> Reporter: David Loegering
>
> After migrating from 2.0.9 to 2.0.15 all nodes on multiple clusters Cassandra nodes shutting themselves down every 24-48 hours. The error reported is:
> {code}
> ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-11 17:14:48,034 CommitLog.java (line 420) Failed to allocate new commit log segments. Commit disk failure policy is stop; terminating thread
> java.lang.AssertionError: attempted to delete non-existing file CommitLog-3-1441961724221.log
> at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:113)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:161)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:228)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:99)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.lang.Thread.run(Unknown Source)
> {code}
> We’ve seen this error before, but now with DEBUG / TRACE on, we see that this file has 18 entries starting at 5:14pm that look something like this:
> {code}
> DEBUG [COMMIT-LOG-WRITER] 2015-09-11 17:14:45,372 CommitLog.java (line 245) Not safe to delete commit log segment CommitLogSegment(/opt/osi/monarch/chronus/resources/cassandra/commitlog/CommitLog-3-1441961724221.log); dirty is soe (a77b7765-1e3b-30eb-9f46-2cac8dfe1ac7), min_max_avg_hourly (f86973a2-f5e6-36b6-9d7f-7fc4e109fb6e),
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)