You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Philip Thompson (JIRA)" <ji...@apache.org> on 2015/09/14 16:02:45 UTC
[jira] [Updated] (CASSANDRA-10315) Cassandra nodes shutting down on
COMMIT-LOG-ALLOCATOR error
[ https://issues.apache.org/jira/browse/CASSANDRA-10315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Philip Thompson updated CASSANDRA-10315:
----------------------------------------
Description:
After migrating from 2.0.9 to 2.0.15 all nodes on multiple clusters Cassandra nodes shutting themselves down every 24-48 hours. The error reported is:
{code}
ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-11 17:14:48,034 CommitLog.java (line 420) Failed to allocate new commit log segments. Commit disk failure policy is stop; terminating thread
java.lang.AssertionError: attempted to delete non-existing file CommitLog-3-1441961724221.log
at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:113)
at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:161)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:228)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:99)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.lang.Thread.run(Unknown Source)
{code}
We’ve seen this error before, but now with DEBUG / TRACE on, we see that this file has 18 entries starting at 5:14pm that look something like this:
DEBUG [COMMIT-LOG-WRITER] 2015-09-11 17:14:45,372 CommitLog.java (line 245) Not safe to delete commit log segment CommitLogSegment(/opt/osi/monarch/chronus/resources/cassandra/commitlog/CommitLog-3-1441961724221.log); dirty is soe (a77b7765-1e3b-30eb-9f46-2cac8dfe1ac7), min_max_avg_hourly (f86973a2-f5e6-36b6-9d7f-7fc4e109fb6e),
was:
After migrating from 2.0.9 to 2.0.15 all nodes on multiple clusters Cassandra nodes shutting themselves down every 24-48 hours. The error reported is:
ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-11 17:14:48,034 CommitLog.java (line 420) Failed to allocate new commit log segments. Commit disk failure policy is stop; terminating thread
java.lang.AssertionError: attempted to delete non-existing file CommitLog-3-1441961724221.log
at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:113)
at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:161)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:228)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:99)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.lang.Thread.run(Unknown Source)
We’ve seen this error before, but now with DEBUG / TRACE on, we see that this file has 18 entries starting at 5:14pm that look something like this:
DEBUG [COMMIT-LOG-WRITER] 2015-09-11 17:14:45,372 CommitLog.java (line 245) Not safe to delete commit log segment CommitLogSegment(/opt/osi/monarch/chronus/resources/cassandra/commitlog/CommitLog-3-1441961724221.log); dirty is soe (a77b7765-1e3b-30eb-9f46-2cac8dfe1ac7), min_max_avg_hourly (f86973a2-f5e6-36b6-9d7f-7fc4e109fb6e),
> Cassandra nodes shutting down on COMMIT-LOG-ALLOCATOR error
> -----------------------------------------------------------
>
> Key: CASSANDRA-10315
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10315
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: 16 GB, Cassandra 2.0.15,
> Reporter: David Loegering
>
> After migrating from 2.0.9 to 2.0.15 all nodes on multiple clusters Cassandra nodes shutting themselves down every 24-48 hours. The error reported is:
> {code}
> ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-11 17:14:48,034 CommitLog.java (line 420) Failed to allocate new commit log segments. Commit disk failure policy is stop; terminating thread
> java.lang.AssertionError: attempted to delete non-existing file CommitLog-3-1441961724221.log
> at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:113)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:161)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:228)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:99)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.lang.Thread.run(Unknown Source)
> {code}
> We’ve seen this error before, but now with DEBUG / TRACE on, we see that this file has 18 entries starting at 5:14pm that look something like this:
> DEBUG [COMMIT-LOG-WRITER] 2015-09-11 17:14:45,372 CommitLog.java (line 245) Not safe to delete commit log segment CommitLogSegment(/opt/osi/monarch/chronus/resources/cassandra/commitlog/CommitLog-3-1441961724221.log); dirty is soe (a77b7765-1e3b-30eb-9f46-2cac8dfe1ac7), min_max_avg_hourly (f86973a2-f5e6-36b6-9d7f-7fc4e109fb6e),
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)