You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Tyler Schade (Jira)" <ji...@apache.org> on 2025/05/20 16:16:00 UTC
[jira] [Updated] (CASSANDRA-20665) Cassandra 5.0.4 crashing repeatedly on restart
[ https://issues.apache.org/jira/browse/CASSANDRA-20665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tyler Schade updated CASSANDRA-20665:
-------------------------------------
Summary: Cassandra 5.0.4 crashing repeatedly on restart (was: Cassandra 5.0.4 crashing repeatedly on restart when last)
> Cassandra 5.0.4 crashing repeatedly on restart
> ----------------------------------------------
>
> Key: CASSANDRA-20665
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20665
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Tyler Schade
> Priority: Normal
>
> Noticing some strange behavior today with a Cassandra 5.0.4 Docker Container that recently likely encountered a forced shutdown. This is a single-node "cluster".
> {noformat}
> INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping playback of empty log: CommitLog-7-1747752126176.logINFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping playback of empty log: CommitLog-7-1747752190016.logINFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping playback of empty log: CommitLog-7-1747752253591.logINFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping playback of empty log: CommitLog-7-1747752317293.logINFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping playback of empty log: CommitLog-7-1747752380740.logINFO [main] 2025-05-20 14:49:37,066 CommitLogReader.java:103 - Skipping playback of empty log: CommitLog-7-1747752445680.logINFO [main] 2025-05-20 14:49:37,066 CommitLogReader.java:103 - Skipping playback of empty log: CommitLog-7-1747752511283.logERROR [main] 2025-05-20 14:49:37,067 JVMStabilityInspector.java:201 - Exiting due to error while processing commit log during initialization.org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Could not read commit log descriptor in file /opt/cassandra/data/commitlog/CommitLog-7-1747377011833.log at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:196) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147) at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:205) at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:229) at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:208) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:345) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:727) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:865){noformat}
> I investigated a little with some help from [~jjirsa] and found that the segment referenced was empty:
> {noformat}
> root@09f46103e3e2:/var/lib/cassandra/commitlog# hexdump CommitLog-7-1747377011833.log
> 0000000 0000 0000 0000 0000 0000 0000 0000 0000
> *
> 2000000{noformat}
> The subsequent segment was _also_ empty:
> {noformat}
> root@09f46103e3e2:/var/lib/cassandra/commitlog# md5sum $(ls | head)
> 58f06dd588d8ffb3beb46ada6309436b CommitLog-7-1747377011833.log
> 58f06dd588d8ffb3beb46ada6309436b CommitLog-7-1747377011834.log
> 1f9bd5ab1f623fd921282713bf3e34cc CommitLog-7-1747377250204.log
> 08ed83631a94bdb902ddeb600e8f562a CommitLog-7-1747377250205.log
> 1093718121a69c2843dd6ed204dc6563 CommitLog-7-1747377266512.log
> 2a48581389c3bb68a58be3d0d14eb6f5 CommitLog-7-1747377266513.log
> 24ad307b55b8d5d1d33ddcc4e21395e8 CommitLog-7-1747377272735.log
> 1e4a7628f5f141c587d2cc94eda3c22b CommitLog-7-1747377272736.log
> 06f498bc5d35e2691761e39e0a6d5476 CommitLog-7-1747377276330.log
> 10f9e64659b333ae3b8b3e9424a4468b CommitLog-7-1747377276331.log{noformat}
> However, there are 8 subsequent segments with data:
> {noformat}
> root@09f46103e3e2:/var/lib/cassandra/commitlog# ls -l | head
> total 143136
> -rw-r--r-- 1 999 999 33554432 May 16 06:30 CommitLog-7-1747377011833.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:30 CommitLog-7-1747377011834.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377250204.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377250205.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377266512.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377266513.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377272735.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377272736.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377276330.log{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org