You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Joe Obernberger <jo...@gmail.com> on 2023/02/03 20:14:35 UTC
Startup fails - 4.1.0
Hi all - cluster had a power outage and one of the nodes in a 14 nodes
cluster isn't starting with:
DEBUG [MemtableFlushWriter:1] 2023-02-03 13:52:45,468
ColumnFamilyStore.java:1329 - Flushed to
[BigTableReader(path='/data/2/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8407-big-Data.db'),
BigTableReader(path='/data/3/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8408-big-Data.db'),
BigTableReader(path='/data/4/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8409-big-Data.db'),
BigTableReader(path='/data/5/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8410-big-Data.db'),
BigTableReader(path='/data/6/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8411-big-Data.db'),
BigTableReader(path='/data/8/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8412-big-Data.db'),
BigTableReader(path='/data/9/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8413-big-Data.db')]
(7 sstables, 92.858MiB), biggest 15.420MiB, smallest 10.307MiB
INFO [main] 2023-02-03 13:52:45,621 CommitLogReader.java:257 - Finished
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126163.log
DEBUG [main] 2023-02-03 13:52:45,622 CommitLogReader.java:266 - Reading
/var/lib/cassandra/commitlog/CommitLog-7-1674161126164.log (CL version
7, messaging version 12, compression null)
INFO [main] 2023-02-03 13:52:46,811 CommitLogReader.java:257 - Finished
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126164.log
DEBUG [main] 2023-02-03 13:52:46,811 CommitLogReader.java:266 - Reading
/var/lib/cassandra/commitlog/CommitLog-7-1674161126165.log (CL version
7, messaging version 12, compression null)
INFO [main] 2023-02-03 13:52:47,985 CommitLogReader.java:257 - Finished
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126165.log
DEBUG [main] 2023-02-03 13:52:47,986 CommitLogReader.java:266 - Reading
/var/lib/cassandra/commitlog/CommitLog-7-1674161126166.log (CL version
7, messaging version 12, compression null)
INFO [main] 2023-02-03 13:52:49,282 CommitLogReader.java:257 - Finished
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126166.log
DEBUG [main] 2023-02-03 13:52:49,283 CommitLogReader.java:266 - Reading
/var/lib/cassandra/commitlog/CommitLog-7-1674161126167.log (CL version
7, messaging version 12, compression null)
ERROR [main] 2023-02-03 13:52:49,651 JVMStabilityInspector.java:196 -
Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
Mutation checksum failure at 11231154 in Next section at 11230925 in
CommitLog-7-1674161126167.log
at
org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:387)
at
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244)
at
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)
at
org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:191)
at
org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:200)
at
org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:181)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:357)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:752)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:876)
How to proceed?
Thank you!
-Joe
--
This email has been checked for viruses by AVG antivirus software.
www.avg.com
Re: Startup fails - 4.1.0
Posted by Joe Obernberger <jo...@gmail.com>.
Thank you Sean. I had to remove two of the files and then it started.
Cheers!
-Joe
On 2/3/2023 3:52 PM, Durity, Sean R via user wrote:
>
> In most cases, I would delete the corrupt commit log file and restart.
> Then run repairs on that node. I have seen cases where multiple files
> are corrupted and it is easier to remove all commit log files to get
> the node restarted.
>
> Sean R. Durity
>
> *From:*Joe Obernberger <jo...@gmail.com>
> *Sent:* Friday, February 3, 2023 3:15 PM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Startup fails - 4.1.0
>
> Hi all - cluster had a power outage and one of the nodes in a 14 nodes
> cluster isn't starting with: DEBUG [MemtableFlushWriter: 1] 2023-02-03
> 13: 52: 45,468 ColumnFamilyStore. java: 1329 - Flushed to
> [BigTableReader(path='/data/2/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8407-big-Data. db'),
>
> INTERNAL USE
>
> Hi all - cluster had a power outage and one of the nodes in a 14 nodes
> cluster isn't starting with:
> DEBUG [MemtableFlushWriter:1] 2023-02-03 13:52:45,468
> ColumnFamilyStore.java:1329 - Flushed to
> [BigTableReader(path='/data/2/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8407-big-Data.db'),
>
> BigTableReader(path='/data/3/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8408-big-Data.db'),
>
> BigTableReader(path='/data/4/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8409-big-Data.db'),
>
> BigTableReader(path='/data/5/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8410-big-Data.db'),
>
> BigTableReader(path='/data/6/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8411-big-Data.db'),
>
> BigTableReader(path='/data/8/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8412-big-Data.db'),
>
> BigTableReader(path='/data/9/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8413-big-Data.db')]
>
> (7 sstables, 92.858MiB), biggest 15.420MiB, smallest 10.307MiB
> INFO [main] 2023-02-03 13:52:45,621 CommitLogReader.java:257 - Finished
> reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126163.log
> DEBUG [main] 2023-02-03 13:52:45,622 CommitLogReader.java:266 - Reading
> /var/lib/cassandra/commitlog/CommitLog-7-1674161126164.log (CL version
> 7, messaging version 12, compression null)
> INFO [main] 2023-02-03 13:52:46,811 CommitLogReader.java:257 - Finished
> reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126164.log
> DEBUG [main] 2023-02-03 13:52:46,811 CommitLogReader.java:266 - Reading
> /var/lib/cassandra/commitlog/CommitLog-7-1674161126165.log (CL version
> 7, messaging version 12, compression null)
> INFO [main] 2023-02-03 13:52:47,985 CommitLogReader.java:257 - Finished
> reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126165.log
> DEBUG [main] 2023-02-03 13:52:47,986 CommitLogReader.java:266 - Reading
> /var/lib/cassandra/commitlog/CommitLog-7-1674161126166.log (CL version
> 7, messaging version 12, compression null)
> INFO [main] 2023-02-03 13:52:49,282 CommitLogReader.java:257 - Finished
> reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126166.log
> DEBUG [main] 2023-02-03 13:52:49,283 CommitLogReader.java:266 - Reading
> /var/lib/cassandra/commitlog/CommitLog-7-1674161126167.log (CL version
> 7, messaging version 12, compression null)
> ERROR [main] 2023-02-03 13:52:49,651 JVMStabilityInspector.java:196 -
> Exiting due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
>
> Mutation checksum failure at 11231154 in Next section at 11230925 in
> CommitLog-7-1674161126167.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:387)
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244)
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:191)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:200)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:181)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:357)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:752)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:876)
> How to proceed?
> Thank you!
> -Joe
> --
> This email has been checked for viruses by AVG antivirus software.
> https://urldefense.com/v3/__http://www.avg.com__;!!M-nmYVHPHQ!Js4daAIOfZ53leb5CNpzGIRZEtXKw9d4dmRxBCGsYe-51J4i2NkrYuazzFt8O5U-KdQ3HCo9xu4_AeqVYBNySKpz31KzFe0cOQ$
> <https://urldefense.com/v3/__http:/www.avg.com__;!!M-nmYVHPHQ!Js4daAIOfZ53leb5CNpzGIRZEtXKw9d4dmRxBCGsYe-51J4i2NkrYuazzFt8O5U-KdQ3HCo9xu4_AeqVYBNySKpz31KzFe0cOQ$>
RE: Startup fails - 4.1.0
Posted by "Durity, Sean R via user" <us...@cassandra.apache.org>.
In most cases, I would delete the corrupt commit log file and restart. Then run repairs on that node. I have seen cases where multiple files are corrupted and it is easier to remove all commit log files to get the node restarted.
Sean R. Durity
From: Joe Obernberger <jo...@gmail.com>
Sent: Friday, February 3, 2023 3:15 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Startup fails - 4.1.0
Hi all - cluster had a power outage and one of the nodes in a 14 nodes cluster isn't starting with: DEBUG [MemtableFlushWriter: 1] 2023-02-03 13: 52: 45,468 ColumnFamilyStore. java: 1329 - Flushed to [BigTableReader(path='/data/2/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8407-big-Data. db'),
INTERNAL USE
Hi all - cluster had a power outage and one of the nodes in a 14 nodes
cluster isn't starting with:
DEBUG [MemtableFlushWriter:1] 2023-02-03 13:52:45,468
ColumnFamilyStore.java:1329 - Flushed to
[BigTableReader(path='/data/2/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8407-big-Data.db'),
BigTableReader(path='/data/3/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8408-big-Data.db'),
BigTableReader(path='/data/4/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8409-big-Data.db'),
BigTableReader(path='/data/5/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8410-big-Data.db'),
BigTableReader(path='/data/6/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8411-big-Data.db'),
BigTableReader(path='/data/8/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8412-big-Data.db'),
BigTableReader(path='/data/9/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8413-big-Data.db')]
(7 sstables, 92.858MiB), biggest 15.420MiB, smallest 10.307MiB
INFO [main] 2023-02-03 13:52:45,621 CommitLogReader.java:257 - Finished
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126163.log
DEBUG [main] 2023-02-03 13:52:45,622 CommitLogReader.java:266 - Reading
/var/lib/cassandra/commitlog/CommitLog-7-1674161126164.log (CL version
7, messaging version 12, compression null)
INFO [main] 2023-02-03 13:52:46,811 CommitLogReader.java:257 - Finished
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126164.log
DEBUG [main] 2023-02-03 13:52:46,811 CommitLogReader.java:266 - Reading
/var/lib/cassandra/commitlog/CommitLog-7-1674161126165.log (CL version
7, messaging version 12, compression null)
INFO [main] 2023-02-03 13:52:47,985 CommitLogReader.java:257 - Finished
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126165.log
DEBUG [main] 2023-02-03 13:52:47,986 CommitLogReader.java:266 - Reading
/var/lib/cassandra/commitlog/CommitLog-7-1674161126166.log (CL version
7, messaging version 12, compression null)
INFO [main] 2023-02-03 13:52:49,282 CommitLogReader.java:257 - Finished
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126166.log
DEBUG [main] 2023-02-03 13:52:49,283 CommitLogReader.java:266 - Reading
/var/lib/cassandra/commitlog/CommitLog-7-1674161126167.log (CL version
7, messaging version 12, compression null)
ERROR [main] 2023-02-03 13:52:49,651 JVMStabilityInspector.java:196 -
Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
Mutation checksum failure at 11231154 in Next section at 11230925 in
CommitLog-7-1674161126167.log
at
org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:387)
at
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244)
at
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)
at
org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:191)
at
org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:200)
at
org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:181)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:357)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:752)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:876)
How to proceed?
Thank you!
-Joe
--
This email has been checked for viruses by AVG antivirus software.
https://urldefense.com/v3/__http://www.avg.com__;!!M-nmYVHPHQ!Js4daAIOfZ53leb5CNpzGIRZEtXKw9d4dmRxBCGsYe-51J4i2NkrYuazzFt8O5U-KdQ3HCo9xu4_AeqVYBNySKpz31KzFe0cOQ$<https://urldefense.com/v3/__http:/www.avg.com__;!!M-nmYVHPHQ!Js4daAIOfZ53leb5CNpzGIRZEtXKw9d4dmRxBCGsYe-51J4i2NkrYuazzFt8O5U-KdQ3HCo9xu4_AeqVYBNySKpz31KzFe0cOQ$>