You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Dikang Gu <di...@gmail.com> on 2017/09/19 23:36:12 UTC
Commitlog without header
Hello,
In our production cluster, we had multiple times that after a *unclean*
shutdown, cassandra sever can not start due to commit log exceptions:
2017-09-17_06:06:32.49830 ERROR 06:06:32 [main]: Exiting due to error while
processing commit log during initialization.
2017-09-17_06:06:32.49831
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
Could not read commit log descriptor in file
/data/cassandra/commitlog/CommitLog-5-1503088780367.log
2017-09-17_06:06:32.49831 at
org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:634)
[apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
2017-09-17_06:06:32.49831 at
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:303)
[apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
2017-09-17_06:06:32.49831 at
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
[apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
2017-09-17_06:06:32.49832 at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
[apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
2017-09-17_06:06:32.49832 at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
[apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
2017-09-17_06:06:32.49832 at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:302)
[apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
2017-09-17_06:06:32.49832 at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:544)
[apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
2017-09-17_06:06:32.49832 at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:607)
[apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
I add some logging to the CommitLogDescriptor.readHeader(), and find the
header is empty in the failure case. By empty, I mean all the fields in the
header are 0:
2017-09-19_22:43:02.22112 INFO 22:43:02 [main]: Dikang: crc: 0, checkcrc:
2077607535
2017-09-19_22:43:02.22130 INFO 22:43:02 [main]: Dikang: version: 0, id: 0,
parametersLength: 0
As a result, it did not pass the crc check, and failed the commit log
replay.
My question is: is it a known issue that some race condition can cause
empty header in commit log? If so, it should be safe just skip last commit
log with empty header, right?
As you can see, we are using Cassandra 2.2.5.
Thanks
Dikang.
Re: Commitlog without header
Posted by Benjamin Lerer <be...@datastax.com>.
It might be related to CASSANDRA-12539 (in cassandra-2.2.9).
We used to use 2 file pointers when creating commitlog segments. So if you
run out of file pointers while requesting the second file pointer you would
end up with an empty segment.
Re: Commitlog without header
Posted by Dikang Gu <di...@gmail.com>.
I will try the fixes, thanks Benjamin & Jeff.
On Thu, Sep 21, 2017 at 8:55 PM, Jeff Jirsa <jj...@gmail.com> wrote:
> https://issues.apache.org/jira/plugins/servlet/mobile#
> issue/CASSANDRA-11995
>
>
>
> --
> Jeff Jirsa
>
>
> On Sep 19, 2017, at 4:36 PM, Dikang Gu <di...@gmail.com> wrote:
>
> Hello,
>
> In our production cluster, we had multiple times that after a *unclean*
> shutdown, cassandra sever can not start due to commit log exceptions:
>
> 2017-09-17_06:06:32.49830 ERROR 06:06:32 [main]: Exiting due to error while
> processing commit log during initialization.
> 2017-09-17_06:06:32.49831
> org.apache.cassandra.db.commitlog.CommitLogReplayer$
> CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/cassandra/commitlog/CommitLog-5-1503088780367.log
> 2017-09-17_06:06:32.49831 at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(
> CommitLogReplayer.java:634)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49831 at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.
> recover(CommitLogReplayer.java:303)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49831 at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.
> recover(CommitLogReplayer.java:147)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.service.CassandraDaemon.setup(
> CassandraDaemon.java:302)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.service.CassandraDaemon.activate(
> CassandraDaemon.java:544)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.service.CassandraDaemon.main(
> CassandraDaemon.java:607)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
>
> I add some logging to the CommitLogDescriptor.readHeader(), and find the
> header is empty in the failure case. By empty, I mean all the fields in the
> header are 0:
>
> 2017-09-19_22:43:02.22112 INFO 22:43:02 [main]: Dikang: crc: 0, checkcrc:
> 2077607535 <(207)%20760-7535>
> 2017-09-19_22:43:02.22130 INFO 22:43:02 [main]: Dikang: version: 0, id: 0,
> parametersLength: 0
>
> As a result, it did not pass the crc check, and failed the commit log
> replay.
>
> My question is: is it a known issue that some race condition can cause
> empty header in commit log? If so, it should be safe just skip last commit
> log with empty header, right?
>
> As you can see, we are using Cassandra 2.2.5.
>
> Thanks
> Dikang.
>
>
--
Dikang
Re: Commitlog without header
Posted by Dikang Gu <di...@gmail.com>.
I will try the fixes, thanks Benjamin & Jeff.
On Thu, Sep 21, 2017 at 8:55 PM, Jeff Jirsa <jj...@gmail.com> wrote:
> https://issues.apache.org/jira/plugins/servlet/mobile#
> issue/CASSANDRA-11995
>
>
>
> --
> Jeff Jirsa
>
>
> On Sep 19, 2017, at 4:36 PM, Dikang Gu <di...@gmail.com> wrote:
>
> Hello,
>
> In our production cluster, we had multiple times that after a *unclean*
> shutdown, cassandra sever can not start due to commit log exceptions:
>
> 2017-09-17_06:06:32.49830 ERROR 06:06:32 [main]: Exiting due to error while
> processing commit log during initialization.
> 2017-09-17_06:06:32.49831
> org.apache.cassandra.db.commitlog.CommitLogReplayer$
> CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/cassandra/commitlog/CommitLog-5-1503088780367.log
> 2017-09-17_06:06:32.49831 at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(
> CommitLogReplayer.java:634)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49831 at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.
> recover(CommitLogReplayer.java:303)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49831 at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.
> recover(CommitLogReplayer.java:147)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.service.CassandraDaemon.setup(
> CassandraDaemon.java:302)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.service.CassandraDaemon.activate(
> CassandraDaemon.java:544)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.service.CassandraDaemon.main(
> CassandraDaemon.java:607)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
>
> I add some logging to the CommitLogDescriptor.readHeader(), and find the
> header is empty in the failure case. By empty, I mean all the fields in the
> header are 0:
>
> 2017-09-19_22:43:02.22112 INFO 22:43:02 [main]: Dikang: crc: 0, checkcrc:
> 2077607535 <(207)%20760-7535>
> 2017-09-19_22:43:02.22130 INFO 22:43:02 [main]: Dikang: version: 0, id: 0,
> parametersLength: 0
>
> As a result, it did not pass the crc check, and failed the commit log
> replay.
>
> My question is: is it a known issue that some race condition can cause
> empty header in commit log? If so, it should be safe just skip last commit
> log with empty header, right?
>
> As you can see, we are using Cassandra 2.2.5.
>
> Thanks
> Dikang.
>
>
--
Dikang
Re: Commitlog without header
Posted by Jeff Jirsa <jj...@gmail.com>.
https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-11995
--
Jeff Jirsa
> On Sep 19, 2017, at 4:36 PM, Dikang Gu <di...@gmail.com> wrote:
>
> Hello,
>
> In our production cluster, we had multiple times that after a *unclean*
> shutdown, cassandra sever can not start due to commit log exceptions:
>
> 2017-09-17_06:06:32.49830 ERROR 06:06:32 [main]: Exiting due to error while
> processing commit log during initialization.
> 2017-09-17_06:06:32.49831
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/cassandra/commitlog/CommitLog-5-1503088780367.log
> 2017-09-17_06:06:32.49831 at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:634)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49831 at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:303)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49831 at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:302)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:544)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:607)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
>
> I add some logging to the CommitLogDescriptor.readHeader(), and find the
> header is empty in the failure case. By empty, I mean all the fields in the
> header are 0:
>
> 2017-09-19_22:43:02.22112 INFO 22:43:02 [main]: Dikang: crc: 0, checkcrc:
> 2077607535
> 2017-09-19_22:43:02.22130 INFO 22:43:02 [main]: Dikang: version: 0, id: 0,
> parametersLength: 0
>
> As a result, it did not pass the crc check, and failed the commit log
> replay.
>
> My question is: is it a known issue that some race condition can cause
> empty header in commit log? If so, it should be safe just skip last commit
> log with empty header, right?
>
> As you can see, we are using Cassandra 2.2.5.
>
> Thanks
> Dikang.
Re: Commitlog without header
Posted by Jeff Jirsa <jj...@gmail.com>.
https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-11995
--
Jeff Jirsa
> On Sep 19, 2017, at 4:36 PM, Dikang Gu <di...@gmail.com> wrote:
>
> Hello,
>
> In our production cluster, we had multiple times that after a *unclean*
> shutdown, cassandra sever can not start due to commit log exceptions:
>
> 2017-09-17_06:06:32.49830 ERROR 06:06:32 [main]: Exiting due to error while
> processing commit log during initialization.
> 2017-09-17_06:06:32.49831
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/cassandra/commitlog/CommitLog-5-1503088780367.log
> 2017-09-17_06:06:32.49831 at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:634)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49831 at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:303)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49831 at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:302)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:544)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
> 2017-09-17_06:06:32.49832 at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:607)
> [apache-cassandra-2.2.5+git20170612.e1857fa.jar:2.2.5+git20170612.e1857fa]
>
> I add some logging to the CommitLogDescriptor.readHeader(), and find the
> header is empty in the failure case. By empty, I mean all the fields in the
> header are 0:
>
> 2017-09-19_22:43:02.22112 INFO 22:43:02 [main]: Dikang: crc: 0, checkcrc:
> 2077607535
> 2017-09-19_22:43:02.22130 INFO 22:43:02 [main]: Dikang: version: 0, id: 0,
> parametersLength: 0
>
> As a result, it did not pass the crc check, and failed the commit log
> replay.
>
> My question is: is it a known issue that some race condition can cause
> empty header in commit log? If so, it should be safe just skip last commit
> log with empty header, right?
>
> As you can see, we are using Cassandra 2.2.5.
>
> Thanks
> Dikang.