You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Dikang Gu <di...@gmail.com> on 2017/04/19 01:57:55 UTC
Exceptions when upgrade from 2.1.14 to 2.2.5
Hello there,
We are upgrading one of our cluster from 2.1.14 to 2.2.5, but cassandra had
problems replaying the commit logs...
Here is the exception, does anyone experience similar before?
2017-04-19_00:22:21.69943 DEBUG 00:22:21 [main]: Finished reading
/data/cassandra/commitlog/CommitLog-4-1487900877734.log
2017-04-19_00:22:21.69960 DEBUG 00:22:21 [main]: Replaying
/data/cassandra/commitlog/CommitLog-4-1487900877735.log (CL version 4,
messaging version 8, compression null)
2017-04-19_00:22:22.26911 ERROR 00:22:22 [main]: Exiting due to error while
processing commit log during initialization.
2017-04-19_00:22:22.26912
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
Unexpected end of segment
2017-04-19_00:22:22.26912 at
org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:623)
[apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170
2017-04-19_00:22:22.26913 at
org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:484)
[apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170
2017-04-19_00:22:22.26913 at
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:389)
[apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170404.5f2187
2017-04-19_00:22:22.26913 at
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
[apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170404.5f2187
2017-04-19_00:22:22.26913 at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
[apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170404.5f2187b]
2017-04-19_00:22:22.26913 at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
[apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170404.5f2187b]
2017-04-19_00:22:22.26914 at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:302)
[apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170404.5f2187b]
2017-04-19_00:22:22.26914 at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:544)
[apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170404.5f2187b]
2017-04-19_00:22:22.26914 at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:607)
[apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170404.5f2187b]
Thanks
--
Dikang
Re: Exceptions when upgrade from 2.1.14 to 2.2.5
Posted by Hannu Kröger <hk...@gmail.com>.
Hello,
It seems that commit log is broken. One way to fix this would be to remove commit logs and then restart.
This will cause you to lose the writes that were in the commit log but hopefully the data is in other nodes.
In the future to avoid this: before you kill Cassandra, run “nodetool drain”. This will flush the memtables to disk and clear out the commit logs.
Cheers,
Hannu
> On 19 Apr 2017, at 4:57, Dikang Gu <di...@gmail.com> wrote:
>
> Hello there,
>
> We are upgrading one of our cluster from 2.1.14 to 2.2.5, but cassandra had problems replaying the commit logs...
>
> Here is the exception, does anyone experience similar before?
>
> 2017-04-19_00:22:21.69943 DEBUG 00:22:21 [main]: Finished reading /data/cassandra/commitlog/CommitLog-4-1487900877734.log
> 2017-04-19_00:22:21.69960 DEBUG 00:22:21 [main]: Replaying /data/cassandra/commitlog/CommitLog-4-1487900877735.log (CL version 4, messaging version 8, compression null)
> 2017-04-19_00:22:22.26911 ERROR 00:22:22 [main]: Exiting due to error while processing commit log during initialization.
> 2017-04-19_00:22:22.26912 org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Unexpected end of segment
> 2017-04-19_00:22:22.26912 at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:623) [apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170
> 2017-04-19_00:22:22.26913 at org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:484) [apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170
> 2017-04-19_00:22:22.26913 at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:389) [apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170404.5f2187
> 2017-04-19_00:22:22.26913 at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147) [apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170404.5f2187
> 2017-04-19_00:22:22.26913 at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) [apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170404.5f2187b]
> 2017-04-19_00:22:22.26913 at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) [apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170404.5f2187b]
> 2017-04-19_00:22:22.26914 at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:302) [apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170404.5f2187b]
> 2017-04-19_00:22:22.26914 at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:544) [apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170404.5f2187b]
> 2017-04-19_00:22:22.26914 at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:607) [apache-cassandra-2.2.5+git20170404.5f2187b.jar:2.2.5+git20170404.5f2187b]
>
> Thanks
>
> --
> Dikang
>
Re: Exceptions when upgrade from 2.1.14 to 2.2.5
Posted by Dikang Gu <di...@gmail.com>.
Thanks Jeff & Hannu,
Yeah, that's my guess too, I walk around this by set
-Dcassandra.commitlog.ignorereplayerrors=true.
Before upgrade, we did run `nodetool drain`, but seems the 2.1 commit log
were not cleared, and still got replayed by 2.2.
Thanks
Dikang.
On Tue, Apr 18, 2017 at 11:09 PM, Jeff Jirsa <jj...@apache.org> wrote:
>
>
> On 2017-04-18 18:57 (-0700), Dikang Gu <di...@gmail.com> wrote:
> > Hello there,
> >
> > We are upgrading one of our cluster from 2.1.14 to 2.2.5, but cassandra
> had
> > problems replaying the commit logs...
> >
> > Here is the exception, does anyone experience similar before?
> >
> > 2017-04-19_00:22:22.26911 ERROR 00:22:22 [main]: Exiting due to error
> while
> > processing commit log during initialization.
> > 2017-04-19_00:22:22.26912
> > org.apache.cassandra.db.commitlog.CommitLogReplayer$
> CommitLogReplayException:
> > Unexpected end of segment
>
>
> A lot of the commitlog replay code changed from 2.1 -> 2.2 to be more
> strict with "unexpected" errors on replay. Just glancing at the message but
> not tracing line numbers, I'm guessing it's either
> https://issues.apache.org/jira/browse/CASSANDRA-13282 (went into 2.2.10)
> or https://issues.apache.org/jira/browse/CASSANDRA-11995 (PA but not yet
> reviewed, would love a reviewer).
>
--
Dikang
Re: Exceptions when upgrade from 2.1.14 to 2.2.5
Posted by Jeff Jirsa <jj...@apache.org>.
On 2017-04-18 18:57 (-0700), Dikang Gu <di...@gmail.com> wrote:
> Hello there,
>
> We are upgrading one of our cluster from 2.1.14 to 2.2.5, but cassandra had
> problems replaying the commit logs...
>
> Here is the exception, does anyone experience similar before?
>
> 2017-04-19_00:22:22.26911 ERROR 00:22:22 [main]: Exiting due to error while
> processing commit log during initialization.
> 2017-04-19_00:22:22.26912
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Unexpected end of segment
A lot of the commitlog replay code changed from 2.1 -> 2.2 to be more strict with "unexpected" errors on replay. Just glancing at the message but not tracing line numbers, I'm guessing it's either https://issues.apache.org/jira/browse/CASSANDRA-13282 (went into 2.2.10) or https://issues.apache.org/jira/browse/CASSANDRA-11995 (PA but not yet reviewed, would love a reviewer).