You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Ali Akhtar <al...@gmail.com> on 2016/10/24 02:18:27 UTC

CommitLogReadHandler$CommitLogReadException: Unexpected error deserializing mutation

I have a single node cassandra installation on my dev laptop, which is used
just for dev / testing.

Recently, whenever I restart my laptop, Cassandra fails to start when I run
it via 'sudo service cassandra start'.

Doing a tail on /var/log/cassandra/system.log gives this log:

*INFO  [main] 2016-10-24 07:08:02,950 CommitLog.java:166 - Replaying
/var/lib/cassandra/commitlog/CommitLog-6-1476907676969.log,
/var/lib/cassandra/commitlog/CommitLog-6-1476907676970.log,
/var/lib/cassandra/commitlog/CommitLog-6-1477269052845.log*
*ERROR [main] 2016-10-24 07:08:03,357 JVMStabilityInspector.java:82 -
Exiting due to error while processing commit log during initialization.*
*org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
Unexpected error deserializing mutation; saved to
/tmp/mutation9186356142128811141dat.  This may be caused by replaying a
mutation against a table with the same name but incompatible schema.
Exception follows: org.apache.cassandra.serializers.MarshalException: Not
enough bytes to read 0th field board_id*
* at
org.apache.cassandra.db.commitlog.CommitLogReader.readMutation(CommitLogReader.java:410)
[apache-cassandra-3.9.jar:3.9]*
* at
org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:343)
[apache-cassandra-3.9.jar:3.9]*
* at
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:202)
[apache-cassandra-3.9.jar:3.9]*
* at
org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:85)
[apache-cassandra-3.9.jar:3.9]*
* at
org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:135)
[apache-cassandra-3.9.jar:3.9]*
* at
org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:187)
[apache-cassandra-3.9.jar:3.9]*
* at
org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:167)
[apache-cassandra-3.9.jar:3.9]*
* at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:323)
[apache-cassandra-3.9.jar:3.9]*
* at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:601)
[apache-cassandra-3.9.jar:3.9]*
* at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:730)
[apache-cassandra-3.9.jar:3.9]*


I then have to do 'sudo rm -rf /var/lib/cassandra/commitlog/*' which fixes
the problem, but then I lose all of my data.

It looks like its saying there wasn't enough data to read the field
'board_id', any ideas why that would be?

Re: CommitLogReadHandler$CommitLogReadException: Unexpected error deserializing mutation

Posted by Jonathan Haddad <jo...@jonhaddad.com>.

Nobody that I'm aware of can verify tick tock releases as stable.  You can
use the latest stuff if you want, just know there's probably bugs (like
you're seeing) that nobody is aware of.  Tick Tock releases (3.1, 3.2, etc)
should really be thought of as development releases.  They won't receive
bug fixes even under circumstances like you've seen in 3.9.  The only way
you will be able to get a bug fix for the issue you've just found is by
backporting it yourself or upgrading to a new version with new features.

On Mon, Oct 24, 2016 at 6:35 PM Ali Akhtar <al...@gmail.com> wrote:

> I want some of the newer UDT features, like not needing to have frozen UDTs
>
> On Tue, Oct 25, 2016 at 6:34 AM, Ali Akhtar <al...@gmail.com> wrote:
>
> 3.0.x? Isn't 3.7 stable?
>
> On Tue, Oct 25, 2016 at 6:32 AM, Jonathan Haddad <jo...@jonhaddad.com>
> wrote:
>
> If you're not in prod *yet*, I once again recommend not using 3.9 for
> anything serious.  Use the latest 3.0.x.
>
> On Mon, Oct 24, 2016 at 6:19 PM Ali Akhtar <al...@gmail.com> wrote:
>
> Stefania,
>
> As this is just on my dev laptop, what I'm really looking for is a quick
> 1-2 min fix to this solution that will act as a workaround.
>
> If I drop the keyspace and recreate it, will that fix this problem? Or do
> I need to uninstall 3.9 and go back to 3.7?
>
> I probably have made a couple of changes to the schema of the tables after
> I first created them.
>
> Happy to share the schema with you privately if it will lead to a 1-2 min
> fix to this problem for me.
>
> On Tue, Oct 25, 2016 at 5:59 AM, Stefania Alborghetti <
> stefania.alborghetti@datastax.com> wrote:
>
> Did the schema change? This would be 12397.
>
> If not, and if you don't mind sharing the data, or you have the steps to
> reproduce it, could you please open a ticket so it can be looked at? You
> need to attach the schema as well.
>
> On Mon, Oct 24, 2016 at 9:33 PM, Ali Akhtar <al...@gmail.com> wrote:
>
> Its 'text'.  Don't know the answer of the 2nd question.
>
> On Mon, Oct 24, 2016 at 6:31 PM, Jonathan Haddad <jo...@jonhaddad.com>
> wrote:
>
> What type is board id? Is the value a tombstone?
>
> On Mon, Oct 24, 2016 at 1:38 AM Ali Akhtar <al...@gmail.com> wrote:
>
> Thanks, but I did come across those, it doesn't look like they provide a
> resolution.
>
> On Mon, Oct 24, 2016 at 1:36 PM, DuyHai Doan <do...@gmail.com> wrote:
>
> You may read those:
>
> https://issues.apache.org/jira/browse/CASSANDRA-12121
> https://issues.apache.org/jira/browse/CASSANDRA-12397
>
> On Mon, Oct 24, 2016 at 10:24 AM, Ali Akhtar <al...@gmail.com> wrote:
>
> Any workarounds that don't involve me having to figure out how to
> uninstall and re-install a different version?
>
> On Mon, Oct 24, 2016 at 1:24 PM, Ali Akhtar <al...@gmail.com> wrote:
>
> 3.9..
>
> On Mon, Oct 24, 2016 at 1:22 PM, DuyHai Doan <do...@gmail.com> wrote:
>
> Which version of C* ? There was similar issues with commitlogs in tic-toc
> versions.
>
> On Mon, Oct 24, 2016 at 4:18 AM, Ali Akhtar <al...@gmail.com> wrote:
>
> I have a single node cassandra installation on my dev laptop, which is
> used just for dev / testing.
>
> Recently, whenever I restart my laptop, Cassandra fails to start when I
> run it via 'sudo service cassandra start'.
>
> Doing a tail on /var/log/cassandra/system.log gives this log:
>
> *INFO  [main] 2016-10-24 07:08:02,950 CommitLog.java:166 - Replaying
> /var/lib/cassandra/commitlog/CommitLog-6-1476907676969.log,
> /var/lib/cassandra/commitlog/CommitLog-6-1476907676970.log,
> /var/lib/cassandra/commitlog/CommitLog-6-1477269052845.log*
> *ERROR [main] 2016-10-24 07:08:03,357 JVMStabilityInspector.java:82 -
> Exiting due to error while processing commit log during initialization.*
> *org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
> Unexpected error deserializing mutation; saved to
> /tmp/mutation9186356142128811141dat.  This may be caused by replaying a
> mutation against a table with the same name but incompatible schema.
> Exception follows: org.apache.cassandra.serializers.MarshalException: Not
> enough bytes to read 0th field board_id*
> * at
> org.apache.cassandra.db.commitlog.CommitLogReader.readMutation(CommitLogReader.java:410)
> [apache-cassandra-3.9.jar:3.9]*
> * at
> org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:343)
> [apache-cassandra-3.9.jar:3.9]*
> * at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:202)
> [apache-cassandra-3.9.jar:3.9]*
> * at
> org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:85)
> [apache-cassandra-3.9.jar:3.9]*
> * at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:135)
> [apache-cassandra-3.9.jar:3.9]*
> * at
> org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:187)
> [apache-cassandra-3.9.jar:3.9]*
> * at
> org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:167)
> [apache-cassandra-3.9.jar:3.9]*
> * at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:323)
> [apache-cassandra-3.9.jar:3.9]*
> * at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:601)
> [apache-cassandra-3.9.jar:3.9]*
> * at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:730)
> [apache-cassandra-3.9.jar:3.9]*
>
>
> I then have to do 'sudo rm -rf /var/lib/cassandra/commitlog/*' which fixes
> the problem, but then I lose all of my data.
>
> It looks like its saying there wasn't enough data to read the field
> 'board_id', any ideas why that would be?
>
>
>
>
>
>
>
>
>
>
> --
>
>
> Stefania Alborghetti
>
> |+852 6114 9265| stefania.alborghetti@datastax.com
>
>
>
>
>

Re: CommitLogReadHandler$CommitLogReadException: Unexpected error deserializing mutation

Posted by Stefania Alborghetti <st...@datastax.com>.

I'm sure you can share the schema and data privately with the ticket
assignee, when the ticket gets assigned and looked at.

If it was a schema change problem, you can try going back to the old schema
if you can recall what it was, but I cannot guarantee this would work
without knowing the root cause. Same thing regarding which release to try,
without knowing the root cause, it's really not possible to advise a
specific release.

The easiest thing to do is to skip the mutations with problems. You still
loose some data but at least not all data. If you see this in your logs:

Replay stopped. If you wish to override this error and continue starting
the node ignoring commit log replay problems, specify
-Dcassandra.commitlog.ignorereplayerrors=true on the command line.

Then it means that you can start Cassandra with
-Dcassandra.commitlog.ignorereplayerrors=true
and it will carry on even if it cannot parse some mutations, which will be
saved in the /tmp folder.

If it was a schema change problem, then you shouldn't need to start with
this property more than once. If the problem persists with new commit log
segments, then it's definitely another problem and you should really open a
ticket.

On Tue, Oct 25, 2016 at 10:36 AM, kurt Greaves <ku...@instaclustr.com> wrote:

>
> On 25 October 2016 at 01:34, Ali Akhtar <al...@gmail.com> wrote:
>
>> I want some of the newer UDT features, like not needing to have frozen
>> UDTs
>
>
> You can try Instaclustr's 3.7 LTS release which is just 3.7 with some
> backported fixes from later versions. If you absolutely need those new
> features it's probably your best bet (until 4.0), however note that it's
> still 3.7 and likely less stable than the latest 3.0.x releases.
>
> https://github.com/instaclustr/cassandra
>
> Read the README at the repo for more info.
>
> Kurt Greaves
> kurt@instaclustr.com
> www.instaclustr.com
>

-- 

Stefania Alborghetti

|+852 6114 9265| stefania.alborghetti@datastax.com