[jira] [Updated] (CASSANDRA-2604) EOFException on commitlogs

["Sylvain Lebresne (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CASSANDRA-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sylvain Lebresne updated CASSANDRA-2604:
----------------------------------------

    Attachment: 0001-avoid-modifying-original-mutation-during-apply.patch

Terje, do you use secondary indexes in your tests ?

I think I've found a race (related to secondary indexes) that would explain the observed behavior. But without a good way to reproduce, hard to say if that is the problem.

The problem is in Table.apply(), where ignoreObsoleteMutations() may remove some columns from the original mutation. And since the commit log aliases the same mutation and, unless you are in batch mode, can write it anytime during apply(), it is possible for a column to be removed between the time where the commit log serialize the number of columns and the actual serialization of the columns (explaining why there is less columns than advertised on replay).

Attaching a patch that lazily clone the column family in ignoreObsoleteMutation to avoid modifying the original one. Note that 0.7 is not concerned, because in 0.7 we give to the commit log the mutation already serialized.

> EOFException on commitlogs
> --------------------------
>
>                 Key: CASSANDRA-2604
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2604
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8 beta 1
>            Reporter: Terje Marthinussen
>            Assignee: Sylvain Lebresne
>             Fix For: 0.8.0
>
>         Attachments: 0001-avoid-modifying-original-mutation-during-apply.patch
>
>
> I have seen this occasionally since we started testing 0.8.
> It happens when reading commitlogs on startups.
> However, I have seen it a lot less on 0.8 beta2 (although this is from beta 2)
> ERROR [main] 2011-05-04 18:02:38,134 AbstractCassandraDaemon.java (line 330) Exception encountered during startup.
> java.io.EOFException
> 	at java.io.DataInputStream.readByte(DataInputStream.java:250)
> 	at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:357)
> 	at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> 	at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:252)
> 	at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:43)
> 	at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:136)
> 	at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
> 	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:368)
> 	at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:256)
> 	at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:157)
> 	at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173)
> 	at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:313)
> 	at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Note that the line numbers on columnserializer may be off due to some local changes, but those changes are in code not executed in this case and I am 100% sure they do not trigger this problem.
> I looked on this in the debugger in eclipse on a trunk from 0.8 2 weeks ago, and the interesting thing I saw was that according to the debugger, the offset of the inputstream to the deserializer was already at the end (very last byte) of the underlying bytebuffer but according to the stack, it was trying to read the length of the column name (first read done in the deserialized).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira