You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Philippe <wa...@gmail.com> on 2011/11/29 08:00:18 UTC
Corrupt (negative) value length encountered
Hello,
While running a cleanup, Cassandra stopped with the following exception and
inspecting the logs revealed several exceptions such as below over the past
3 days.Given that it's dying on compactions, I'm really worried.
If a row was trashed, will the error propagate from node to node or will it
be localized and a repair will fix it ?
Thanks
java.io.IOError:
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
column name length 0
at
org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
at
java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
java.io.IOError: java.io.IOException: Corrupt (negative) value length
encountered
at
org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
at
java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
java.io.IOError: java.io.IOException: mmap segment underflow; remaining is
113362603 but 157851071 requested
at
org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
at
java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
at
java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:445)
java.lang.AssertionError: DecoratedKey(Token(bytes[]), ) !=
DecoratedKey(Token(bytes[02c67a8a54000000005311]), 02c67a8a54000000005311)
in /var/lib/cassandra/data/XXXXXX-g-159-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1407)
at
org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1274)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1314)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1261)
at org.apache.cassandra.db.Table.getRow(Table.java:385)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
at
org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:105)
Re: Corrupt (negative) value length encountered
Posted by Philippe <wa...@gmail.com>.
Didn't mention this is a 0.8.6 cluster with 5 nodes and RF=3
I bootstrapped a new node 2 days ago. What's weird is that it didn't pickup
the token i provided in the yaml file so i had to move it.
It looks like CASSANDRA-1992 but I'm not on 0.7.x
Le 29 nov. 2011 08:00, "Philippe" <wa...@gmail.com> a écrit :
> Hello,
> While running a cleanup, Cassandra stopped with the following exception
> and inspecting the logs revealed several exceptions such as below over the
> past 3 days.Given that it's dying on compactions, I'm really worried.
>
> If a row was trashed, will the error propagate from node to node or will
> it be localized and a repair will fix it ?
>
> Thanks
>
> java.io.IOError:
> org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
> column name length 0
> at
> org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at
> java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
>
> java.io.IOError: java.io.IOException: Corrupt (negative) value length
> encountered
> at
> org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at
> java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
>
> java.io.IOError: java.io.IOException: mmap segment underflow; remaining is
> 113362603 but 157851071 requested
> at
> org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at
> java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at
> java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:445)
> java.lang.AssertionError: DecoratedKey(Token(bytes[]), ) !=
> DecoratedKey(Token(bytes[02c67a8a54000000005311]), 02c67a8a54000000005311)
> in /var/lib/cassandra/data/XXXXXX-g-159-Data.db
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1407)
> at
> org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1274)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1314)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1261)
> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> at
> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
> at
> org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:105)
>
>
>