You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Ran Tavory <ra...@gmail.com> on 2013/10/06 18:23:14 UTC

AssertionError: DecoratedKey(... ) != DecoratedKey (...)

Pardon me, now with the appropriate subject line...

Hi, I have a small cluster of 1.2.6 and after some config changes I started
seeing errors int the logs.

Not sure that's related, but the changes I performed were to disable hinted
handoff and disable auto snapshot. I'll try to reverte these, see if the
picture changes.

But anyway, that seems like a bug, right?

I see this across many nodes, not only one.

ERROR [ReplicateOnWriteStage:105] 2013-10-06 16:13:13,799
CassandraDaemon.java (line 192) Exception in thread
Thread[ReplicateOnWriteStage:105,5,main]
java.lang.AssertionError: DecoratedKey(-9223372036854775808, ) !=
DecoratedKey(-1854619418400985942, 000338393900000a4769676f707469782d3100)
in
/raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db
 at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:119)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60)
 at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
 at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
 at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
 at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
 at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
at
org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
 at
org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:772)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1593)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
ERROR [ReplicateOnWriteStage:82] 2013-10-06 16:13:14,249
CassandraDaemon.java (line 192) Exception in thread
Thread[ReplicateOnWriteStage:82,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to
seek to position 2171332 in
/raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db
(1250125 bytes) in read-only mode
 at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1597)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException: unable to seek to position
2171332 in
/raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db
(1250125 bytes) in read-only mode
 at
org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:306)
at
org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:42)
 at
org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1054)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:94)
 at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:112)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60)
 at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
 at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
 at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
 at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
 at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
at
org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
 at
org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:772)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1593)

-- 
/Ran
http://tavory.com

Re: AssertionError: DecoratedKey(... ) != DecoratedKey (...)

Posted by Ran Tavory <ra...@gmail.com>.

Update: I've reverted hinted_handoff_enabled back to its default value of
true and the errors stopped. Is this just a coincidence, or could be
related?



On Sun, Oct 6, 2013 at 7:23 PM, Ran Tavory <ra...@gmail.com> wrote:

> Pardon me, now with the appropriate subject line...
>
> Hi, I have a small cluster of 1.2.6 and after some config changes I
> started seeing errors int the logs.
>
> Not sure that's related, but the changes I performed were to disable
> hinted handoff and disable auto snapshot. I'll try to reverte these, see if
> the picture changes.
>
> But anyway, that seems like a bug, right?
>
> I see this across many nodes, not only one.
>
> ERROR [ReplicateOnWriteStage:105] 2013-10-06 16:13:13,799
> CassandraDaemon.java (line 192) Exception in thread
> Thread[ReplicateOnWriteStage:105,5,main]
> java.lang.AssertionError: DecoratedKey(-9223372036854775808, ) !=
> DecoratedKey(-1854619418400985942, 000338393900000a4769676f707469782d3100)
> in
> /raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db
>  at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:119)
> at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60)
>  at
> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
>  at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
>  at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
>  at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
> at org.apache.cassandra.db.Table.getRow(Table.java:347)
>  at
> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
> at
> org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
>  at
> org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:772)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1593)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
> ERROR [ReplicateOnWriteStage:82] 2013-10-06 16:13:14,249
> CassandraDaemon.java (line 192) Exception in thread
> Thread[ReplicateOnWriteStage:82,5,main]
> java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to
> seek to position 2171332 in
> /raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db
> (1250125 bytes) in read-only mode
>  at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1597)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.IllegalArgumentException: unable to seek to position
> 2171332 in
> /raid0/cassandra/data/test_realtime/activities_summary_realtime/test_realtime-activities_summary_realtime-ic-2-Data.db
> (1250125 bytes) in read-only mode
>  at
> org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:306)
> at
> org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:42)
>  at
> org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1054)
> at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:94)
>  at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:112)
> at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60)
>  at
> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
>  at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
>  at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
>  at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
> at org.apache.cassandra.db.Table.getRow(Table.java:347)
>  at
> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
> at
> org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
>  at
> org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:772)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1593)
>
> --
> /Ran
> http://tavory.com
>



-- 
/Ran
http://tavory.com