You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Christopher Wirt (JIRA)" <ji...@apache.org> on 2013/09/30 11:22:45 UTC
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx,
yyy) != DecoratedKey(zzz, kkk)
[ https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13781699#comment-13781699 ]
Christopher Wirt commented on CASSANDRA-4687:
---------------------------------------------
Ran in to this assertion error after upgrading to 2.0.1 from 1.2.10 and switching to Hsha.
We ran for a whole day on the sync server after the migration to 2.0.1 without seeing this error. Once I'd switched over to HsHa I started to see this every 5-10minutes.
We were doing around 400writes/3000reads per second per machine, about half of the writes would be increments to this counter CF.
We were running at INFO logging level.
Looks like it was only happening on the RelicateOnWrite stage for a counter CF
Compaction pre heat cache was set to true.
Worth mentioning also that we had two other DC running 1.2.10 which hadn't been upgraded yet.
ERROR [ReplicateOnWriteStage:19] 2013-09-27 10:17:14,778 CassandraDaemon.java (line 185) Exception in thread Thread[ReplicateOnWriteStage:19,5,main]
java.lang.AssertionError: DecoratedKey(-1754949563326053382, a414b0c07f0547f8a75410555716ced6) != DecoratedKey(-1754949563326053382, aeadcec8184445d4ab631ef4250927d0) in /disk3/cassandra/data/struqrealtime/counters/struqrealtime-counters-jb-831953-Data.db
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:114)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:62)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:87)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1468)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1294)
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:332)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:100)
at org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1107)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1897)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
> Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
> -----------------------------------------------------------
>
> Key: CASSANDRA-4687
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single node cluster
> Reporter: Leonid Shalupov
> Priority: Minor
> Attachments: 4687-debugging.txt
>
>
> Under heavy write load sometimes cassandra fails with assertion error.
> git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
> works fine if global key/row caches disabled in code.
> {quote}
> java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in /var/lib/cassandra/data/...-he-1-Data.db
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:60)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
> at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
> at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
> at org.apache.cassandra.db.Table.getRow(Table.java:378)
> at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
> at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.1#6144)