You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ran Tavory <ra...@gmail.com> on 2010/04/15 11:06:15 UTC

AssertionError: DecoratedKey(...) != DecoratedKey(...)

When restarting one of the nodes in my cluster I found this error in the
log. What does this mean?

 INFO [GC inspection] 2010-04-15 05:03:04,898 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 712 ms, 11149016 reclaimed leaving 442336680 used;
max is 4432068608
ERROR [HINTED-HANDOFF-POOL:1] 2010-04-15 05:03:17,948
DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
java.util.concurrent.ExecutionException: java.lang.AssertionError:
DecoratedKey(163143070370570938845670096830182058073,
1K2i35+B8RuuRDP7Gwz3Xw==) !=
DecoratedKey(163143368384879375649994309361429628039,
4k54mGvj7JoT5rBH68K+9A==) in
/outbrain/cassandra/data/outbrain/DocumentMapping-305-Data.db
        at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.AssertionError:
DecoratedKey(163143070370570938845670096830182058073,
1K2i35+B8RuuRDP7Gwz3Xw==) !=
DecoratedKey(163143368384879375649994309361429628039,
4k54mGvj7JoT5rBH68K+9A==) in
/outbrain/cassandra/data/outbrain/DocumentMapping-305-Data.db
        at
org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:127)
        at
org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
        at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
        at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:830)
        at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750)
        at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719)
        at
org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:122)
        at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
        at
org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:80)
        at
org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:280)
        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        ... 2 more

Re: AssertionError: DecoratedKey(...) != DecoratedKey(...)

Posted by Gary Dusbabek <gd...@gmail.com>.
No data corruption.  There was a bug in the way that the index was
scanned that was manifesting itself when when the index got bigger
than 2GB.

Gary.


On Thu, Apr 15, 2010 at 08:03, Ran Tavory <ra...@gmail.com> wrote:
> yes, this looks like the same issue, thanks Gary.
> Other than seeing the errors in the log I haven't seen any other
> irregularities. (maybe there are, but they haven't surfaced). Does this
> assertion mean data corruption or something else that's worth waiting to
> 0.6.1 for?
>
> On Thu, Apr 15, 2010 at 2:00 PM, Gary Dusbabek <gd...@gmail.com> wrote:
>>
>> Ran,
>>
>> It looks like you're seeing
>> https://issues.apache.org/jira/browse/CASSANDRA-866.  It's fixed in
>> 0.6.1.
>>
>> Gary
>>
>> On Thu, Apr 15, 2010 at 04:06, Ran Tavory <ra...@gmail.com> wrote:
>> > When restarting one of the nodes in my cluster I found this error in the
>> > log. What does this mean?
>> >
>> >  INFO [GC inspection] 2010-04-15 05:03:04,898 GCInspector.java (line
>> > 110) GC
>> > for ConcurrentMarkSweep: 712 ms, 11149016 reclaimed leaving 442336680
>> > used;
>> > max is 4432068608
>> > ERROR [HINTED-HANDOFF-POOL:1] 2010-04-15 05:03:17,948
>> > DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
>> > java.util.concurrent.ExecutionException: java.lang.AssertionError:
>> > DecoratedKey(163143070370570938845670096830182058073,
>> > 1K2i35+B8RuuRDP7Gwz3Xw==) !=
>> > DecoratedKey(163143368384879375649994309361429628039,
>> > 4k54mGvj7JoT5rBH68K+9A==) in
>> > /outbrain/cassandra/data/outbrain/DocumentMapping-305-Data.db
>> >         at
>> > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>> >         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>> >         at
>> >
>> > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
>> >         at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
>> >         at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >         at java.lang.Thread.run(Thread.java:619)
>> > Caused by: java.lang.AssertionError:
>> > DecoratedKey(163143070370570938845670096830182058073,
>> > 1K2i35+B8RuuRDP7Gwz3Xw==) !=
>> > DecoratedKey(163143368384879375649994309361429628039,
>> > 4k54mGvj7JoT5rBH68K+9A==) in
>> > /outbrain/cassandra/data/outbrain/DocumentMapping-305-Data.db
>> >         at
>> >
>> > org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:127)
>> >         at
>> >
>> > org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
>> >         at
>> >
>> > org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:830)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719)
>> >         at
>> >
>> > org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:122)
>> >         at
>> >
>> > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
>> >         at
>> >
>> > org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:80)
>> >         at
>> >
>> > org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:280)
>> >         at
>> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> >         at
>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> >         at
>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >         at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >         ... 2 more
>> >
>
>

Re: AssertionError: DecoratedKey(...) != DecoratedKey(...)

Posted by Ran Tavory <ra...@gmail.com>.
yes, this looks like the same issue, thanks Gary.

Other than seeing the errors in the log I haven't seen any other
irregularities. (maybe there are, but they haven't surfaced). Does this
assertion mean data corruption or something else that's worth waiting to
0.6.1 for?

On Thu, Apr 15, 2010 at 2:00 PM, Gary Dusbabek <gd...@gmail.com> wrote:

> Ran,
>
> It looks like you're seeing
> https://issues.apache.org/jira/browse/CASSANDRA-866.  It's fixed in
> 0.6.1.
>
> Gary
>
> On Thu, Apr 15, 2010 at 04:06, Ran Tavory <ra...@gmail.com> wrote:
> > When restarting one of the nodes in my cluster I found this error in the
> > log. What does this mean?
> >
> >  INFO [GC inspection] 2010-04-15 05:03:04,898 GCInspector.java (line 110)
> GC
> > for ConcurrentMarkSweep: 712 ms, 11149016 reclaimed leaving 442336680
> used;
> > max is 4432068608
> > ERROR [HINTED-HANDOFF-POOL:1] 2010-04-15 05:03:17,948
> > DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
> > java.util.concurrent.ExecutionException: java.lang.AssertionError:
> > DecoratedKey(163143070370570938845670096830182058073,
> > 1K2i35+B8RuuRDP7Gwz3Xw==) !=
> > DecoratedKey(163143368384879375649994309361429628039,
> > 4k54mGvj7JoT5rBH68K+9A==) in
> > /outbrain/cassandra/data/outbrain/DocumentMapping-305-Data.db
> >         at
> > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> >         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> >         at
> >
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >         at java.lang.Thread.run(Thread.java:619)
> > Caused by: java.lang.AssertionError:
> > DecoratedKey(163143070370570938845670096830182058073,
> > 1K2i35+B8RuuRDP7Gwz3Xw==) !=
> > DecoratedKey(163143368384879375649994309361429628039,
> > 4k54mGvj7JoT5rBH68K+9A==) in
> > /outbrain/cassandra/data/outbrain/DocumentMapping-305-Data.db
> >         at
> >
> org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:127)
> >         at
> >
> org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
> >         at
> >
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
> >         at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:830)
> >         at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750)
> >         at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719)
> >         at
> >
> org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:122)
> >         at
> >
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> >         at
> >
> org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:80)
> >         at
> >
> org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:280)
> >         at
> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> >         at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> >         at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >         ... 2 more
> >
>

Re: AssertionError: DecoratedKey(...) != DecoratedKey(...)

Posted by Gary Dusbabek <gd...@gmail.com>.
Ran,

It looks like you're seeing
https://issues.apache.org/jira/browse/CASSANDRA-866.  It's fixed in
0.6.1.

Gary

On Thu, Apr 15, 2010 at 04:06, Ran Tavory <ra...@gmail.com> wrote:
> When restarting one of the nodes in my cluster I found this error in the
> log. What does this mean?
>
>  INFO [GC inspection] 2010-04-15 05:03:04,898 GCInspector.java (line 110) GC
> for ConcurrentMarkSweep: 712 ms, 11149016 reclaimed leaving 442336680 used;
> max is 4432068608
> ERROR [HINTED-HANDOFF-POOL:1] 2010-04-15 05:03:17,948
> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
> java.util.concurrent.ExecutionException: java.lang.AssertionError:
> DecoratedKey(163143070370570938845670096830182058073,
> 1K2i35+B8RuuRDP7Gwz3Xw==) !=
> DecoratedKey(163143368384879375649994309361429628039,
> 4k54mGvj7JoT5rBH68K+9A==) in
> /outbrain/cassandra/data/outbrain/DocumentMapping-305-Data.db
>         at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>         at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.AssertionError:
> DecoratedKey(163143070370570938845670096830182058073,
> 1K2i35+B8RuuRDP7Gwz3Xw==) !=
> DecoratedKey(163143368384879375649994309361429628039,
> 4k54mGvj7JoT5rBH68K+9A==) in
> /outbrain/cassandra/data/outbrain/DocumentMapping-305-Data.db
>         at
> org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:127)
>         at
> org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
>         at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:830)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719)
>         at
> org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:122)
>         at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
>         at
> org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:80)
>         at
> org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:280)
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         ... 2 more
>