You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Kearby (JIRA)" <ji...@apache.org> on 2014/01/15 19:51:20 UTC
[jira] [Comment Edited] (CASSANDRA-6285) LCS compaction failing
with Exception
[ https://issues.apache.org/jira/browse/CASSANDRA-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872409#comment-13872409 ]
Brandon Kearby edited comment on CASSANDRA-6285 at 1/15/14 6:49 PM:
--------------------------------------------------------------------
Snapshot of compaction before failing
Added attachment: system-compactions_in_progress-jb-25-Data.db
Logs before failing
{code}
INFO [CompactionExecutor:6] 2014-01-15 18:38:47,690 ColumnFamilyStore.java (line 740) Enqueuing flush of Memtable-compactions_in_progress@856586691(847/8470 serialized/live bytes, 35 ops)
INFO [FlushWriter:3] 2014-01-15 18:38:47,691 Memtable.java (line 333) Writing Memtable-compactions_in_progress@856586691(847/8470 serialized/live bytes, 35 ops)
INFO [FlushWriter:3] 2014-01-15 18:38:47,700 Memtable.java (line 373) Completed flushing /BigData/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-jb-24-Data.db (304 bytes) for commitlog position ReplayPosition(segmentId=1389810508756, position=33429429)
INFO [CompactionExecutor:6] 2014-01-15 18:38:47,703 CompactionTask.java (line 115) Compacting [SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-45-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-46-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-67-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-72-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-78-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-75-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-56-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-62-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-66-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-49-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-47-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-57-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-61-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-79-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-65-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-50-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-58-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-77-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-54-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-53-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-48-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-64-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-68-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-76-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-55-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-74-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-60-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-52-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-69-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-71-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-73-Data.db'), SSTableReader(path='/BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-jb-63-Data.db')]
INFO [CompactionExecutor:6] 2014-01-15 18:39:05,208 ColumnFamilyStore.java (line 740) Enqueuing flush of Memtable-compactions_in_progress@2046507929(0/0 serialized/live bytes, 1 ops)
INFO [FlushWriter:3] 2014-01-15 18:39:05,208 Memtable.java (line 333) Writing Memtable-compactions_in_progress@2046507929(0/0 serialized/live bytes, 1 ops)
INFO [FlushWriter:3] 2014-01-15 18:39:05,218 Memtable.java (line 373) Completed flushing /BigData/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-jb-25-Data.db (42 bytes) for commitlog position ReplayPosition(segmentId=1389810508756, position=33430117)
ERROR [CompactionExecutor:6] 2014-01-15 18:39:05,220 CassandraDaemon.java (line 187) Exception in thread Thread[CompactionExecutor:6,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(-5705444534806265577, 0000000000000000000000000000000000000000000000000000) >= current key DecoratedKey(-7490754936938484492, 00ab1b0000000000000000000000000000000000000000000000) writing into /BigData/lib/cassandra/data/SocialData/signal/SocialData-signal-tmp-jb-80-Data.db
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:142)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:165)
at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:160)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
INFO [MemoryMeter:1] 2014-01-15 18:39:08,865 Memtable.java (line 451) CFS(Keyspace='system', ColumnFamily='sstable_activity') liveRatio is 14.596825396825396 (just-counted was 14.596825396825396). calculation took 1ms for 84 cells
INFO [MemoryMeter:1] 2014-01-15 18:43:44,575 Memtable.java (line 451) CFS(Keyspace='system', ColumnFamily='sstable_activity') liveRatio is 14.591111111111111 (just-counted was 14.585396825396826). calculation took 4ms for 210 cells
(END)
{code}
was (Author: brandon.kearby):
snapshot of compaction before failing
> LCS compaction failing with Exception
> -------------------------------------
>
> Key: CASSANDRA-6285
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6285
> Project: Cassandra
> Issue Type: Bug
> Environment: 4 nodes, shortly updated from 1.2.11 to 2.0.2
> Reporter: David Sauer
> Attachments: system-compactions_in_progress-jb-25-Data.db
>
>
> After altering everything to LCS the table OpsCenter.rollups60 amd one other none OpsCenter-Table got stuck with everything hanging around in L0.
> The compaction started and ran until the logs showed this:
> ERROR [CompactionExecutor:111] 2013-11-01 19:14:53,865 CassandraDaemon.java (line 187) Exception in thread Thread[CompactionExecutor:111,1,RMI Runtime]
> java.lang.RuntimeException: Last written key DecoratedKey(1326283851463420237, 37382e34362e3132382e3139382d6a7576616c69735f6e6f72785f696e6465785f323031335f31305f30382d63616368655f646f63756d656e74736c6f6f6b75702d676574426c6f6f6d46696c746572537061636555736564) >= current key DecoratedKey(954210699457429663, 37382e34362e3132382e3139382d6a7576616c69735f6e6f72785f696e6465785f323031335f31305f30382d63616368655f646f63756d656e74736c6f6f6b75702d676574546f74616c4469736b5370616365557365640b0f) writing into /var/lib/cassandra/data/OpsCenter/rollups60/OpsCenter-rollups60-tmp-jb-58656-Data.db
> at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:141)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:164)
> at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:160)
> at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:296)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Moving back to STC worked to keep the compactions running.
> Especialy my own Table i would like to move to LCS.
> After a major compaction with STC the move to LCS fails with the same Exception.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)