You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ramesh Natarajan <ra...@gmail.com> on 2011/10/20 21:04:59 UTC
Cassandra 1.0: Exception in compaction
We are running a 8 node cassandra 1.0 cluster. We are seeing this
exception quite often. Any idea how to debug this issue?
java.lang.IllegalArgumentException: Illegal Capacity: -2
at java.util.ArrayList.<init>(ArrayList.java:110)
at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
few lines before this error
INFO [FlushWriter:222] 2011-10-20 10:52:25,885 Memtable.java (line
237) Writing Memtable-participants@1907757288(6777526/153164388
serialized/live bytes, 199339 ops)
INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
488) Discarding obsolete commit
log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319115938691.log)
INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
488) Discarding obsolete commit
log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319122061956.log)
INFO [FlushWriter:222] 2011-10-20 10:52:26,865 Memtable.java (line
273) Completed flushing
/var/lib/cassandra/data/MSA/participants-h-87-Data.db (14695382 bytes)
INFO [FlushWriter:222] 2011-10-20 10:52:26,866 Memtable.java (line
237) Writing Memtable-modseq@1745889706(13206/311769 serialized/live
bytes, 426 ops)
INFO [FlushWriter:222] 2011-10-20 10:52:26,896 Memtable.java (line
273) Completed flushing
/var/lib/cassandra/data/MSA/modseq-h-262-Data.db (38646 bytes)
INFO [FlushWriter:222] 2011-10-20 10:52:26,897 Memtable.java (line
237) Writing Memtable-msgid@2099219781(3571249/77183008
serialized/live bytes, 109823 ops)
INFO [FlushWriter:222] 2011-10-20 10:52:27,497 Memtable.java (line
273) Completed flushing /var/lib/cassandra/data/MSA/msgid-h-47-Data.db
(8125165 bytes)
INFO [FlushWriter:222] 2011-10-20 10:52:27,498 Memtable.java (line
237) Writing Memtable-uid@578022704(43734344/317200563 serialized/live
bytes, 611301 ops)
INFO [FlushWriter:222] 2011-10-20 10:52:29,802 Memtable.java (line
273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-291-Data.db
(48225128 bytes)
INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
488) Discarding obsolete commit
log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125356477.log)
INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
488) Discarding obsolete commit
log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125683351.log)
INFO [MutationStage:88] 2011-10-20 10:52:29,905
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-modseq@339630706(155217/3664394 serialized/live bytes, 5007
ops)
INFO [FlushWriter:222] 2011-10-20 10:52:29,905 Memtable.java (line
237) Writing Memtable-modseq@339630706(155217/3664394 serialized/live
bytes, 5007 ops)
INFO [FlushWriter:222] 2011-10-20 10:52:30,216 Memtable.java (line
273) Completed flushing
/var/lib/cassandra/data/MSA/modseq-h-263-Data.db (450477 bytes)
ERROR [CompactionExecutor:538] 2011-10-20 10:52:36,132
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[CompactionExecutor:538,1,main]
Another one
INFO [FlushWriter:222] 2011-10-20 10:52:39,623 Memtable.java (line
237) Writing Memtable-uid@2018688194(79740/578345 serialized/live
bytes, 1111 ops)
INFO [FlushWriter:222] 2011-10-20 10:52:39,777 Memtable.java (line
273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-295-Data.db
(142584 bytes)
INFO [CompactionExecutor:544] 2011-10-20 10:52:39,778
CompactionTask.java (line 119) Compacting
[SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-294-Data.db')]
ERROR [CompactionExecutor:544] 2011-10-20 10:52:39,935
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[CompactionExecutor:544,1,main]
java.lang.AssertionError
at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
INFO [MutationStage:89] 2011-10-20 10:52:46,599
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-msgid@917568062(85287/1843250 serialized/live bytes, 2624
ops)
INFO [FlushWriter:222] 2011-10-20 10:52:46,600 Memtable.java (line
237) Writing Memtable-msgid@917568062(85287/1843250 serialized/live
bytes, 2624 ops)
INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:47,057 CommitLog.java (line
488) Discarding obsolete commit
log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125959143.log)
INFO [CompactionExecutor:544] 2011-10-20 10:52:47,058
CompactionTask.java (line 119) Compacting
[SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-298-Data.db')]
INFO [CompactionExecutor:544] 2011-10-20 10:52:47,552
CompactionTask.java (line 222) Compacted to
[/var/lib/cassandra/data/MSA/uid-h-299-Data.db,]. 1,849,016 to
1,842,872 (~99% of original) bytes for 13,592 keys at 3.557692MBPS.
Time: 494ms.
ERROR [CompactionExecutor:539] 2011-10-20 10:52:49,901
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[CompactionExecutor:539,1,main]
java.lang.AssertionError
at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [CompactionExecutor:542] 2011-10-20 10:52:51,983
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[CompactionExecutor:542,1,main]
java.lang.IllegalArgumentException: Illegal Capacity: -3
at java.util.ArrayList.<init>(ArrayList.java:110)
at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
INFO [CompactionExecutor:540] 2011-10-20 10:52:53,618
CompactionTask.java (line 222) Compacted to
[/var/lib/cassandra/data/MSA/participants-h-89-Data.db,]. 73,285,308
to 54,342,740 (~74% of original) bytes for 223,836 keys at
3.594734MBPS. Time: 14,417ms.
INFO [MutationStage:75] 2011-10-20 10:52:53,877
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-participants@1814149120(143888/3251705 serialized/live bytes,
4232 ops)
INFO [FlushWriter:222] 2011-10-20 10:52:53,878 Memtable.java (line
237) Writing Memtable-participants@1814149120(143888/3251705
serialized/live bytes, 4232 ops)
INFO [CompactionExecutor:542] 2011-10-20 10:52:54,313
CompactionTask.java (line 119) Compacting
[SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-93-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-90-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-92-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-94-Data.db')]
ERROR [CompactionExecutor:542] 2011-10-20 10:52:54,532
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[CompactionExecutor:542,1,main]
java.lang.IllegalArgumentException: Illegal Capacity: -2
at java.util.ArrayList.<init>(ArrayList.java:110)
at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
INFO [CompactionExecutor:539] 2011-10-20 10:52:54,787
CompactionTask.java (line 222) Compacted to
[/var/lib/cassandra/data/MSA/uid-h-303-Data.db,]. 3,479,848 to
3,418,524 (~98% of original) bytes for 24,789 keys at 5.388692MBPS.
Time: 605ms.
INFO [CompactionExecutor:541] 2011-10-20 10:52:54,806
CompactionTask.java (line 222) Compacted to
[/var/lib/cassandra/data/MSA/modseq-h-268-Data.db,]. 4,467,791 to
4,227,199 (~94% of original) bytes for 43,572 keys at 5.389534MBPS.
Time: 748ms.
thanks
Ramesh
Re: Cassandra 1.0: Exception in compaction
Posted by Sylvain Lebresne <sy...@datastax.com>.
You don't have the full logs of a node leading to the exception by any change ?
Especially one that leads to a
java.lang.IllegalArgumentException: Illegal Capacity: -2
would be great.
--
Sylvain
On Wed, Oct 26, 2011 at 6:26 PM, Ramesh Natarajan <ra...@gmail.com> wrote:
> We are using the final 1.0.0.
>
> Thanks
> Ramesh
>
> On Wed, Oct 26, 2011 at 11:16 AM, Sylvain Lebresne <sy...@datastax.com> wrote:
>> Also, to be sure, were you using the 1.0.0 final or some RC when getting this
>> exception?
>>
>> On Fri, Oct 21, 2011 at 8:24 PM, Sylvain Lebresne <sy...@datastax.com> wrote:
>>> Would you have the full log for one of those node leading to the exception
>>> that you could share? Not sure that'll help but who knows.
>>>
>>> --
>>> Sylvain
>>>
>>> On Fri, Oct 21, 2011 at 4:34 PM, Ramesh Natarajan <ra...@gmail.com> wrote:
>>>> i am using size based compaction ( the default one ). Also this is on linux.
>>>>
>>>> thanks
>>>> Ramesh
>>>>
>>>> On Fri, Oct 21, 2011 at 4:25 AM, Sylvain Lebresne <sy...@datastax.com> wrote:
>>>>> I believe this is the same as
>>>>> https://issues.apache.org/jira/browse/CASSANDRA-3306.
>>>>>
>>>>> The initial reporter only got this exception with leveled compaction,
>>>>> is it what you are
>>>>> using too (to help narrow it down)? Also, are you using windows by any chance?
>>>>>
>>>>> --
>>>>> Sylvain
>>>>>
>>>>>
>>>>> On Thu, Oct 20, 2011 at 9:04 PM, Ramesh Natarajan <ra...@gmail.com> wrote:
>>>>>> We are running a 8 node cassandra 1.0 cluster. We are seeing this
>>>>>> exception quite often. Any idea how to debug this issue?
>>>>>>
>>>>>>
>>>>>> java.lang.IllegalArgumentException: Illegal Capacity: -2
>>>>>> at java.util.ArrayList.<init>(ArrayList.java:110)
>>>>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>>>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>>
>>>>>>
>>>>>> few lines before this error
>>>>>>
>>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:25,885 Memtable.java (line
>>>>>> 237) Writing Memtable-participants@1907757288(6777526/153164388
>>>>>> serialized/live bytes, 199339 ops)
>>>>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
>>>>>> 488) Discarding obsolete commit
>>>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319115938691.log)
>>>>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
>>>>>> 488) Discarding obsolete commit
>>>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319122061956.log)
>>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,865 Memtable.java (line
>>>>>> 273) Completed flushing
>>>>>> /var/lib/cassandra/data/MSA/participants-h-87-Data.db (14695382 bytes)
>>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,866 Memtable.java (line
>>>>>> 237) Writing Memtable-modseq@1745889706(13206/311769 serialized/live
>>>>>> bytes, 426 ops)
>>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,896 Memtable.java (line
>>>>>> 273) Completed flushing
>>>>>> /var/lib/cassandra/data/MSA/modseq-h-262-Data.db (38646 bytes)
>>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,897 Memtable.java (line
>>>>>> 237) Writing Memtable-msgid@2099219781(3571249/77183008
>>>>>> serialized/live bytes, 109823 ops)
>>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:27,497 Memtable.java (line
>>>>>> 273) Completed flushing /var/lib/cassandra/data/MSA/msgid-h-47-Data.db
>>>>>> (8125165 bytes)
>>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:27,498 Memtable.java (line
>>>>>> 237) Writing Memtable-uid@578022704(43734344/317200563 serialized/live
>>>>>> bytes, 611301 ops)
>>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:29,802 Memtable.java (line
>>>>>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-291-Data.db
>>>>>> (48225128 bytes)
>>>>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
>>>>>> 488) Discarding obsolete commit
>>>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125356477.log)
>>>>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
>>>>>> 488) Discarding obsolete commit
>>>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125683351.log)
>>>>>> INFO [MutationStage:88] 2011-10-20 10:52:29,905
>>>>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>>>>> Memtable-modseq@339630706(155217/3664394 serialized/live bytes, 5007
>>>>>> ops)
>>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:29,905 Memtable.java (line
>>>>>> 237) Writing Memtable-modseq@339630706(155217/3664394 serialized/live
>>>>>> bytes, 5007 ops)
>>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:30,216 Memtable.java (line
>>>>>> 273) Completed flushing
>>>>>> /var/lib/cassandra/data/MSA/modseq-h-263-Data.db (450477 bytes)
>>>>>> ERROR [CompactionExecutor:538] 2011-10-20 10:52:36,132
>>>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>>>> Thread[CompactionExecutor:538,1,main]
>>>>>>
>>>>>>
>>>>>>
>>>>>> Another one
>>>>>>
>>>>>>
>>>>>>
>>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:39,623 Memtable.java (line
>>>>>> 237) Writing Memtable-uid@2018688194(79740/578345 serialized/live
>>>>>> bytes, 1111 ops)
>>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:39,777 Memtable.java (line
>>>>>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-295-Data.db
>>>>>> (142584 bytes)
>>>>>> INFO [CompactionExecutor:544] 2011-10-20 10:52:39,778
>>>>>> CompactionTask.java (line 119) Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
>>>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-294-Data.db')]
>>>>>> ERROR [CompactionExecutor:544] 2011-10-20 10:52:39,935
>>>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>>>> Thread[CompactionExecutor:544,1,main]
>>>>>> java.lang.AssertionError
>>>>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
>>>>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>> INFO [MutationStage:89] 2011-10-20 10:52:46,599
>>>>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>>>>> Memtable-msgid@917568062(85287/1843250 serialized/live bytes, 2624
>>>>>> ops)
>>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:46,600 Memtable.java (line
>>>>>> 237) Writing Memtable-msgid@917568062(85287/1843250 serialized/live
>>>>>> bytes, 2624 ops)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:47,057 CommitLog.java (line
>>>>>> 488) Discarding obsolete commit
>>>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125959143.log)
>>>>>> INFO [CompactionExecutor:544] 2011-10-20 10:52:47,058
>>>>>> CompactionTask.java (line 119) Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
>>>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-298-Data.db')]
>>>>>> INFO [CompactionExecutor:544] 2011-10-20 10:52:47,552
>>>>>> CompactionTask.java (line 222) Compacted to
>>>>>> [/var/lib/cassandra/data/MSA/uid-h-299-Data.db,]. 1,849,016 to
>>>>>> 1,842,872 (~99% of original) bytes for 13,592 keys at 3.557692MBPS.
>>>>>> Time: 494ms.
>>>>>> ERROR [CompactionExecutor:539] 2011-10-20 10:52:49,901
>>>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>>>> Thread[CompactionExecutor:539,1,main]
>>>>>> java.lang.AssertionError
>>>>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
>>>>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>> ERROR [CompactionExecutor:542] 2011-10-20 10:52:51,983
>>>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>>>> Thread[CompactionExecutor:542,1,main]
>>>>>> java.lang.IllegalArgumentException: Illegal Capacity: -3
>>>>>> at java.util.ArrayList.<init>(ArrayList.java:110)
>>>>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>>>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>> INFO [CompactionExecutor:540] 2011-10-20 10:52:53,618
>>>>>> CompactionTask.java (line 222) Compacted to
>>>>>> [/var/lib/cassandra/data/MSA/participants-h-89-Data.db,]. 73,285,308
>>>>>> to 54,342,740 (~74% of original) bytes for 223,836 keys at
>>>>>> 3.594734MBPS. Time: 14,417ms.
>>>>>> INFO [MutationStage:75] 2011-10-20 10:52:53,877
>>>>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>>>>> Memtable-participants@1814149120(143888/3251705 serialized/live bytes,
>>>>>> 4232 ops)
>>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:53,878 Memtable.java (line
>>>>>> 237) Writing Memtable-participants@1814149120(143888/3251705
>>>>>> serialized/live bytes, 4232 ops)
>>>>>>
>>>>>>
>>>>>>
>>>>>> INFO [CompactionExecutor:542] 2011-10-20 10:52:54,313
>>>>>> CompactionTask.java (line 119) Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-93-Data.db'),
>>>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-90-Data.db'),
>>>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-92-Data.db'),
>>>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-94-Data.db')]
>>>>>> ERROR [CompactionExecutor:542] 2011-10-20 10:52:54,532
>>>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>>>> Thread[CompactionExecutor:542,1,main]
>>>>>> java.lang.IllegalArgumentException: Illegal Capacity: -2
>>>>>> at java.util.ArrayList.<init>(ArrayList.java:110)
>>>>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>>>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>> INFO [CompactionExecutor:539] 2011-10-20 10:52:54,787
>>>>>> CompactionTask.java (line 222) Compacted to
>>>>>> [/var/lib/cassandra/data/MSA/uid-h-303-Data.db,]. 3,479,848 to
>>>>>> 3,418,524 (~98% of original) bytes for 24,789 keys at 5.388692MBPS.
>>>>>> Time: 605ms.
>>>>>> INFO [CompactionExecutor:541] 2011-10-20 10:52:54,806
>>>>>> CompactionTask.java (line 222) Compacted to
>>>>>> [/var/lib/cassandra/data/MSA/modseq-h-268-Data.db,]. 4,467,791 to
>>>>>> 4,227,199 (~94% of original) bytes for 43,572 keys at 5.389534MBPS.
>>>>>> Time: 748ms.
>>>>>>
>>>>>>
>>>>>> thanks
>>>>>> Ramesh
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Cassandra 1.0: Exception in compaction
Posted by Ramesh Natarajan <ra...@gmail.com>.
We are using the final 1.0.0.
Thanks
Ramesh
On Wed, Oct 26, 2011 at 11:16 AM, Sylvain Lebresne <sy...@datastax.com> wrote:
> Also, to be sure, were you using the 1.0.0 final or some RC when getting this
> exception?
>
> On Fri, Oct 21, 2011 at 8:24 PM, Sylvain Lebresne <sy...@datastax.com> wrote:
>> Would you have the full log for one of those node leading to the exception
>> that you could share? Not sure that'll help but who knows.
>>
>> --
>> Sylvain
>>
>> On Fri, Oct 21, 2011 at 4:34 PM, Ramesh Natarajan <ra...@gmail.com> wrote:
>>> i am using size based compaction ( the default one ). Also this is on linux.
>>>
>>> thanks
>>> Ramesh
>>>
>>> On Fri, Oct 21, 2011 at 4:25 AM, Sylvain Lebresne <sy...@datastax.com> wrote:
>>>> I believe this is the same as
>>>> https://issues.apache.org/jira/browse/CASSANDRA-3306.
>>>>
>>>> The initial reporter only got this exception with leveled compaction,
>>>> is it what you are
>>>> using too (to help narrow it down)? Also, are you using windows by any chance?
>>>>
>>>> --
>>>> Sylvain
>>>>
>>>>
>>>> On Thu, Oct 20, 2011 at 9:04 PM, Ramesh Natarajan <ra...@gmail.com> wrote:
>>>>> We are running a 8 node cassandra 1.0 cluster. We are seeing this
>>>>> exception quite often. Any idea how to debug this issue?
>>>>>
>>>>>
>>>>> java.lang.IllegalArgumentException: Illegal Capacity: -2
>>>>> at java.util.ArrayList.<init>(ArrayList.java:110)
>>>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>>
>>>>>
>>>>> few lines before this error
>>>>>
>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:25,885 Memtable.java (line
>>>>> 237) Writing Memtable-participants@1907757288(6777526/153164388
>>>>> serialized/live bytes, 199339 ops)
>>>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
>>>>> 488) Discarding obsolete commit
>>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319115938691.log)
>>>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
>>>>> 488) Discarding obsolete commit
>>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319122061956.log)
>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,865 Memtable.java (line
>>>>> 273) Completed flushing
>>>>> /var/lib/cassandra/data/MSA/participants-h-87-Data.db (14695382 bytes)
>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,866 Memtable.java (line
>>>>> 237) Writing Memtable-modseq@1745889706(13206/311769 serialized/live
>>>>> bytes, 426 ops)
>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,896 Memtable.java (line
>>>>> 273) Completed flushing
>>>>> /var/lib/cassandra/data/MSA/modseq-h-262-Data.db (38646 bytes)
>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,897 Memtable.java (line
>>>>> 237) Writing Memtable-msgid@2099219781(3571249/77183008
>>>>> serialized/live bytes, 109823 ops)
>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:27,497 Memtable.java (line
>>>>> 273) Completed flushing /var/lib/cassandra/data/MSA/msgid-h-47-Data.db
>>>>> (8125165 bytes)
>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:27,498 Memtable.java (line
>>>>> 237) Writing Memtable-uid@578022704(43734344/317200563 serialized/live
>>>>> bytes, 611301 ops)
>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:29,802 Memtable.java (line
>>>>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-291-Data.db
>>>>> (48225128 bytes)
>>>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
>>>>> 488) Discarding obsolete commit
>>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125356477.log)
>>>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
>>>>> 488) Discarding obsolete commit
>>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125683351.log)
>>>>> INFO [MutationStage:88] 2011-10-20 10:52:29,905
>>>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>>>> Memtable-modseq@339630706(155217/3664394 serialized/live bytes, 5007
>>>>> ops)
>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:29,905 Memtable.java (line
>>>>> 237) Writing Memtable-modseq@339630706(155217/3664394 serialized/live
>>>>> bytes, 5007 ops)
>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:30,216 Memtable.java (line
>>>>> 273) Completed flushing
>>>>> /var/lib/cassandra/data/MSA/modseq-h-263-Data.db (450477 bytes)
>>>>> ERROR [CompactionExecutor:538] 2011-10-20 10:52:36,132
>>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>>> Thread[CompactionExecutor:538,1,main]
>>>>>
>>>>>
>>>>>
>>>>> Another one
>>>>>
>>>>>
>>>>>
>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:39,623 Memtable.java (line
>>>>> 237) Writing Memtable-uid@2018688194(79740/578345 serialized/live
>>>>> bytes, 1111 ops)
>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:39,777 Memtable.java (line
>>>>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-295-Data.db
>>>>> (142584 bytes)
>>>>> INFO [CompactionExecutor:544] 2011-10-20 10:52:39,778
>>>>> CompactionTask.java (line 119) Compacting
>>>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
>>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-294-Data.db')]
>>>>> ERROR [CompactionExecutor:544] 2011-10-20 10:52:39,935
>>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>>> Thread[CompactionExecutor:544,1,main]
>>>>> java.lang.AssertionError
>>>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
>>>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> INFO [MutationStage:89] 2011-10-20 10:52:46,599
>>>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>>>> Memtable-msgid@917568062(85287/1843250 serialized/live bytes, 2624
>>>>> ops)
>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:46,600 Memtable.java (line
>>>>> 237) Writing Memtable-msgid@917568062(85287/1843250 serialized/live
>>>>> bytes, 2624 ops)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:47,057 CommitLog.java (line
>>>>> 488) Discarding obsolete commit
>>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125959143.log)
>>>>> INFO [CompactionExecutor:544] 2011-10-20 10:52:47,058
>>>>> CompactionTask.java (line 119) Compacting
>>>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
>>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-298-Data.db')]
>>>>> INFO [CompactionExecutor:544] 2011-10-20 10:52:47,552
>>>>> CompactionTask.java (line 222) Compacted to
>>>>> [/var/lib/cassandra/data/MSA/uid-h-299-Data.db,]. 1,849,016 to
>>>>> 1,842,872 (~99% of original) bytes for 13,592 keys at 3.557692MBPS.
>>>>> Time: 494ms.
>>>>> ERROR [CompactionExecutor:539] 2011-10-20 10:52:49,901
>>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>>> Thread[CompactionExecutor:539,1,main]
>>>>> java.lang.AssertionError
>>>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
>>>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> ERROR [CompactionExecutor:542] 2011-10-20 10:52:51,983
>>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>>> Thread[CompactionExecutor:542,1,main]
>>>>> java.lang.IllegalArgumentException: Illegal Capacity: -3
>>>>> at java.util.ArrayList.<init>(ArrayList.java:110)
>>>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> INFO [CompactionExecutor:540] 2011-10-20 10:52:53,618
>>>>> CompactionTask.java (line 222) Compacted to
>>>>> [/var/lib/cassandra/data/MSA/participants-h-89-Data.db,]. 73,285,308
>>>>> to 54,342,740 (~74% of original) bytes for 223,836 keys at
>>>>> 3.594734MBPS. Time: 14,417ms.
>>>>> INFO [MutationStage:75] 2011-10-20 10:52:53,877
>>>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>>>> Memtable-participants@1814149120(143888/3251705 serialized/live bytes,
>>>>> 4232 ops)
>>>>> INFO [FlushWriter:222] 2011-10-20 10:52:53,878 Memtable.java (line
>>>>> 237) Writing Memtable-participants@1814149120(143888/3251705
>>>>> serialized/live bytes, 4232 ops)
>>>>>
>>>>>
>>>>>
>>>>> INFO [CompactionExecutor:542] 2011-10-20 10:52:54,313
>>>>> CompactionTask.java (line 119) Compacting
>>>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-93-Data.db'),
>>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-90-Data.db'),
>>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-92-Data.db'),
>>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-94-Data.db')]
>>>>> ERROR [CompactionExecutor:542] 2011-10-20 10:52:54,532
>>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>>> Thread[CompactionExecutor:542,1,main]
>>>>> java.lang.IllegalArgumentException: Illegal Capacity: -2
>>>>> at java.util.ArrayList.<init>(ArrayList.java:110)
>>>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> INFO [CompactionExecutor:539] 2011-10-20 10:52:54,787
>>>>> CompactionTask.java (line 222) Compacted to
>>>>> [/var/lib/cassandra/data/MSA/uid-h-303-Data.db,]. 3,479,848 to
>>>>> 3,418,524 (~98% of original) bytes for 24,789 keys at 5.388692MBPS.
>>>>> Time: 605ms.
>>>>> INFO [CompactionExecutor:541] 2011-10-20 10:52:54,806
>>>>> CompactionTask.java (line 222) Compacted to
>>>>> [/var/lib/cassandra/data/MSA/modseq-h-268-Data.db,]. 4,467,791 to
>>>>> 4,227,199 (~94% of original) bytes for 43,572 keys at 5.389534MBPS.
>>>>> Time: 748ms.
>>>>>
>>>>>
>>>>> thanks
>>>>> Ramesh
>>>>>
>>>>
>>>
>>
>
Re: Cassandra 1.0: Exception in compaction
Posted by Sylvain Lebresne <sy...@datastax.com>.
Also, to be sure, were you using the 1.0.0 final or some RC when getting this
exception?
On Fri, Oct 21, 2011 at 8:24 PM, Sylvain Lebresne <sy...@datastax.com> wrote:
> Would you have the full log for one of those node leading to the exception
> that you could share? Not sure that'll help but who knows.
>
> --
> Sylvain
>
> On Fri, Oct 21, 2011 at 4:34 PM, Ramesh Natarajan <ra...@gmail.com> wrote:
>> i am using size based compaction ( the default one ). Also this is on linux.
>>
>> thanks
>> Ramesh
>>
>> On Fri, Oct 21, 2011 at 4:25 AM, Sylvain Lebresne <sy...@datastax.com> wrote:
>>> I believe this is the same as
>>> https://issues.apache.org/jira/browse/CASSANDRA-3306.
>>>
>>> The initial reporter only got this exception with leveled compaction,
>>> is it what you are
>>> using too (to help narrow it down)? Also, are you using windows by any chance?
>>>
>>> --
>>> Sylvain
>>>
>>>
>>> On Thu, Oct 20, 2011 at 9:04 PM, Ramesh Natarajan <ra...@gmail.com> wrote:
>>>> We are running a 8 node cassandra 1.0 cluster. We are seeing this
>>>> exception quite often. Any idea how to debug this issue?
>>>>
>>>>
>>>> java.lang.IllegalArgumentException: Illegal Capacity: -2
>>>> at java.util.ArrayList.<init>(ArrayList.java:110)
>>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>>
>>>>
>>>> few lines before this error
>>>>
>>>> INFO [FlushWriter:222] 2011-10-20 10:52:25,885 Memtable.java (line
>>>> 237) Writing Memtable-participants@1907757288(6777526/153164388
>>>> serialized/live bytes, 199339 ops)
>>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
>>>> 488) Discarding obsolete commit
>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319115938691.log)
>>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
>>>> 488) Discarding obsolete commit
>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319122061956.log)
>>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,865 Memtable.java (line
>>>> 273) Completed flushing
>>>> /var/lib/cassandra/data/MSA/participants-h-87-Data.db (14695382 bytes)
>>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,866 Memtable.java (line
>>>> 237) Writing Memtable-modseq@1745889706(13206/311769 serialized/live
>>>> bytes, 426 ops)
>>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,896 Memtable.java (line
>>>> 273) Completed flushing
>>>> /var/lib/cassandra/data/MSA/modseq-h-262-Data.db (38646 bytes)
>>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,897 Memtable.java (line
>>>> 237) Writing Memtable-msgid@2099219781(3571249/77183008
>>>> serialized/live bytes, 109823 ops)
>>>> INFO [FlushWriter:222] 2011-10-20 10:52:27,497 Memtable.java (line
>>>> 273) Completed flushing /var/lib/cassandra/data/MSA/msgid-h-47-Data.db
>>>> (8125165 bytes)
>>>> INFO [FlushWriter:222] 2011-10-20 10:52:27,498 Memtable.java (line
>>>> 237) Writing Memtable-uid@578022704(43734344/317200563 serialized/live
>>>> bytes, 611301 ops)
>>>> INFO [FlushWriter:222] 2011-10-20 10:52:29,802 Memtable.java (line
>>>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-291-Data.db
>>>> (48225128 bytes)
>>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
>>>> 488) Discarding obsolete commit
>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125356477.log)
>>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
>>>> 488) Discarding obsolete commit
>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125683351.log)
>>>> INFO [MutationStage:88] 2011-10-20 10:52:29,905
>>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>>> Memtable-modseq@339630706(155217/3664394 serialized/live bytes, 5007
>>>> ops)
>>>> INFO [FlushWriter:222] 2011-10-20 10:52:29,905 Memtable.java (line
>>>> 237) Writing Memtable-modseq@339630706(155217/3664394 serialized/live
>>>> bytes, 5007 ops)
>>>> INFO [FlushWriter:222] 2011-10-20 10:52:30,216 Memtable.java (line
>>>> 273) Completed flushing
>>>> /var/lib/cassandra/data/MSA/modseq-h-263-Data.db (450477 bytes)
>>>> ERROR [CompactionExecutor:538] 2011-10-20 10:52:36,132
>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>> Thread[CompactionExecutor:538,1,main]
>>>>
>>>>
>>>>
>>>> Another one
>>>>
>>>>
>>>>
>>>> INFO [FlushWriter:222] 2011-10-20 10:52:39,623 Memtable.java (line
>>>> 237) Writing Memtable-uid@2018688194(79740/578345 serialized/live
>>>> bytes, 1111 ops)
>>>> INFO [FlushWriter:222] 2011-10-20 10:52:39,777 Memtable.java (line
>>>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-295-Data.db
>>>> (142584 bytes)
>>>> INFO [CompactionExecutor:544] 2011-10-20 10:52:39,778
>>>> CompactionTask.java (line 119) Compacting
>>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-294-Data.db')]
>>>> ERROR [CompactionExecutor:544] 2011-10-20 10:52:39,935
>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>> Thread[CompactionExecutor:544,1,main]
>>>> java.lang.AssertionError
>>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
>>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>> INFO [MutationStage:89] 2011-10-20 10:52:46,599
>>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>>> Memtable-msgid@917568062(85287/1843250 serialized/live bytes, 2624
>>>> ops)
>>>> INFO [FlushWriter:222] 2011-10-20 10:52:46,600 Memtable.java (line
>>>> 237) Writing Memtable-msgid@917568062(85287/1843250 serialized/live
>>>> bytes, 2624 ops)
>>>>
>>>>
>>>>
>>>>
>>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:47,057 CommitLog.java (line
>>>> 488) Discarding obsolete commit
>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125959143.log)
>>>> INFO [CompactionExecutor:544] 2011-10-20 10:52:47,058
>>>> CompactionTask.java (line 119) Compacting
>>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-298-Data.db')]
>>>> INFO [CompactionExecutor:544] 2011-10-20 10:52:47,552
>>>> CompactionTask.java (line 222) Compacted to
>>>> [/var/lib/cassandra/data/MSA/uid-h-299-Data.db,]. 1,849,016 to
>>>> 1,842,872 (~99% of original) bytes for 13,592 keys at 3.557692MBPS.
>>>> Time: 494ms.
>>>> ERROR [CompactionExecutor:539] 2011-10-20 10:52:49,901
>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>> Thread[CompactionExecutor:539,1,main]
>>>> java.lang.AssertionError
>>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
>>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>> ERROR [CompactionExecutor:542] 2011-10-20 10:52:51,983
>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>> Thread[CompactionExecutor:542,1,main]
>>>> java.lang.IllegalArgumentException: Illegal Capacity: -3
>>>> at java.util.ArrayList.<init>(ArrayList.java:110)
>>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>> INFO [CompactionExecutor:540] 2011-10-20 10:52:53,618
>>>> CompactionTask.java (line 222) Compacted to
>>>> [/var/lib/cassandra/data/MSA/participants-h-89-Data.db,]. 73,285,308
>>>> to 54,342,740 (~74% of original) bytes for 223,836 keys at
>>>> 3.594734MBPS. Time: 14,417ms.
>>>> INFO [MutationStage:75] 2011-10-20 10:52:53,877
>>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>>> Memtable-participants@1814149120(143888/3251705 serialized/live bytes,
>>>> 4232 ops)
>>>> INFO [FlushWriter:222] 2011-10-20 10:52:53,878 Memtable.java (line
>>>> 237) Writing Memtable-participants@1814149120(143888/3251705
>>>> serialized/live bytes, 4232 ops)
>>>>
>>>>
>>>>
>>>> INFO [CompactionExecutor:542] 2011-10-20 10:52:54,313
>>>> CompactionTask.java (line 119) Compacting
>>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-93-Data.db'),
>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-90-Data.db'),
>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-92-Data.db'),
>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-94-Data.db')]
>>>> ERROR [CompactionExecutor:542] 2011-10-20 10:52:54,532
>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>> Thread[CompactionExecutor:542,1,main]
>>>> java.lang.IllegalArgumentException: Illegal Capacity: -2
>>>> at java.util.ArrayList.<init>(ArrayList.java:110)
>>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>> INFO [CompactionExecutor:539] 2011-10-20 10:52:54,787
>>>> CompactionTask.java (line 222) Compacted to
>>>> [/var/lib/cassandra/data/MSA/uid-h-303-Data.db,]. 3,479,848 to
>>>> 3,418,524 (~98% of original) bytes for 24,789 keys at 5.388692MBPS.
>>>> Time: 605ms.
>>>> INFO [CompactionExecutor:541] 2011-10-20 10:52:54,806
>>>> CompactionTask.java (line 222) Compacted to
>>>> [/var/lib/cassandra/data/MSA/modseq-h-268-Data.db,]. 4,467,791 to
>>>> 4,227,199 (~94% of original) bytes for 43,572 keys at 5.389534MBPS.
>>>> Time: 748ms.
>>>>
>>>>
>>>> thanks
>>>> Ramesh
>>>>
>>>
>>
>
Re: Cassandra 1.0: Exception in compaction
Posted by Sylvain Lebresne <sy...@datastax.com>.
Would you have the full log for one of those node leading to the exception
that you could share? Not sure that'll help but who knows.
--
Sylvain
On Fri, Oct 21, 2011 at 4:34 PM, Ramesh Natarajan <ra...@gmail.com> wrote:
> i am using size based compaction ( the default one ). Also this is on linux.
>
> thanks
> Ramesh
>
> On Fri, Oct 21, 2011 at 4:25 AM, Sylvain Lebresne <sy...@datastax.com> wrote:
>> I believe this is the same as
>> https://issues.apache.org/jira/browse/CASSANDRA-3306.
>>
>> The initial reporter only got this exception with leveled compaction,
>> is it what you are
>> using too (to help narrow it down)? Also, are you using windows by any chance?
>>
>> --
>> Sylvain
>>
>>
>> On Thu, Oct 20, 2011 at 9:04 PM, Ramesh Natarajan <ra...@gmail.com> wrote:
>>> We are running a 8 node cassandra 1.0 cluster. We are seeing this
>>> exception quite often. Any idea how to debug this issue?
>>>
>>>
>>> java.lang.IllegalArgumentException: Illegal Capacity: -2
>>> at java.util.ArrayList.<init>(ArrayList.java:110)
>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>>
>>>
>>> few lines before this error
>>>
>>> INFO [FlushWriter:222] 2011-10-20 10:52:25,885 Memtable.java (line
>>> 237) Writing Memtable-participants@1907757288(6777526/153164388
>>> serialized/live bytes, 199339 ops)
>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
>>> 488) Discarding obsolete commit
>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319115938691.log)
>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
>>> 488) Discarding obsolete commit
>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319122061956.log)
>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,865 Memtable.java (line
>>> 273) Completed flushing
>>> /var/lib/cassandra/data/MSA/participants-h-87-Data.db (14695382 bytes)
>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,866 Memtable.java (line
>>> 237) Writing Memtable-modseq@1745889706(13206/311769 serialized/live
>>> bytes, 426 ops)
>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,896 Memtable.java (line
>>> 273) Completed flushing
>>> /var/lib/cassandra/data/MSA/modseq-h-262-Data.db (38646 bytes)
>>> INFO [FlushWriter:222] 2011-10-20 10:52:26,897 Memtable.java (line
>>> 237) Writing Memtable-msgid@2099219781(3571249/77183008
>>> serialized/live bytes, 109823 ops)
>>> INFO [FlushWriter:222] 2011-10-20 10:52:27,497 Memtable.java (line
>>> 273) Completed flushing /var/lib/cassandra/data/MSA/msgid-h-47-Data.db
>>> (8125165 bytes)
>>> INFO [FlushWriter:222] 2011-10-20 10:52:27,498 Memtable.java (line
>>> 237) Writing Memtable-uid@578022704(43734344/317200563 serialized/live
>>> bytes, 611301 ops)
>>> INFO [FlushWriter:222] 2011-10-20 10:52:29,802 Memtable.java (line
>>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-291-Data.db
>>> (48225128 bytes)
>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
>>> 488) Discarding obsolete commit
>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125356477.log)
>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
>>> 488) Discarding obsolete commit
>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125683351.log)
>>> INFO [MutationStage:88] 2011-10-20 10:52:29,905
>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>> Memtable-modseq@339630706(155217/3664394 serialized/live bytes, 5007
>>> ops)
>>> INFO [FlushWriter:222] 2011-10-20 10:52:29,905 Memtable.java (line
>>> 237) Writing Memtable-modseq@339630706(155217/3664394 serialized/live
>>> bytes, 5007 ops)
>>> INFO [FlushWriter:222] 2011-10-20 10:52:30,216 Memtable.java (line
>>> 273) Completed flushing
>>> /var/lib/cassandra/data/MSA/modseq-h-263-Data.db (450477 bytes)
>>> ERROR [CompactionExecutor:538] 2011-10-20 10:52:36,132
>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>> Thread[CompactionExecutor:538,1,main]
>>>
>>>
>>>
>>> Another one
>>>
>>>
>>>
>>> INFO [FlushWriter:222] 2011-10-20 10:52:39,623 Memtable.java (line
>>> 237) Writing Memtable-uid@2018688194(79740/578345 serialized/live
>>> bytes, 1111 ops)
>>> INFO [FlushWriter:222] 2011-10-20 10:52:39,777 Memtable.java (line
>>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-295-Data.db
>>> (142584 bytes)
>>> INFO [CompactionExecutor:544] 2011-10-20 10:52:39,778
>>> CompactionTask.java (line 119) Compacting
>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
>>> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-294-Data.db')]
>>> ERROR [CompactionExecutor:544] 2011-10-20 10:52:39,935
>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>> Thread[CompactionExecutor:544,1,main]
>>> java.lang.AssertionError
>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>> INFO [MutationStage:89] 2011-10-20 10:52:46,599
>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>> Memtable-msgid@917568062(85287/1843250 serialized/live bytes, 2624
>>> ops)
>>> INFO [FlushWriter:222] 2011-10-20 10:52:46,600 Memtable.java (line
>>> 237) Writing Memtable-msgid@917568062(85287/1843250 serialized/live
>>> bytes, 2624 ops)
>>>
>>>
>>>
>>>
>>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:47,057 CommitLog.java (line
>>> 488) Discarding obsolete commit
>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125959143.log)
>>> INFO [CompactionExecutor:544] 2011-10-20 10:52:47,058
>>> CompactionTask.java (line 119) Compacting
>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
>>> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-298-Data.db')]
>>> INFO [CompactionExecutor:544] 2011-10-20 10:52:47,552
>>> CompactionTask.java (line 222) Compacted to
>>> [/var/lib/cassandra/data/MSA/uid-h-299-Data.db,]. 1,849,016 to
>>> 1,842,872 (~99% of original) bytes for 13,592 keys at 3.557692MBPS.
>>> Time: 494ms.
>>> ERROR [CompactionExecutor:539] 2011-10-20 10:52:49,901
>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>> Thread[CompactionExecutor:539,1,main]
>>> java.lang.AssertionError
>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>> ERROR [CompactionExecutor:542] 2011-10-20 10:52:51,983
>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>> Thread[CompactionExecutor:542,1,main]
>>> java.lang.IllegalArgumentException: Illegal Capacity: -3
>>> at java.util.ArrayList.<init>(ArrayList.java:110)
>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>> INFO [CompactionExecutor:540] 2011-10-20 10:52:53,618
>>> CompactionTask.java (line 222) Compacted to
>>> [/var/lib/cassandra/data/MSA/participants-h-89-Data.db,]. 73,285,308
>>> to 54,342,740 (~74% of original) bytes for 223,836 keys at
>>> 3.594734MBPS. Time: 14,417ms.
>>> INFO [MutationStage:75] 2011-10-20 10:52:53,877
>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>> Memtable-participants@1814149120(143888/3251705 serialized/live bytes,
>>> 4232 ops)
>>> INFO [FlushWriter:222] 2011-10-20 10:52:53,878 Memtable.java (line
>>> 237) Writing Memtable-participants@1814149120(143888/3251705
>>> serialized/live bytes, 4232 ops)
>>>
>>>
>>>
>>> INFO [CompactionExecutor:542] 2011-10-20 10:52:54,313
>>> CompactionTask.java (line 119) Compacting
>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-93-Data.db'),
>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-90-Data.db'),
>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-92-Data.db'),
>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-94-Data.db')]
>>> ERROR [CompactionExecutor:542] 2011-10-20 10:52:54,532
>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>> Thread[CompactionExecutor:542,1,main]
>>> java.lang.IllegalArgumentException: Illegal Capacity: -2
>>> at java.util.ArrayList.<init>(ArrayList.java:110)
>>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>> INFO [CompactionExecutor:539] 2011-10-20 10:52:54,787
>>> CompactionTask.java (line 222) Compacted to
>>> [/var/lib/cassandra/data/MSA/uid-h-303-Data.db,]. 3,479,848 to
>>> 3,418,524 (~98% of original) bytes for 24,789 keys at 5.388692MBPS.
>>> Time: 605ms.
>>> INFO [CompactionExecutor:541] 2011-10-20 10:52:54,806
>>> CompactionTask.java (line 222) Compacted to
>>> [/var/lib/cassandra/data/MSA/modseq-h-268-Data.db,]. 4,467,791 to
>>> 4,227,199 (~94% of original) bytes for 43,572 keys at 5.389534MBPS.
>>> Time: 748ms.
>>>
>>>
>>> thanks
>>> Ramesh
>>>
>>
>
Re: Cassandra 1.0: Exception in compaction
Posted by Ramesh Natarajan <ra...@gmail.com>.
i am using size based compaction ( the default one ). Also this is on linux.
thanks
Ramesh
On Fri, Oct 21, 2011 at 4:25 AM, Sylvain Lebresne <sy...@datastax.com> wrote:
> I believe this is the same as
> https://issues.apache.org/jira/browse/CASSANDRA-3306.
>
> The initial reporter only got this exception with leveled compaction,
> is it what you are
> using too (to help narrow it down)? Also, are you using windows by any chance?
>
> --
> Sylvain
>
>
> On Thu, Oct 20, 2011 at 9:04 PM, Ramesh Natarajan <ra...@gmail.com> wrote:
>> We are running a 8 node cassandra 1.0 cluster. We are seeing this
>> exception quite often. Any idea how to debug this issue?
>>
>>
>> java.lang.IllegalArgumentException: Illegal Capacity: -2
>> at java.util.ArrayList.<init>(ArrayList.java:110)
>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>>
>>
>> few lines before this error
>>
>> INFO [FlushWriter:222] 2011-10-20 10:52:25,885 Memtable.java (line
>> 237) Writing Memtable-participants@1907757288(6777526/153164388
>> serialized/live bytes, 199339 ops)
>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
>> 488) Discarding obsolete commit
>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319115938691.log)
>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
>> 488) Discarding obsolete commit
>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319122061956.log)
>> INFO [FlushWriter:222] 2011-10-20 10:52:26,865 Memtable.java (line
>> 273) Completed flushing
>> /var/lib/cassandra/data/MSA/participants-h-87-Data.db (14695382 bytes)
>> INFO [FlushWriter:222] 2011-10-20 10:52:26,866 Memtable.java (line
>> 237) Writing Memtable-modseq@1745889706(13206/311769 serialized/live
>> bytes, 426 ops)
>> INFO [FlushWriter:222] 2011-10-20 10:52:26,896 Memtable.java (line
>> 273) Completed flushing
>> /var/lib/cassandra/data/MSA/modseq-h-262-Data.db (38646 bytes)
>> INFO [FlushWriter:222] 2011-10-20 10:52:26,897 Memtable.java (line
>> 237) Writing Memtable-msgid@2099219781(3571249/77183008
>> serialized/live bytes, 109823 ops)
>> INFO [FlushWriter:222] 2011-10-20 10:52:27,497 Memtable.java (line
>> 273) Completed flushing /var/lib/cassandra/data/MSA/msgid-h-47-Data.db
>> (8125165 bytes)
>> INFO [FlushWriter:222] 2011-10-20 10:52:27,498 Memtable.java (line
>> 237) Writing Memtable-uid@578022704(43734344/317200563 serialized/live
>> bytes, 611301 ops)
>> INFO [FlushWriter:222] 2011-10-20 10:52:29,802 Memtable.java (line
>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-291-Data.db
>> (48225128 bytes)
>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
>> 488) Discarding obsolete commit
>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125356477.log)
>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
>> 488) Discarding obsolete commit
>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125683351.log)
>> INFO [MutationStage:88] 2011-10-20 10:52:29,905
>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>> Memtable-modseq@339630706(155217/3664394 serialized/live bytes, 5007
>> ops)
>> INFO [FlushWriter:222] 2011-10-20 10:52:29,905 Memtable.java (line
>> 237) Writing Memtable-modseq@339630706(155217/3664394 serialized/live
>> bytes, 5007 ops)
>> INFO [FlushWriter:222] 2011-10-20 10:52:30,216 Memtable.java (line
>> 273) Completed flushing
>> /var/lib/cassandra/data/MSA/modseq-h-263-Data.db (450477 bytes)
>> ERROR [CompactionExecutor:538] 2011-10-20 10:52:36,132
>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>> Thread[CompactionExecutor:538,1,main]
>>
>>
>>
>> Another one
>>
>>
>>
>> INFO [FlushWriter:222] 2011-10-20 10:52:39,623 Memtable.java (line
>> 237) Writing Memtable-uid@2018688194(79740/578345 serialized/live
>> bytes, 1111 ops)
>> INFO [FlushWriter:222] 2011-10-20 10:52:39,777 Memtable.java (line
>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-295-Data.db
>> (142584 bytes)
>> INFO [CompactionExecutor:544] 2011-10-20 10:52:39,778
>> CompactionTask.java (line 119) Compacting
>> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
>> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-294-Data.db')]
>> ERROR [CompactionExecutor:544] 2011-10-20 10:52:39,935
>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>> Thread[CompactionExecutor:544,1,main]
>> java.lang.AssertionError
>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> INFO [MutationStage:89] 2011-10-20 10:52:46,599
>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>> Memtable-msgid@917568062(85287/1843250 serialized/live bytes, 2624
>> ops)
>> INFO [FlushWriter:222] 2011-10-20 10:52:46,600 Memtable.java (line
>> 237) Writing Memtable-msgid@917568062(85287/1843250 serialized/live
>> bytes, 2624 ops)
>>
>>
>>
>>
>> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:47,057 CommitLog.java (line
>> 488) Discarding obsolete commit
>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125959143.log)
>> INFO [CompactionExecutor:544] 2011-10-20 10:52:47,058
>> CompactionTask.java (line 119) Compacting
>> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
>> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-298-Data.db')]
>> INFO [CompactionExecutor:544] 2011-10-20 10:52:47,552
>> CompactionTask.java (line 222) Compacted to
>> [/var/lib/cassandra/data/MSA/uid-h-299-Data.db,]. 1,849,016 to
>> 1,842,872 (~99% of original) bytes for 13,592 keys at 3.557692MBPS.
>> Time: 494ms.
>> ERROR [CompactionExecutor:539] 2011-10-20 10:52:49,901
>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>> Thread[CompactionExecutor:539,1,main]
>> java.lang.AssertionError
>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> ERROR [CompactionExecutor:542] 2011-10-20 10:52:51,983
>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>> Thread[CompactionExecutor:542,1,main]
>> java.lang.IllegalArgumentException: Illegal Capacity: -3
>> at java.util.ArrayList.<init>(ArrayList.java:110)
>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> INFO [CompactionExecutor:540] 2011-10-20 10:52:53,618
>> CompactionTask.java (line 222) Compacted to
>> [/var/lib/cassandra/data/MSA/participants-h-89-Data.db,]. 73,285,308
>> to 54,342,740 (~74% of original) bytes for 223,836 keys at
>> 3.594734MBPS. Time: 14,417ms.
>> INFO [MutationStage:75] 2011-10-20 10:52:53,877
>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>> Memtable-participants@1814149120(143888/3251705 serialized/live bytes,
>> 4232 ops)
>> INFO [FlushWriter:222] 2011-10-20 10:52:53,878 Memtable.java (line
>> 237) Writing Memtable-participants@1814149120(143888/3251705
>> serialized/live bytes, 4232 ops)
>>
>>
>>
>> INFO [CompactionExecutor:542] 2011-10-20 10:52:54,313
>> CompactionTask.java (line 119) Compacting
>> [SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-93-Data.db'),
>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-90-Data.db'),
>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-92-Data.db'),
>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-94-Data.db')]
>> ERROR [CompactionExecutor:542] 2011-10-20 10:52:54,532
>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>> Thread[CompactionExecutor:542,1,main]
>> java.lang.IllegalArgumentException: Illegal Capacity: -2
>> at java.util.ArrayList.<init>(ArrayList.java:110)
>> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> INFO [CompactionExecutor:539] 2011-10-20 10:52:54,787
>> CompactionTask.java (line 222) Compacted to
>> [/var/lib/cassandra/data/MSA/uid-h-303-Data.db,]. 3,479,848 to
>> 3,418,524 (~98% of original) bytes for 24,789 keys at 5.388692MBPS.
>> Time: 605ms.
>> INFO [CompactionExecutor:541] 2011-10-20 10:52:54,806
>> CompactionTask.java (line 222) Compacted to
>> [/var/lib/cassandra/data/MSA/modseq-h-268-Data.db,]. 4,467,791 to
>> 4,227,199 (~94% of original) bytes for 43,572 keys at 5.389534MBPS.
>> Time: 748ms.
>>
>>
>> thanks
>> Ramesh
>>
>
Re: Cassandra 1.0: Exception in compaction
Posted by Sylvain Lebresne <sy...@datastax.com>.
I believe this is the same as
https://issues.apache.org/jira/browse/CASSANDRA-3306.
The initial reporter only got this exception with leveled compaction,
is it what you are
using too (to help narrow it down)? Also, are you using windows by any chance?
--
Sylvain
On Thu, Oct 20, 2011 at 9:04 PM, Ramesh Natarajan <ra...@gmail.com> wrote:
> We are running a 8 node cassandra 1.0 cluster. We are seeing this
> exception quite often. Any idea how to debug this issue?
>
>
> java.lang.IllegalArgumentException: Illegal Capacity: -2
> at java.util.ArrayList.<init>(ArrayList.java:110)
> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
>
>
> few lines before this error
>
> INFO [FlushWriter:222] 2011-10-20 10:52:25,885 Memtable.java (line
> 237) Writing Memtable-participants@1907757288(6777526/153164388
> serialized/live bytes, 199339 ops)
> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
> 488) Discarding obsolete commit
> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319115938691.log)
> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line
> 488) Discarding obsolete commit
> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319122061956.log)
> INFO [FlushWriter:222] 2011-10-20 10:52:26,865 Memtable.java (line
> 273) Completed flushing
> /var/lib/cassandra/data/MSA/participants-h-87-Data.db (14695382 bytes)
> INFO [FlushWriter:222] 2011-10-20 10:52:26,866 Memtable.java (line
> 237) Writing Memtable-modseq@1745889706(13206/311769 serialized/live
> bytes, 426 ops)
> INFO [FlushWriter:222] 2011-10-20 10:52:26,896 Memtable.java (line
> 273) Completed flushing
> /var/lib/cassandra/data/MSA/modseq-h-262-Data.db (38646 bytes)
> INFO [FlushWriter:222] 2011-10-20 10:52:26,897 Memtable.java (line
> 237) Writing Memtable-msgid@2099219781(3571249/77183008
> serialized/live bytes, 109823 ops)
> INFO [FlushWriter:222] 2011-10-20 10:52:27,497 Memtable.java (line
> 273) Completed flushing /var/lib/cassandra/data/MSA/msgid-h-47-Data.db
> (8125165 bytes)
> INFO [FlushWriter:222] 2011-10-20 10:52:27,498 Memtable.java (line
> 237) Writing Memtable-uid@578022704(43734344/317200563 serialized/live
> bytes, 611301 ops)
> INFO [FlushWriter:222] 2011-10-20 10:52:29,802 Memtable.java (line
> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-291-Data.db
> (48225128 bytes)
> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
> 488) Discarding obsolete commit
> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125356477.log)
> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line
> 488) Discarding obsolete commit
> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125683351.log)
> INFO [MutationStage:88] 2011-10-20 10:52:29,905
> ColumnFamilyStore.java (line 664) Enqueuing flush of
> Memtable-modseq@339630706(155217/3664394 serialized/live bytes, 5007
> ops)
> INFO [FlushWriter:222] 2011-10-20 10:52:29,905 Memtable.java (line
> 237) Writing Memtable-modseq@339630706(155217/3664394 serialized/live
> bytes, 5007 ops)
> INFO [FlushWriter:222] 2011-10-20 10:52:30,216 Memtable.java (line
> 273) Completed flushing
> /var/lib/cassandra/data/MSA/modseq-h-263-Data.db (450477 bytes)
> ERROR [CompactionExecutor:538] 2011-10-20 10:52:36,132
> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
> Thread[CompactionExecutor:538,1,main]
>
>
>
> Another one
>
>
>
> INFO [FlushWriter:222] 2011-10-20 10:52:39,623 Memtable.java (line
> 237) Writing Memtable-uid@2018688194(79740/578345 serialized/live
> bytes, 1111 ops)
> INFO [FlushWriter:222] 2011-10-20 10:52:39,777 Memtable.java (line
> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-295-Data.db
> (142584 bytes)
> INFO [CompactionExecutor:544] 2011-10-20 10:52:39,778
> CompactionTask.java (line 119) Compacting
> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-294-Data.db')]
> ERROR [CompactionExecutor:544] 2011-10-20 10:52:39,935
> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
> Thread[CompactionExecutor:544,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [MutationStage:89] 2011-10-20 10:52:46,599
> ColumnFamilyStore.java (line 664) Enqueuing flush of
> Memtable-msgid@917568062(85287/1843250 serialized/live bytes, 2624
> ops)
> INFO [FlushWriter:222] 2011-10-20 10:52:46,600 Memtable.java (line
> 237) Writing Memtable-msgid@917568062(85287/1843250 serialized/live
> bytes, 2624 ops)
>
>
>
>
> INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:47,057 CommitLog.java (line
> 488) Discarding obsolete commit
> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125959143.log)
> INFO [CompactionExecutor:544] 2011-10-20 10:52:47,058
> CompactionTask.java (line 119) Compacting
> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-298-Data.db')]
> INFO [CompactionExecutor:544] 2011-10-20 10:52:47,552
> CompactionTask.java (line 222) Compacted to
> [/var/lib/cassandra/data/MSA/uid-h-299-Data.db,]. 1,849,016 to
> 1,842,872 (~99% of original) bytes for 13,592 keys at 3.557692MBPS.
> Time: 494ms.
> ERROR [CompactionExecutor:539] 2011-10-20 10:52:49,901
> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
> Thread[CompactionExecutor:539,1,main]
> java.lang.AssertionError
> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [CompactionExecutor:542] 2011-10-20 10:52:51,983
> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
> Thread[CompactionExecutor:542,1,main]
> java.lang.IllegalArgumentException: Illegal Capacity: -3
> at java.util.ArrayList.<init>(ArrayList.java:110)
> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:540] 2011-10-20 10:52:53,618
> CompactionTask.java (line 222) Compacted to
> [/var/lib/cassandra/data/MSA/participants-h-89-Data.db,]. 73,285,308
> to 54,342,740 (~74% of original) bytes for 223,836 keys at
> 3.594734MBPS. Time: 14,417ms.
> INFO [MutationStage:75] 2011-10-20 10:52:53,877
> ColumnFamilyStore.java (line 664) Enqueuing flush of
> Memtable-participants@1814149120(143888/3251705 serialized/live bytes,
> 4232 ops)
> INFO [FlushWriter:222] 2011-10-20 10:52:53,878 Memtable.java (line
> 237) Writing Memtable-participants@1814149120(143888/3251705
> serialized/live bytes, 4232 ops)
>
>
>
> INFO [CompactionExecutor:542] 2011-10-20 10:52:54,313
> CompactionTask.java (line 119) Compacting
> [SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-93-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-90-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-92-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-94-Data.db')]
> ERROR [CompactionExecutor:542] 2011-10-20 10:52:54,532
> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
> Thread[CompactionExecutor:542,1,main]
> java.lang.IllegalArgumentException: Illegal Capacity: -2
> at java.util.ArrayList.<init>(ArrayList.java:110)
> at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
> at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
> at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
> at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:539] 2011-10-20 10:52:54,787
> CompactionTask.java (line 222) Compacted to
> [/var/lib/cassandra/data/MSA/uid-h-303-Data.db,]. 3,479,848 to
> 3,418,524 (~98% of original) bytes for 24,789 keys at 5.388692MBPS.
> Time: 605ms.
> INFO [CompactionExecutor:541] 2011-10-20 10:52:54,806
> CompactionTask.java (line 222) Compacted to
> [/var/lib/cassandra/data/MSA/modseq-h-268-Data.db,]. 4,467,791 to
> 4,227,199 (~94% of original) bytes for 43,572 keys at 5.389534MBPS.
> Time: 748ms.
>
>
> thanks
> Ramesh
>