You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Mateusz Korniak <ma...@ant.gliwice.pl> on 2011/03/18 12:44:49 UTC
[0.7.2] Compacting exception
Hi !
I have run with Cassandra 0.7.2 out of disc space and after moving to bigger
partition I experience compaction failures[1].
1) I suspect one of SSTables is broken. If I am right how can I find which one
exactly ?
2) Knowing which one is broken is it safe to stop Cassandra, remove
-Data.db -Filter.db -Index.db -Statistics.db of broken SSTable and restart ?
Sure I will loose data but it's one of RF=3 nodes so it's not big problem.
Any suggestions, hints ?
Thanks in advance, regards,
[1]
INFO [CompactionExecutor:1] 2011-03-18 12:27:17,103 CompactionManager.java
(line 458) Compacted to /var/lib/cassandra/data/foo/bar-tmp-f-707-Data.db.
42,276,185 to 42,103,628 (~99% of original) bytes for 33,869 keys. Time:
25,924ms.
INFO [CompactionExecutor:1] 2011-03-18 12:27:17,133 CompactionManager.java
(line 373) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path=
'/var/lib/cassandra/data/foo/bar-f-238-Data.db'),
org.apache.cassandra.io.sstable.SSTableReader(
path='/var/lib/cassandra/data/foo/bar-f-243-Data.db'),
org.apache.cassandra.io.sstable.SSTableReader(
path='/var/lib/cassandra/data/foo/bar-f-361-Data.db'),
org.apache.cassandra.io.sstable.SSTableReader(
path='/var/lib/cassandra/data/foo/bar-f-635-Data.db'),
org.apache.cassandra.io.sstable.SSTableReader(
path='/var/lib/cassandra/data/foo/bar-f-643-Data.db'),
org.apache.cassandra.io.sstable.SSTableReader(
path='/var/lib/cassandra/data/foo/bar-f-680-Data.db'),
org.apache.cassandra.io.sstable.SSTableReader(
path='/var/lib/cassandra/data/foo/bar-f-684-Data.db'),
org.apache.cassandra.io.sstable.SSTableReader(
path='/var/lib/cassandra/data/foo/bar-f-685-Data.db'),
org.apache.cassandra.io.sstable.SSTableReader(
path='/var/lib/cassandra/data/foo/bar-f-687-Data.db')]
ERROR [CompactionExecutor:1] 2011-03-18 12:27:18,241
AbstractCassandraDaemon.java (line 114) Fatal exception in thread
Thread[CompactionExecutor:1,1,main]
java.io.IOError: java.io.EOFException
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:78)
at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179)
at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144)
at
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136)
at
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:39)
at
org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
at
org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
at
org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:427)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:123)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:93)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at
org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:70)
... 20 more
--
Mateusz Korniak
Re: [0.7.2] Compacting exception
Posted by Mateusz Korniak <ma...@ant.gliwice.pl>.
On Friday 18 of March 2011, Jonathan Ellis wrote:
> Upgrade to 0.7.4 and run nodetool scrub (then watch the log).
Unfortunately nodetool scrub fails with:
WARN [CompactionExecutor:1] 2011-03-19 00:34:53,511 CompactionManager.java
(line 607) Non-fatal error reading row (stacktrace follows)
java.io.IOError: java.io.IOException: Impossible row size 3458764513820542976
at
org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
at
org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
at
org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Impossible row size 3458764513820542976
... 8 more
INFO [CompactionExecutor:1] 2011-03-19 00:34:53,512 CompactionManager.java
(line 613) Retrying from row index; data is 10296 bytes starting at 2648761
ERROR [CompactionExecutor:1] 2011-03-19 00:34:53,512
AbstractCassandraDaemon.java (line 112) Fatal exception in thread
Thread[CompactionExecutor:1,1,main]
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at
org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilterSerializer.java:49)
at
org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilterSerializer.java:30)
at
org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:117)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
at
org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
at
org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
at
org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
> If it
> reports bad rows then run repair post-scrub.
You mean:
nodetool repair
?
Thanks for help, regards,
>
> On Fri, Mar 18, 2011 at 6:44 AM, Mateusz Korniak
>
> <ma...@ant.gliwice.pl> wrote:
> > Hi !
> > I have run with Cassandra 0.7.2 out of disc space and after moving to
> > bigger partition I experience compaction failures[1].
> >
> > 1) I suspect one of SSTables is broken. If I am right how can I find
> > which one exactly ?
> >
> > 2) Knowing which one is broken is it safe to stop Cassandra, remove
> > -Data.db -Filter.db -Index.db -Statistics.db of broken SSTable and
> > restart ? Sure I will loose data but it's one of RF=3 nodes so it's not
> > big problem.
> >
> > Any suggestions, hints ?
> > Thanks in advance, regards,
> >
> > [1]
> > INFO [CompactionExecutor:1] 2011-03-18 12:27:17,103
> > CompactionManager.java (line 458) Compacted to
> > /var/lib/cassandra/data/foo/bar-tmp-f-707-Data.db. 42,276,185 to
> > 42,103,628 (~99% of original) bytes for 33,869 keys. Time: 25,924ms.
> > INFO [CompactionExecutor:1] 2011-03-18 12:27:17,133
> > CompactionManager.java (line 373) Compacting
> > [org.apache.cassandra.io.sstable.SSTableReader(path=
> > '/var/lib/cassandra/data/foo/bar-f-238-Data.db'),
> > org.apache.cassandra.io.sstable.SSTableReader(
> > path='/var/lib/cassandra/data/foo/bar-f-243-Data.db'),
> > org.apache.cassandra.io.sstable.SSTableReader(
> > path='/var/lib/cassandra/data/foo/bar-f-361-Data.db'),
> > org.apache.cassandra.io.sstable.SSTableReader(
> > path='/var/lib/cassandra/data/foo/bar-f-635-Data.db'),
> > org.apache.cassandra.io.sstable.SSTableReader(
> > path='/var/lib/cassandra/data/foo/bar-f-643-Data.db'),
> > org.apache.cassandra.io.sstable.SSTableReader(
> > path='/var/lib/cassandra/data/foo/bar-f-680-Data.db'),
> > org.apache.cassandra.io.sstable.SSTableReader(
> > path='/var/lib/cassandra/data/foo/bar-f-684-Data.db'),
> > org.apache.cassandra.io.sstable.SSTableReader(
> > path='/var/lib/cassandra/data/foo/bar-f-685-Data.db'),
> > org.apache.cassandra.io.sstable.SSTableReader(
> > path='/var/lib/cassandra/data/foo/bar-f-687-Data.db')]
> > ERROR [CompactionExecutor:1] 2011-03-18 12:27:18,241
> > AbstractCassandraDaemon.java (line 114) Fatal exception in thread
> > Thread[CompactionExecutor:1,1,main]
> > java.io.IOError: java.io.EOFException
> > at
> > org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIde
> > ntityIterator.java:78) at
> > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(S
> > STableScanner.java:179) at
> > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(S
> > STableScanner.java:144) at
> > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:1
> > 36) at
> > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:3
> > 9) at
> > org.apache.commons.collections.iterators.CollatingIterator.set(CollatingI
> > terator.java:284) at
> > org.apache.commons.collections.iterators.CollatingIterator.least(Collatin
> > gIterator.java:326) at
> > org.apache.commons.collections.iterators.CollatingIterator.next(Collating
> > Iterator.java:230) at
> > org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.
> > java:68) at
> > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractItera
> > tor.java:136) at
> > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:
> > 131) at
> > org.apache.commons.collections.iterators.FilterIterator.setNextObject(Fil
> > terIterator.java:183) at
> > org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIte
> > rator.java:94) at
> > org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.
> > java:427) at
> > org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:1
> > 23) at
> > org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:9
> > 3) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138) at
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor
> > .java:886) at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
> > a:908) at java.lang.Thread.run(Thread.java:662)
> > Caused by: java.io.EOFException
> > at
> > org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65
> > ) at
> > org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIde
> > ntityIterator.java:70) ... 20 more
> >
> >
> > --
> > Mateusz Korniak
--
Mateusz Korniak
Re: [0.7.2] Compacting exception
Posted by Jonathan Ellis <jb...@gmail.com>.
Upgrade to 0.7.4 and run nodetool scrub (then watch the log). If it
reports bad rows then run repair post-scrub.
On Fri, Mar 18, 2011 at 6:44 AM, Mateusz Korniak
<ma...@ant.gliwice.pl> wrote:
> Hi !
> I have run with Cassandra 0.7.2 out of disc space and after moving to bigger
> partition I experience compaction failures[1].
>
> 1) I suspect one of SSTables is broken. If I am right how can I find which one
> exactly ?
>
> 2) Knowing which one is broken is it safe to stop Cassandra, remove
> -Data.db -Filter.db -Index.db -Statistics.db of broken SSTable and restart ?
> Sure I will loose data but it's one of RF=3 nodes so it's not big problem.
>
> Any suggestions, hints ?
> Thanks in advance, regards,
>
> [1]
> INFO [CompactionExecutor:1] 2011-03-18 12:27:17,103 CompactionManager.java
> (line 458) Compacted to /var/lib/cassandra/data/foo/bar-tmp-f-707-Data.db.
> 42,276,185 to 42,103,628 (~99% of original) bytes for 33,869 keys. Time:
> 25,924ms.
> INFO [CompactionExecutor:1] 2011-03-18 12:27:17,133 CompactionManager.java
> (line 373) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path=
> '/var/lib/cassandra/data/foo/bar-f-238-Data.db'),
> org.apache.cassandra.io.sstable.SSTableReader(
> path='/var/lib/cassandra/data/foo/bar-f-243-Data.db'),
> org.apache.cassandra.io.sstable.SSTableReader(
> path='/var/lib/cassandra/data/foo/bar-f-361-Data.db'),
> org.apache.cassandra.io.sstable.SSTableReader(
> path='/var/lib/cassandra/data/foo/bar-f-635-Data.db'),
> org.apache.cassandra.io.sstable.SSTableReader(
> path='/var/lib/cassandra/data/foo/bar-f-643-Data.db'),
> org.apache.cassandra.io.sstable.SSTableReader(
> path='/var/lib/cassandra/data/foo/bar-f-680-Data.db'),
> org.apache.cassandra.io.sstable.SSTableReader(
> path='/var/lib/cassandra/data/foo/bar-f-684-Data.db'),
> org.apache.cassandra.io.sstable.SSTableReader(
> path='/var/lib/cassandra/data/foo/bar-f-685-Data.db'),
> org.apache.cassandra.io.sstable.SSTableReader(
> path='/var/lib/cassandra/data/foo/bar-f-687-Data.db')]
> ERROR [CompactionExecutor:1] 2011-03-18 12:27:18,241
> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
> Thread[CompactionExecutor:1,1,main]
> java.io.IOError: java.io.EOFException
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:78)
> at
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179)
> at
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:39)
> at
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> at
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> at
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:427)
> at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:123)
> at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:93)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.EOFException
> at
> org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:70)
> ... 20 more
>
>
> --
> Mateusz Korniak
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com