Re: How to repair HintsColumnFamily?

[Terje Marthinussen <tm...@gmail.com>]

Seeing similar errors on another system (0.7.4). Maybe something bogus with
the hint columnfamilies.

Terje

On Mon, Mar 28, 2011 at 7:15 PM, Shotaro Kamio <ka...@gmail.com> wrote:

> I see. Then, I'll remove the HintsColumnFamily.
>
> Because our cluster has a lot of data, running repair takes much time
> (more than a day). And it's a kind of pain. It often causes disk full,
> creates many sstables and degrades read performance.
> If it's easy to fix the hint, it could be less painful solution. But I
> understand there's no other option in this case.
>
>
> Thanks,
> Shotaro
>
>
> On Sun, Mar 27, 2011 at 11:51 PM, Jonathan Ellis <jb...@gmail.com>
> wrote:
> > Why would you try to repair hints?
> >
> > If you run repair on the non-system data then you don't need the hint
> > data and can remove it.
> >
> > On Sun, Mar 27, 2011 at 12:17 AM, Shotaro Kamio <ka...@gmail.com>
> wrote:
> >> Hi,
> >>
> >> Our cluster uses cassandra 0.7.4 (upgraded from 0.7.3) with
> >> replication = 3. I found that error occurs on one node during hinted
> >> handoff with following error (log #1 below).
> >> When I tried out "scrub system HintsColumnFamily", I saw an ERROR in
> >> log (log #2 below).
> >> Do you think these errors are critical ?
> >> I tried to "repair system HintsColumnFamily". But, it refuses to run
> >> with "No neighbors". I can understand because hints are not
> >> replicated. But then, is there any way to fix it without data loss?
> >>
> >>  INFO [manual-repair-0996a2ec-26d3-4243-9586-d56daf30f9bd] 2011-03-27
> >> 13:55:05,664 AntiEntropyService.java (line 752) No neighbors to repair
> >> with: manual-repair-0996a2ec-26d3-4243-9586-d56daf30f9bd completed.
> >>
> >>
> >> Best regards,
> >> Shotaro
> >>
> >>
> >> ---------------- Log #1: Error on hinted handoff
> >> ------------------------------------------------
> >>
> >> ERROR [HintedHandoff:1] 2011-03-26 20:04:22,528
> >> DebuggableThreadPoolExecutor.java (line 103) Error in
> >> ThreadPoolExecutor
> >> java.lang.RuntimeException: java.lang.RuntimeException: error reading
> >> 4976040 of 4976067
> >>        at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> >>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>        at java.lang.Thread.run(Thread.java:662)
> >> Caused by: java.lang.RuntimeException: error reading 4976040 of 4976067
> >>        at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
> >>        at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
> >>        at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >>        at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >>        at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
> >>        at
> org.apache.commons.collections.iterators.CollatingIterator.anyHasNext(CollatingIterator.java:364)
> >>        at
> org.apache.commons.collections.iterators.CollatingIterator.hasNext(CollatingIterator.java:217)
> >>        at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:63)
> >>        at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >>        at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >>        at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> >>        at
> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
> >>        at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
> >>        at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
> >>        at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
> >>        at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:321)
> >>        at
> org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
> >>        at
> org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:409)
> >>        at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> >>        ... 3 more
> >> Caused by: java.io.EOFException
> >>        at java.io.RandomAccessFile.readByte(RandomAccessFile.java:591)
> >>        at
> org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:324)
> >>        at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:335)
> >>        at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:351)
> >>        at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:311)
> >>        at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
> >>        ... 21 more
> >>
> >> ------------------------------------------
> >>
> >> ---- Log #2: Error on scrub -------------------
> >>
> >>  INFO [CompactionExecutor:1] 2011-03-27 08:07:34,527
> >> CompactionManager.java (line 512) Scrubbing
> >>
> SSTableReader(path='/data/cassandra/system/HintsColumnFamily-f-530-Data.db')
> >>  WARN [CompactionExecutor:1] 2011-03-27 08:07:34,602
> >> CompactionManager.java (line 607) Non-fatal error reading row
> >> (stacktrace follows)
> >> java.io.IOError: java.io.IOException: Impossible row size 406136901
> >>        at
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
> >>        at
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >>        at
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >>        at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>        at java.lang.Thread.run(Thread.java:662)
> >> Caused by: java.io.IOException: Impossible row size 406136901
> >>        ... 8 more
> >>  INFO [CompactionExecutor:1] 2011-03-27 08:07:34,602
> >> CompactionManager.java (line 613) Retrying from row index; data is
> >> 406134739 bytes starting at 21
> >>  INFO [CompactionExecutor:1] 2011-03-27 08:08:10,065
> >> CompactionManager.java (line 653) Scrub of
> >>
> SSTableReader(path='/data/cassandra/system/HintsColumnFamily-f-530-Data.db')
> >> complete: 1 rows in new sstable and 0 empty (tombstoned) rows dropped
> >>  INFO [CompactionExecutor:1] 2011-03-27 08:08:10,065
> >> CompactionManager.java (line 512) Scrubbing
> >>
> SSTableReader(path='/data/cassandra/system/HintsColumnFamily-f-531-Data.db')
> >>  INFO [CompactionExecutor:1] 2011-03-27 08:08:10,145
> >> CompactionManager.java (line 653) Scrub of
> >>
> SSTableReader(path='/data/cassandra/system/HintsColumnFamily-f-531-Data.db')
> >> complete: 1 rows in new sstable and 0 empty (tombstoned) rows dropped
> >>  INFO [CompactionExecutor:1] 2011-03-27 08:08:10,145
> >> CompactionManager.java (line 512) Scrubbing
> >>
> SSTableReader(path='/data/cassandra/system/HintsColumnFamily-f-532-Data.db')
> >>  INFO [CompactionExecutor:1] 2011-03-27 08:08:10,363
> >> CompactionManager.java (line 653) Scrub of
> >>
> SSTableReader(path='/data/cassandra/system/HintsColumnFamily-f-532-Data.db')
> >> complete: 1 rows in new sstable and 0 empty (tombstoned) rows dropped
> >>  INFO [CompactionExecutor:1] 2011-03-27 08:08:10,364
> >> CompactionManager.java (line 512) Scrubbing
> >>
> SSTableReader(path='/data/cassandra/system/HintsColumnFamily-f-533-Data.db')
> >>  INFO [CompactionExecutor:1] 2011-03-27 08:08:10,540
> >> CompactionManager.java (line 653) Scrub of
> >>
> SSTableReader(path='/data/cassandra/system/HintsColumnFamily-f-533-Data.db')
> >> complete: 1 rows in new sstable and 0 empty (tombstoned) rows dropped
> >> --------------------------------------------------------
> >>
> >
> >
> >
> > --
> > Jonathan Ellis
> > Project Chair, Apache Cassandra
> > co-founder of DataStax, the source for professional Cassandra support
> > http://www.datastax.com
> >
>
>
>
> --
> Shotaro Kamio
>