You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by ifjke <j....@enercast.de> on 2013/07/03 16:04:00 UTC
CorruptBlockException - recover?
Hi together,
I found that one of my cassandra nodes died recently (machine hangs). I
restarted the node an run a nodetool repair, while running it has thrown
a org.apache.cassandra.io.compress.CorruptBlockException. Is there any
way to recover from this? Or would it be best to delete the nodes
contents and bootstrap it again?
Thanks for any hints,
Jan
org.apache.cassandra.io.sstable.CorruptSSTableException:
org.apache.cassandra.io.compress.CorruptBlockException: (/home/cassandra
/data/example/EXAMPLE/example-EXAMPLE-ib-8240-Data.db): corruption
detected, chunk at 98414229 of length 8098.
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:92)
at
org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:355)
at
java.io.RandomAccessFile.readFully(RandomAccessFile.java:414)
at
java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
at
org.apache.cassandra.utils.BytesReadTracker.readFully(BytesReadTracker.java:94)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
at
org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:108)
at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235)
at
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109)
at
org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:93)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:146)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Re: CorruptBlockException - recover?
Posted by Michał Michalski <mi...@opera.com>.
I think I'd try removing "broken" SSTables (when node is down) and
running repair then.
M.
W dniu 05.07.2013 09:10, Jan Kesten pisze:
> Hi,
>
> i tried to scrub the keyspace - but with no success either, the process
> threw an exception when hitting the corrupt block and stopped then. I
> will rebootstrap the node :-)
>
> Thanks anyways,
> Jan
>
> On 03.07.2013 19:10, Glenn Thompson wrote:
>> For what its worth. I did this when I had this problem. It didn't
>> work out for me. Perhaps I did something wrong.
>>
>>
>> On Wed, Jul 3, 2013 at 11:06 AM, Robert Coli <rcoli@eventbrite.com
>> <ma...@eventbrite.com>> wrote:
>>
>> On Wed, Jul 3, 2013 at 7:04 AM, ifjke <j.kesten@enercast.de
>> <ma...@enercast.de>> wrote:
>>
>> I found that one of my cassandra nodes died recently (machine
>> hangs). I restarted the node an run a nodetool repair, while
>> running it has thrown a org.apache.cassandra.io
>> <http://org.apache.cassandra.io>.compress.CorruptBlockException.
>> Is there any way to recover from this? Or would it be best to
>> delete the nodes contents and bootstrap it again?
>>
>>
>> If you "scrub" this SSTable (either with the online or offline
>> version of "scrub") it will remove the corrupt data and re-write
>> the rest of the SSTable which isn't corrupt into a new SSTable.
>> That is probably safer for your data than deleting the entire set
>> of data on this replica. When that's done, restart the repair.
>>
>> =Rob
>>
>>
>
>
Re: CorruptBlockException - recover?
Posted by Robert Coli <rc...@eventbrite.com>.
On Fri, Jul 5, 2013 at 12:10 AM, Jan Kesten <j....@enercast.de> wrote:
> i tried to scrub the keyspace - but with no success either, the process
> threw an exception when hitting the corrupt block and stopped then. I will
> rebootstrap the node :-)
>
http://www.datastax.com/docs/1.1/install/upgrading
Did you try offline scrub?
=Rob
Re: CorruptBlockException - recover?
Posted by Jan Kesten <j....@enercast.de>.
Hi,
i tried to scrub the keyspace - but with no success either, the process
threw an exception when hitting the corrupt block and stopped then. I
will rebootstrap the node :-)
Thanks anyways,
Jan
On 03.07.2013 19:10, Glenn Thompson wrote:
> For what its worth. I did this when I had this problem. It didn't
> work out for me. Perhaps I did something wrong.
>
>
> On Wed, Jul 3, 2013 at 11:06 AM, Robert Coli <rcoli@eventbrite.com
> <ma...@eventbrite.com>> wrote:
>
> On Wed, Jul 3, 2013 at 7:04 AM, ifjke <j.kesten@enercast.de
> <ma...@enercast.de>> wrote:
>
> I found that one of my cassandra nodes died recently (machine
> hangs). I restarted the node an run a nodetool repair, while
> running it has thrown a org.apache.cassandra.io
> <http://org.apache.cassandra.io>.compress.CorruptBlockException.
> Is there any way to recover from this? Or would it be best to
> delete the nodes contents and bootstrap it again?
>
>
> If you "scrub" this SSTable (either with the online or offline
> version of "scrub") it will remove the corrupt data and re-write
> the rest of the SSTable which isn't corrupt into a new SSTable.
> That is probably safer for your data than deleting the entire set
> of data on this replica. When that's done, restart the repair.
>
> =Rob
>
>
--
Jan Kesten, mailto:j.kesten@enercast.de
Tel.: +49 561/4739664-0 FAX: -9
enercast GmbH Friedrich-Ebert-Str. 104 D-34119 Kassel HRB15471
http://www.enercast.de Online-Prognosen für erneuerbare Energien
Geschäftsführung: Dipl. Ing. Thomas Landgraf, Bernd Kratz
Diese E-Mail und etwaige Anhänge können vertrauliche und/oder rechtlich geschützte Informationen enthalten. Falls Sie nicht der angegebene Empfänger sind oder falls diese E-Mail irrtümlich an Sie adressiert wurde, benachrichtigen Sie uns bitte sofort durch Antwort-E-Mail und löschen Sie diese E-Mail nebst etwaigen Anlagen von Ihrem System. Ebenso dürfen Sie diese E-Mail oder ihre Anlagen nicht kopieren oder an Dritte weitergeben. Vielen Dank.
This e-mail and any attachment may contain confidential and/or privileged information. If you are not the named addressee or if this transmission has been addressed to you in error, please notify us immediately by reply e-mail and then delete this e-mail and any attachment from your system. Please understand that you must not copy this e-mail or any attachment or disclose the contents to any other person. Thank you for your cooperation.
Re: CorruptBlockException - recover?
Posted by Glenn Thompson <ga...@gmail.com>.
For what its worth. I did this when I had this problem. It didn't work
out for me. Perhaps I did something wrong.
On Wed, Jul 3, 2013 at 11:06 AM, Robert Coli <rc...@eventbrite.com> wrote:
> On Wed, Jul 3, 2013 at 7:04 AM, ifjke <j....@enercast.de> wrote:
>
>> I found that one of my cassandra nodes died recently (machine hangs). I
>> restarted the node an run a nodetool repair, while running it has thrown a
>> org.apache.cassandra.io.**compress.**CorruptBlockException. Is there any
>> way to recover from this? Or would it be best to delete the nodes contents
>> and bootstrap it again?
>>
>
> If you "scrub" this SSTable (either with the online or offline version of
> "scrub") it will remove the corrupt data and re-write the rest of the
> SSTable which isn't corrupt into a new SSTable. That is probably safer for
> your data than deleting the entire set of data on this replica. When that's
> done, restart the repair.
>
> =Rob
>
Re: CorruptBlockException - recover?
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, Jul 3, 2013 at 7:04 AM, ifjke <j....@enercast.de> wrote:
> I found that one of my cassandra nodes died recently (machine hangs). I
> restarted the node an run a nodetool repair, while running it has thrown a
> org.apache.cassandra.io.**compress.**CorruptBlockException. Is there any
> way to recover from this? Or would it be best to delete the nodes contents
> and bootstrap it again?
>
If you "scrub" this SSTable (either with the online or offline version of
"scrub") it will remove the corrupt data and re-write the rest of the
SSTable which isn't corrupt into a new SSTable. That is probably safer for
your data than deleting the entire set of data on this replica. When that's
done, restart the repair.
=Rob