You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jan Kesten <j....@enercast.de> on 2013/07/18 17:52:10 UTC
Corrupted sstable and sstableloader
Hello together,
today I experienced a problem while loading a snapshot from our
cassandra cluster to test cluster. The cluster has six nodes and I took
a snapshot from all nodes concurrently and tried to import them in the
other cluster.
From 5 out of 6 nodes importing went well with no errors. But one
snapshot of one node cannot be imported - I tried serveral times. I got
the following while running sstableloader:
ERROR 09:13:06,084 Error in ThreadPoolExecutor
java.lang.RuntimeException: java.io.IOException: Datenübergabe
unterbrochen (broken pipe)
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Datenübergabe unterbrochen (broken pipe)
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at
sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:420)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:552)
at
org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:93)
at
org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
... 3 more
Exception in thread "Streaming to /172.17.2.216:1"
java.lang.RuntimeException: java.io.IOException: Datenübergabe
unterbrochen (broken pipe)
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Datenübergabe unterbrochen (broken pipe)
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at
sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:420)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:552)
at
org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:93)
at
org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
... 3 more
I suspect that the sstable on the node is corrupted in some way - and a
scrub and repair should fix that I suppose.
Since the original cluster has a replication factor of 3 - shoudn't the
import from 5 of 6 snapshots contain all data? Or is the sstableloader
tool too clever and avoids importing double data?
Thanks for hints,
Jan
--
Jan Kesten, mailto:j.kesten@enercast.de
Tel.: +49 561/4739664-0 FAX: -9
enercast GmbH Friedrich-Ebert-Str. 104 D-34119 Kassel HRB15471
http://www.enercast.de Online-Prognosen für erneuerbare Energien
Geschäftsführung: Dipl. Ing. Thomas Landgraf, Bernd Kratz
Diese E-Mail und etwaige Anhänge können vertrauliche und/oder rechtlich geschützte Informationen enthalten. Falls Sie nicht der angegebene Empfänger sind oder falls diese E-Mail irrtümlich an Sie adressiert wurde, benachrichtigen Sie uns bitte sofort durch Antwort-E-Mail und löschen Sie diese E-Mail nebst etwaigen Anlagen von Ihrem System. Ebenso dürfen Sie diese E-Mail oder ihre Anlagen nicht kopieren oder an Dritte weitergeben. Vielen Dank.
This e-mail and any attachment may contain confidential and/or privileged information. If you are not the named addressee or if this transmission has been addressed to you in error, please notify us immediately by reply e-mail and then delete this e-mail and any attachment from your system. Please understand that you must not copy this e-mail or any attachment or disclose the contents to any other person. Thank you for your cooperation.
Re: Corrupted sstable and sstableloader
Posted by Jan Kesten <j....@enercast.de>.
On 18.07.2013 19:19, Robert Coli wrote:
>
> Why not just determine which SSTable is corrupt, remove it from the
> restore set, then run a repair when you're done to be totally sure all
> data is on all nodes?
This is what I did finally - was some kind of work, since sstableloader
just stopped with exception but no hint which file was affected. So I
replayed the sstables one by one and finally found the corrupt one.
Thanks to all,
Jan
--
Jan Kesten, mailto:j.kesten@enercast.de
Tel.: +49 561/4739664-0 FAX: -9
enercast GmbH Friedrich-Ebert-Str. 104 D-34119 Kassel HRB15471
http://www.enercast.de Online-Prognosen für erneuerbare Energien
Geschäftsführung: Dipl. Ing. Thomas Landgraf, Bernd Kratz
Diese E-Mail und etwaige Anhänge können vertrauliche und/oder rechtlich geschützte Informationen enthalten. Falls Sie nicht der angegebene Empfänger sind oder falls diese E-Mail irrtümlich an Sie adressiert wurde, benachrichtigen Sie uns bitte sofort durch Antwort-E-Mail und löschen Sie diese E-Mail nebst etwaigen Anlagen von Ihrem System. Ebenso dürfen Sie diese E-Mail oder ihre Anlagen nicht kopieren oder an Dritte weitergeben. Vielen Dank.
This e-mail and any attachment may contain confidential and/or privileged information. If you are not the named addressee or if this transmission has been addressed to you in error, please notify us immediately by reply e-mail and then delete this e-mail and any attachment from your system. Please understand that you must not copy this e-mail or any attachment or disclose the contents to any other person. Thank you for your cooperation.
Re: Corrupted sstable and sstableloader
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Jul 18, 2013 at 10:17 AM, Jan Kesten <j....@enercast.de> wrote:
> I think it might be corrupted due to a poweroutage. Apart from this issue
> reading the data with consistency level quorum (I have three replicas) did
> not issue an error - only the import to a different cluster.
>
> So, if I import all nodes except the one with the corrupted sstable -
> shoudn't I import two of the three replicas, so that the data is complete?
>
Why not just determine which SSTable is corrupt, remove it from the restore
set, then run a repair when you're done to be totally sure all data is on
all nodes?
=Rob
Re: Corrupted sstable and sstableloader
Posted by Jan Kesten <j....@enercast.de>.
Hi,
I think it might be corrupted due to a poweroutage. Apart from this issue reading the data with consistency level quorum (I have three replicas) did not issue an error - only the import to a different cluster.
So, if I import all nodes except the one with the corrupted sstable - shoudn't I import two of the three replicas, so that the data is complete?
Von meinem iPhone gesendet
Am 18.07.2013 um 19:06 schrieb sankalp kohli <ko...@gmail.com>:
> sstable might be corrupted due to bad disk. In that case, replication does not matter.
>
>
> On Thu, Jul 18, 2013 at 8:52 AM, Jan Kesten <j....@enercast.de> wrote:
>> Hello together,
>>
>> today I experienced a problem while loading a snapshot from our cassandra cluster to test cluster. The cluster has six nodes and I took a snapshot from all nodes concurrently and tried to import them in the other cluster.
>>
>> From 5 out of 6 nodes importing went well with no errors. But one snapshot of one node cannot be imported - I tried serveral times. I got the following while running sstableloader:
>>
>> ERROR 09:13:06,084 Error in ThreadPoolExecutor
>> java.lang.RuntimeException: java.io.IOException: Datenübergabe unterbrochen (broken pipe)
>> at com.google.common.base.Throwables.propagate(Throwables.java:160)
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:724)
>> Caused by: java.io.IOException: Datenübergabe unterbrochen (broken pipe)
>> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
>> at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:420)
>> at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:552)
>> at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:93)
>> at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>> ... 3 more
>> Exception in thread "Streaming to /172.17.2.216:1" java.lang.RuntimeException: java.io.IOException: Datenübergabe unterbrochen (broken pipe)
>> at com.google.common.base.Throwables.propagate(Throwables.java:160)
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:724)
>> Caused by: java.io.IOException: Datenübergabe unterbrochen (broken pipe)
>> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
>> at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:420)
>> at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:552)
>> at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:93)
>> at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>> ... 3 more
>>
>> I suspect that the sstable on the node is corrupted in some way - and a scrub and repair should fix that I suppose.
>>
>> Since the original cluster has a replication factor of 3 - shoudn't the import from 5 of 6 snapshots contain all data? Or is the sstableloader tool too clever and avoids importing double data?
>>
>> Thanks for hints,
>> Jan
>>
>> --
>> Jan Kesten, mailto:j.kesten@enercast.de
>> Tel.: +49 561/4739664-0 FAX: -9
>> enercast GmbH Friedrich-Ebert-Str. 104 D-34119 Kassel HRB15471
>> http://www.enercast.de Online-Prognosen für erneuerbare Energien
>> Geschäftsführung: Dipl. Ing. Thomas Landgraf, Bernd Kratz
>>
>> Diese E-Mail und etwaige Anhänge können vertrauliche und/oder rechtlich geschützte Informationen enthalten. Falls Sie nicht der angegebene Empfänger sind oder falls diese E-Mail irrtümlich an Sie adressiert wurde, benachrichtigen Sie uns bitte sofort durch Antwort-E-Mail und löschen Sie diese E-Mail nebst etwaigen Anlagen von Ihrem System. Ebenso dürfen Sie diese E-Mail oder ihre Anlagen nicht kopieren oder an Dritte weitergeben. Vielen Dank.
>>
>> This e-mail and any attachment may contain confidential and/or privileged information. If you are not the named addressee or if this transmission has been addressed to you in error, please notify us immediately by reply e-mail and then delete this e-mail and any attachment from your system. Please understand that you must not copy this e-mail or any attachment or disclose the contents to any other person. Thank you for your cooperation.
>
Re: Corrupted sstable and sstableloader
Posted by sankalp kohli <ko...@gmail.com>.
sstable might be corrupted due to bad disk. In that case, replication does
not matter.
On Thu, Jul 18, 2013 at 8:52 AM, Jan Kesten <j....@enercast.de> wrote:
> Hello together,
>
> today I experienced a problem while loading a snapshot from our cassandra
> cluster to test cluster. The cluster has six nodes and I took a snapshot
> from all nodes concurrently and tried to import them in the other cluster.
>
> From 5 out of 6 nodes importing went well with no errors. But one snapshot
> of one node cannot be imported - I tried serveral times. I got the
> following while running sstableloader:
>
> ERROR 09:13:06,084 Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.io.IOException: Datenübergabe
> unterbrochen (broken pipe)
> at com.google.common.base.**Throwables.propagate(**
> Throwables.java:160)
> at org.apache.cassandra.utils.**WrappedRunnable.run(**
> WrappedRunnable.java:32)
> at java.util.concurrent.**ThreadPoolExecutor.runWorker(**
> ThreadPoolExecutor.java:1145)
> at java.util.concurrent.**ThreadPoolExecutor$Worker.run(**
> ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.**java:724)
> Caused by: java.io.IOException: Datenübergabe unterbrochen (broken pipe)
> at sun.nio.ch.FileChannelImpl.**transferTo0(Native Method)
> at sun.nio.ch.FileChannelImpl.**transferToDirectly(**
> FileChannelImpl.java:420)
> at sun.nio.ch.FileChannelImpl.**transferTo(FileChannelImpl.**java:552)
> at org.apache.cassandra.**streaming.compress.**
> CompressedFileStreamTask.**stream(**CompressedFileStreamTask.java:**93)
> at org.apache.cassandra.**streaming.FileStreamTask.**
> runMayThrow(FileStreamTask.**java:91)
> at org.apache.cassandra.utils.**WrappedRunnable.run(**
> WrappedRunnable.java:28)
> ... 3 more
> Exception in thread "Streaming to /172.17.2.216:1"
> java.lang.RuntimeException: java.io.IOException: Datenübergabe unterbrochen
> (broken pipe)
> at com.google.common.base.**Throwables.propagate(**
> Throwables.java:160)
> at org.apache.cassandra.utils.**WrappedRunnable.run(**
> WrappedRunnable.java:32)
> at java.util.concurrent.**ThreadPoolExecutor.runWorker(**
> ThreadPoolExecutor.java:1145)
> at java.util.concurrent.**ThreadPoolExecutor$Worker.run(**
> ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.**java:724)
> Caused by: java.io.IOException: Datenübergabe unterbrochen (broken pipe)
> at sun.nio.ch.FileChannelImpl.**transferTo0(Native Method)
> at sun.nio.ch.FileChannelImpl.**transferToDirectly(**
> FileChannelImpl.java:420)
> at sun.nio.ch.FileChannelImpl.**transferTo(FileChannelImpl.**java:552)
> at org.apache.cassandra.**streaming.compress.**
> CompressedFileStreamTask.**stream(**CompressedFileStreamTask.java:**93)
> at org.apache.cassandra.**streaming.FileStreamTask.**
> runMayThrow(FileStreamTask.**java:91)
> at org.apache.cassandra.utils.**WrappedRunnable.run(**
> WrappedRunnable.java:28)
> ... 3 more
>
> I suspect that the sstable on the node is corrupted in some way - and a
> scrub and repair should fix that I suppose.
>
> Since the original cluster has a replication factor of 3 - shoudn't the
> import from 5 of 6 snapshots contain all data? Or is the sstableloader tool
> too clever and avoids importing double data?
>
> Thanks for hints,
> Jan
>
> --
> Jan Kesten, mailto:j.kesten@enercast.de
> Tel.: +49 561/4739664-0 FAX: -9
> enercast GmbH Friedrich-Ebert-Str. 104 D-34119 Kassel HRB15471
> http://www.enercast.de Online-Prognosen für erneuerbare Energien
> Geschäftsführung: Dipl. Ing. Thomas Landgraf, Bernd Kratz
>
> Diese E-Mail und etwaige Anhänge können vertrauliche und/oder rechtlich
> geschützte Informationen enthalten. Falls Sie nicht der angegebene
> Empfänger sind oder falls diese E-Mail irrtümlich an Sie adressiert wurde,
> benachrichtigen Sie uns bitte sofort durch Antwort-E-Mail und löschen Sie
> diese E-Mail nebst etwaigen Anlagen von Ihrem System. Ebenso dürfen Sie
> diese E-Mail oder ihre Anlagen nicht kopieren oder an Dritte weitergeben.
> Vielen Dank.
>
> This e-mail and any attachment may contain confidential and/or privileged
> information. If you are not the named addressee or if this transmission has
> been addressed to you in error, please notify us immediately by reply
> e-mail and then delete this e-mail and any attachment from your system.
> Please understand that you must not copy this e-mail or any attachment or
> disclose the contents to any other person. Thank you for your cooperation.
>
>