You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by aaron morton <aa...@thelastpickle.com> on 2012/01/02 11:26:47 UTC

Re: Dealing with "Corrupt (negative) value length encountered"

I would try to nodetool scrub the data on the node that that sent the bad data in the stream. You may be able to work which node from the logs, or it may be easier to just scrub them all. 

Hope that helps. 
 
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 31/12/2011, at 12:20 AM, Philippe wrote:

> Hello,
> Running a combination of 0.8.6 and 0.8.8 with RF=3, I am getting the following while repairing one node (all other nodes completed successfully).
> Can I just stop the instance, erase the SSTable and restart cleanup ?
> Thanks
> 
> ERROR [Thread-402484] 2011-12-29 14:51:03,687 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-402484,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException: Corrupt (negative) value length encountered
>         at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:154)
>         at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
>         at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:189)
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
> Caused by: java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException: Corrupt (negative) value length encountered
>         at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>         at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:138)
>         ... 3 more
>

Re: Dealing with "Corrupt (negative) value length encountered"

Posted by Philippe <wa...@gmail.com>.

Thanks Aaron, I was able to complete the repair by scrubbing the column
family on all three replicas.

Cheers

2012/1/4 aaron morton <aa...@thelastpickle.com>

> I was able to scrub the node the repair that failed was running on. Are
> you saying the error could be displayed on that node but the bad data
> coming from another node ?
>
> Yes. The error occurred the node was receiving a data stream from another,
> you will need to clean the source of the data. You can either crawl through
> the logs or scrub the entire cluster.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 4/01/2012, at 9:15 AM, Philippe wrote:
>
> I was able to scrub the node the repair that failed was running on. Are
> you saying the error could be displayed on that node but the bad data
> coming from another node ?
>
> Log inspection also showed many of these, they seem to happen around when
> a stream transfer finishes.
> ERROR [Thread-550876] 2012-01-03 16:35:31,922
> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
> Thread[Thread-550876,5,main]
> java.lang.IllegalArgumentException
>         at
> sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:586)
>         at
> org.apache.cassandra.streaming.IncomingStreamReader.readnwrite(IncomingStreamReader.java:110)
>         at
> org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:85)
>         at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
>         at
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:189)
>         at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
>
> Thanks
>
> 2012/1/2 aaron morton <aa...@thelastpickle.com>
>
>> I would try to nodetool scrub the data on the node that that sent the bad
>> data in the stream. You may be able to work which node from the logs, or it
>> may be easier to just scrub them all.
>>
>> Hope that helps.
>>
>>   -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 31/12/2011, at 12:20 AM, Philippe wrote:
>>
>> Hello,
>> Running a combination of 0.8.6 and 0.8.8 with RF=3, I am getting the
>> following while repairing one node (all other nodes completed successfully).
>> Can I just stop the instance, erase the SSTable and restart cleanup ?
>> Thanks
>>
>> ERROR [Thread-402484] 2011-12-29 14:51:03,687
>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>> Thread[Thread-402484,5,main]
>> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
>> java.io.IOError: java.io.IOException: Corrupt (negative) value length
>> encountered
>>         at
>> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:154)
>>         at
>> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
>>         at
>> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:189)
>>         at
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
>> Caused by: java.util.concurrent.ExecutionException: java.io.IOError:
>> java.io.IOException: Corrupt (negative) value length encountered
>>         at
>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>>         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>>         at
>> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:138)
>>         ... 3 more
>>
>>
>>
>
>

Re: Dealing with "Corrupt (negative) value length encountered"

Posted by aaron morton <aa...@thelastpickle.com>.

> I was able to scrub the node the repair that failed was running on. Are you saying the error could be displayed on that node but the bad data coming from another node ?
Yes. The error occurred the node was receiving a data stream from another, you will need to clean the source of the data. You can either crawl through the logs or scrub the entire cluster. 

Cheers
 
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 4/01/2012, at 9:15 AM, Philippe wrote:

> I was able to scrub the node the repair that failed was running on. Are you saying the error could be displayed on that node but the bad data coming from another node ?
> 
> Log inspection also showed many of these, they seem to happen around when a stream transfer finishes.
> ERROR [Thread-550876] 2012-01-03 16:35:31,922 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-550876,5,main]
> java.lang.IllegalArgumentException
>         at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:586)
>         at org.apache.cassandra.streaming.IncomingStreamReader.readnwrite(IncomingStreamReader.java:110)
>         at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:85)
>         at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
>         at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:189)
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
> 
> Thanks
> 
> 2012/1/2 aaron morton <aa...@thelastpickle.com>
> I would try to nodetool scrub the data on the node that that sent the bad data in the stream. You may be able to work which node from the logs, or it may be easier to just scrub them all. 
> 
> Hope that helps. 
>  
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 31/12/2011, at 12:20 AM, Philippe wrote:
> 
>> Hello,
>> Running a combination of 0.8.6 and 0.8.8 with RF=3, I am getting the following while repairing one node (all other nodes completed successfully).
>> Can I just stop the instance, erase the SSTable and restart cleanup ?
>> Thanks
>> 
>> ERROR [Thread-402484] 2011-12-29 14:51:03,687 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-402484,5,main]
>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException: Corrupt (negative) value length encountered
>>         at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:154)
>>         at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
>>         at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:189)
>>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
>> Caused by: java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException: Corrupt (negative) value length encountered
>>         at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>>         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>>         at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:138)
>>         ... 3 more
>> 
> 
>

Re: Dealing with "Corrupt (negative) value length encountered"

Posted by Philippe <wa...@gmail.com>.

I was able to scrub the node the repair that failed was running on. Are you
saying the error could be displayed on that node but the bad data coming
from another node ?

Log inspection also showed many of these, they seem to happen around when a
stream transfer finishes.
ERROR [Thread-550876] 2012-01-03 16:35:31,922 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[Thread-550876,5,main]
java.lang.IllegalArgumentException
        at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:586)
        at
org.apache.cassandra.streaming.IncomingStreamReader.readnwrite(IncomingStreamReader.java:110)
        at
org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:85)
        at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
        at
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:189)
        at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)

Thanks

2012/1/2 aaron morton <aa...@thelastpickle.com>

> I would try to nodetool scrub the data on the node that that sent the bad
> data in the stream. You may be able to work which node from the logs, or it
> may be easier to just scrub them all.
>
> Hope that helps.
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 31/12/2011, at 12:20 AM, Philippe wrote:
>
> Hello,
> Running a combination of 0.8.6 and 0.8.8 with RF=3, I am getting the
> following while repairing one node (all other nodes completed successfully).
> Can I just stop the instance, erase the SSTable and restart cleanup ?
> Thanks
>
> ERROR [Thread-402484] 2011-12-29 14:51:03,687 AbstractCassandraDaemon.java
> (line 139) Fatal exception in thread Thread[Thread-402484,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.io.IOError: java.io.IOException: Corrupt (negative) value length
> encountered
>         at
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:154)
>         at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
>         at
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:189)
>         at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
> Caused by: java.util.concurrent.ExecutionException: java.io.IOError:
> java.io.IOException: Corrupt (negative) value length encountered
>         at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>         at
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:138)
>         ... 3 more
>
>
>