You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jean Carlo <je...@gmail.com> on 2016/08/05 14:15:26 UTC
Sync failed between in AntiEntropySessions - Repair
Hi guys, Doing a repair I got this error for 2 tokenranges.
ERROR [Thread-2499244] 2016-08-04 20:05:24,288 StorageService.java:3068 -
Repair session 41e4bab0-5a63-11e6-9993-e11d93fd5b40 for range
(487410372471205090,492009442088088379] failed with error
org.apache.cassandra.exceptions.RepairExcep
tion: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
(487410372471205090,492009442088088379]] Sync failed between /192.168.0.144
and /192.168.0.37
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
org.apache.cassandra.exceptions.RepairException: [repair
#41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
(487410372471205090,492009442088088379]] Sync failed between /192.168.0.144
and /192.168.0.37
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
[na:1.8.0_60]
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
[na:1.8.0_60]
at
org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:3059)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[apache-cassandra-2.1.14.jar:2.1.14]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
Caused by: java.lang.RuntimeException:
org.apache.cassandra.exceptions.RepairException: [repair
#41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
(487410372471205090,492009442088088379]] Sync failed between /10.234.72.144
and
/10.234.86.37
at com.google.common.base.Throwables.propagate(Throwables.java:160)
~[guava-16.0.jar:na]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
[apache-cassandra-2.1.14.jar:2.1.14]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_60]
... 1 common frames omitted
Caused by: org.apache.cassandra.exceptions.RepairException: [repair
#41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
(487410372471205090,492009442088088379]] Sync failed between /192.168.0.144
and /192.168.0.37
at
org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:422)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-2.1.14.jar:2.1.14]
... 3 common frames omitted
Is it a network problem or we are having a bug ? There is nothing else in
the log of cassandra more than
Sync failed between.
We are having cassandra 2.1.14
Best regards
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
Re: Sync failed between in AntiEntropySessions - Repair
Posted by Jean Carlo <je...@gmail.com>.
Thank you very much Paulo
On Aug 5, 2016 17:31, "Paulo Motta" <pa...@gmail.com> wrote:
> https://issues.apache.org/jira/browse/CASSANDRA-11840
>
> increase streaming_socket_timeout to 86400000 or upgrade to
> cassandra-2.1.15.
>
> 2016-08-05 12:28 GMT-03:00 Jean Carlo <je...@gmail.com>:
>
>>
>> Hello Paulo,
>>
>> Thx for your fast replay.
>>
>> You are right about that node, I did not see it that fast. In this node
>> we have errors of .SocketTimeoutException: null
>>
>>
>>
>> ERROR [STREAM-IN-/192.168.0.146] 2016-08-04 19:10:59,456
>> StreamSession.java:505 - [Stream #06c02460-5a5e-11e6-8e9a-a5bf51981ad8]
>> Streaming error occurred
>> java.net.SocketTimeoutException: null
>> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
>> ~[na:1.8.0_60]
>> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
>> ~[na:1.8.0_60]
>> at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
>> ~[na:1.8.0_60]
>> at org.apache.cassandra.streaming.messages.StreamMessage.
>> deserialize(StreamMessage.java:51) ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at org.apache.cassandra.streaming.ConnectionHandler$IncomingMes
>> sageHandler.run(ConnectionHandler.java:257)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>
>> We have this error with at least 5 more nodes in the same log. But the
>> error doesn't say that much
>>
>>
>> Saludos
>>
>> Jean Carlo
>>
>> "The best way to predict the future is to invent it" Alan Kay
>>
>> On Fri, Aug 5, 2016 at 5:16 PM, Paulo Motta <pa...@gmail.com>
>> wrote:
>>
>>> you need to check 192.168.0.36/10.234.86.36 for streaming ERRORS
>>>
>>> 2016-08-05 12:08 GMT-03:00 Jean Carlo <je...@gmail.com>:
>>>
>>>> Hi Paulo
>>>>
>>>> I found the lines, we got an exception "Outgoing stream handler has
>>>> been closed" these are
>>>>
>>>> ERROR [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772
>>>> StreamSession.java:621 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>>>> Remote peer 192.168.0.36 failed stream session.
>>>> INFO [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772
>>>> StreamResultFuture.java:180 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>>>> Session with /192.168.0.36 is complete
>>>> WARN [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,773
>>>> StreamResultFuture.java:207 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>>>> Stream failed
>>>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782
>>>> StreamReceiveTask.java:183 - Error applying streamed data:
>>>> java.lang.RuntimeException: Outgoing stream handler has been closed
>>>> at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio
>>>> nRunnable.run(StreamReceiveTask.java:179)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> [na:1.8.0_60]
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> [na:1.8.0_60]
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> [na:1.8.0_60]
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> [na:1.8.0_60]
>>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>>>
>>>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782
>>>> StreamSession.java:505 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>>>> Streaming error occurred
>>>> java.lang.RuntimeException: Outgoing stream handler has been closed
>>>> at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio
>>>> nRunnable.run(StreamReceiveTask.java:179)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> [na:1.8.0_60]
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> [na:1.8.0_60]
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> [na:1.8.0_60]
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> [na:1.8.0_60]
>>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>>>
>>>> I al looking through the changes files to see if it is a bug fixed in
>>>> the 2.1.15
>>>>
>>>>
>>>>
>>>> Saludos
>>>>
>>>> Jean Carlo
>>>>
>>>> "The best way to predict the future is to invent it" Alan Kay
>>>>
>>>> On Fri, Aug 5, 2016 at 4:24 PM, Paulo Motta <pa...@gmail.com>
>>>> wrote:
>>>>
>>>>> It seems you have a streaming error, look for ERROR statement in the
>>>>> streaming classes before that which may give you a more specific root
>>>>> cause. In any case, I'd suggest you to upgrade to 2.1.15 as there were a
>>>>> couple of streaming fixes on this version that might help.
>>>>>
>>>>> 2016-08-05 11:15 GMT-03:00 Jean Carlo <je...@gmail.com>:
>>>>>
>>>>>>
>>>>>> Hi guys, Doing a repair I got this error for 2 tokenranges.
>>>>>>
>>>>>> ERROR [Thread-2499244] 2016-08-04 20:05:24,288
>>>>>> StorageService.java:3068 - Repair session 41e4bab0-5a63-11e6-9993-e11d93fd5b40
>>>>>> for range (487410372471205090,492009442088088379] failed with error
>>>>>> org.apache.cassandra.exceptions.RepairExcep
>>>>>> tion: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>>>> 192.168.0.144 and /192.168.0.37
>>>>>> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
>>>>>> org.apache.cassandra.exceptions.RepairException: [repair
>>>>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>>>> 192.168.0.144 and /192.168.0.37
>>>>>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>>>>>> [na:1.8.0_60]
>>>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>>>>>> [na:1.8.0_60]
>>>>>> at org.apache.cassandra.service.S
>>>>>> torageService$4.runMayThrow(StorageService.java:3059)
>>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>>> at org.apache.cassandra.utils.Wra
>>>>>> ppedRunnable.run(WrappedRunnable.java:28)
>>>>>> [apache-cassandra-2.1.14.jar:2.1.14]
>>>>>> at java.util.concurrent.Executors
>>>>>> $RunnableAdapter.call(Executors.java:511) [na:1.8.0_60]
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>> [na:1.8.0_60]
>>>>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>>>>> Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException:
>>>>>> [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>>>> 10.234.72.144 and
>>>>>> /10.234.86.37
>>>>>> at com.google.common.base.Throwab
>>>>>> les.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
>>>>>> at org.apache.cassandra.utils.Wra
>>>>>> ppedRunnable.run(WrappedRunnable.java:32)
>>>>>> [apache-cassandra-2.1.14.jar:2.1.14]
>>>>>> at java.util.concurrent.Executors
>>>>>> $RunnableAdapter.call(Executors.java:511) [na:1.8.0_60]
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>> [na:1.8.0_60]
>>>>>> at java.util.concurrent.ThreadPoo
>>>>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
>>>>>> at java.util.concurrent.ThreadPoo
>>>>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_60]
>>>>>> ... 1 common frames omitted
>>>>>> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
>>>>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>>>> 192.168.0.144 and /192.168.0.37
>>>>>> at org.apache.cassandra.repair.Re
>>>>>> pairSession.syncComplete(RepairSession.java:223)
>>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>>> at org.apache.cassandra.service.A
>>>>>> ctiveRepairService.handleMessage(ActiveRepairService.java:422)
>>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>>> at org.apache.cassandra.repair.Re
>>>>>> pairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134)
>>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>>> at org.apache.cassandra.net.Messa
>>>>>> geDeliveryTask.run(MessageDeliveryTask.java:64)
>>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>>> ... 3 common frames omitted
>>>>>>
>>>>>> Is it a network problem or we are having a bug ? There is nothing
>>>>>> else in the log of cassandra more than
>>>>>> Sync failed between.
>>>>>>
>>>>>> We are having cassandra 2.1.14
>>>>>>
>>>>>> Best regards
>>>>>>
>>>>>> Jean Carlo
>>>>>>
>>>>>> "The best way to predict the future is to invent it" Alan Kay
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Sync failed between in AntiEntropySessions - Repair
Posted by Paulo Motta <pa...@gmail.com>.
https://issues.apache.org/jira/browse/CASSANDRA-11840
increase streaming_socket_timeout to 86400000 or upgrade to
cassandra-2.1.15.
2016-08-05 12:28 GMT-03:00 Jean Carlo <je...@gmail.com>:
>
> Hello Paulo,
>
> Thx for your fast replay.
>
> You are right about that node, I did not see it that fast. In this node we
> have errors of .SocketTimeoutException: null
>
>
>
> ERROR [STREAM-IN-/192.168.0.146] 2016-08-04 19:10:59,456
> StreamSession.java:505 - [Stream #06c02460-5a5e-11e6-8e9a-a5bf51981ad8]
> Streaming error occurred
> java.net.SocketTimeoutException: null
> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
> ~[na:1.8.0_60]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_60]
> at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> ~[na:1.8.0_60]
> at org.apache.cassandra.streaming.messages.
> StreamMessage.deserialize(StreamMessage.java:51)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at org.apache.cassandra.streaming.ConnectionHandler$
> IncomingMessageHandler.run(ConnectionHandler.java:257)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>
> We have this error with at least 5 more nodes in the same log. But the
> error doesn't say that much
>
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
> On Fri, Aug 5, 2016 at 5:16 PM, Paulo Motta <pa...@gmail.com>
> wrote:
>
>> you need to check 192.168.0.36/10.234.86.36 for streaming ERRORS
>>
>> 2016-08-05 12:08 GMT-03:00 Jean Carlo <je...@gmail.com>:
>>
>>> Hi Paulo
>>>
>>> I found the lines, we got an exception "Outgoing stream handler has been
>>> closed" these are
>>>
>>> ERROR [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772
>>> StreamSession.java:621 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>>> Remote peer 192.168.0.36 failed stream session.
>>> INFO [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772
>>> StreamResultFuture.java:180 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>>> Session with /192.168.0.36 is complete
>>> WARN [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,773
>>> StreamResultFuture.java:207 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>>> Stream failed
>>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782
>>> StreamReceiveTask.java:183 - Error applying streamed data:
>>> java.lang.RuntimeException: Outgoing stream handler has been closed
>>> at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>> at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697)
>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>> at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653)
>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio
>>> nRunnable.run(StreamReceiveTask.java:179) ~[apache-cassandra-2.1.14.jar:
>>> 2.1.14]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> [na:1.8.0_60]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> [na:1.8.0_60]
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> [na:1.8.0_60]
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> [na:1.8.0_60]
>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>>
>>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782
>>> StreamSession.java:505 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>>> Streaming error occurred
>>> java.lang.RuntimeException: Outgoing stream handler has been closed
>>> at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>> at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697)
>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>> at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653)
>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio
>>> nRunnable.run(StreamReceiveTask.java:179) ~[apache-cassandra-2.1.14.jar:
>>> 2.1.14]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> [na:1.8.0_60]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> [na:1.8.0_60]
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> [na:1.8.0_60]
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> [na:1.8.0_60]
>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>>
>>> I al looking through the changes files to see if it is a bug fixed in
>>> the 2.1.15
>>>
>>>
>>>
>>> Saludos
>>>
>>> Jean Carlo
>>>
>>> "The best way to predict the future is to invent it" Alan Kay
>>>
>>> On Fri, Aug 5, 2016 at 4:24 PM, Paulo Motta <pa...@gmail.com>
>>> wrote:
>>>
>>>> It seems you have a streaming error, look for ERROR statement in the
>>>> streaming classes before that which may give you a more specific root
>>>> cause. In any case, I'd suggest you to upgrade to 2.1.15 as there were a
>>>> couple of streaming fixes on this version that might help.
>>>>
>>>> 2016-08-05 11:15 GMT-03:00 Jean Carlo <je...@gmail.com>:
>>>>
>>>>>
>>>>> Hi guys, Doing a repair I got this error for 2 tokenranges.
>>>>>
>>>>> ERROR [Thread-2499244] 2016-08-04 20:05:24,288
>>>>> StorageService.java:3068 - Repair session 41e4bab0-5a63-11e6-9993-e11d93fd5b40
>>>>> for range (487410372471205090,492009442088088379] failed with error
>>>>> org.apache.cassandra.exceptions.RepairExcep
>>>>> tion: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>>> 192.168.0.144 and /192.168.0.37
>>>>> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
>>>>> org.apache.cassandra.exceptions.RepairException: [repair
>>>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>>> 192.168.0.144 and /192.168.0.37
>>>>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>>>>> [na:1.8.0_60]
>>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>>>>> [na:1.8.0_60]
>>>>> at org.apache.cassandra.service.S
>>>>> torageService$4.runMayThrow(StorageService.java:3059)
>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>> at org.apache.cassandra.utils.Wra
>>>>> ppedRunnable.run(WrappedRunnable.java:28)
>>>>> [apache-cassandra-2.1.14.jar:2.1.14]
>>>>> at java.util.concurrent.Executors
>>>>> $RunnableAdapter.call(Executors.java:511) [na:1.8.0_60]
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>> [na:1.8.0_60]
>>>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>>>> Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException:
>>>>> [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>>> 10.234.72.144 and
>>>>> /10.234.86.37
>>>>> at com.google.common.base.Throwab
>>>>> les.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
>>>>> at org.apache.cassandra.utils.Wra
>>>>> ppedRunnable.run(WrappedRunnable.java:32)
>>>>> [apache-cassandra-2.1.14.jar:2.1.14]
>>>>> at java.util.concurrent.Executors
>>>>> $RunnableAdapter.call(Executors.java:511) [na:1.8.0_60]
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>> [na:1.8.0_60]
>>>>> at java.util.concurrent.ThreadPoo
>>>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
>>>>> at java.util.concurrent.ThreadPoo
>>>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_60]
>>>>> ... 1 common frames omitted
>>>>> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
>>>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>>> 192.168.0.144 and /192.168.0.37
>>>>> at org.apache.cassandra.repair.Re
>>>>> pairSession.syncComplete(RepairSession.java:223)
>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>> at org.apache.cassandra.service.A
>>>>> ctiveRepairService.handleMessage(ActiveRepairService.java:422)
>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>> at org.apache.cassandra.repair.Re
>>>>> pairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134)
>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>> at org.apache.cassandra.net.Messa
>>>>> geDeliveryTask.run(MessageDeliveryTask.java:64)
>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>> ... 3 common frames omitted
>>>>>
>>>>> Is it a network problem or we are having a bug ? There is nothing else
>>>>> in the log of cassandra more than
>>>>> Sync failed between.
>>>>>
>>>>> We are having cassandra 2.1.14
>>>>>
>>>>> Best regards
>>>>>
>>>>> Jean Carlo
>>>>>
>>>>> "The best way to predict the future is to invent it" Alan Kay
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Sync failed between in AntiEntropySessions - Repair
Posted by Jean Carlo <je...@gmail.com>.
Hello Paulo,
Thx for your fast replay.
You are right about that node, I did not see it that fast. In this node we
have errors of .SocketTimeoutException: null
ERROR [STREAM-IN-/192.168.0.146] 2016-08-04 19:10:59,456
StreamSession.java:505 - [Stream #06c02460-5a5e-11e6-8e9a-a5bf51981ad8]
Streaming error occurred
java.net.SocketTimeoutException: null
at
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
~[na:1.8.0_60]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
~[na:1.8.0_60]
at
java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
~[na:1.8.0_60]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:51)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:257)
~[apache-cassandra-2.1.14.jar:2.1.14]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
We have this error with at least 5 more nodes in the same log. But the
error doesn't say that much
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Fri, Aug 5, 2016 at 5:16 PM, Paulo Motta <pa...@gmail.com>
wrote:
> you need to check 192.168.0.36/10.234.86.36 for streaming ERRORS
>
> 2016-08-05 12:08 GMT-03:00 Jean Carlo <je...@gmail.com>:
>
>> Hi Paulo
>>
>> I found the lines, we got an exception "Outgoing stream handler has been
>> closed" these are
>>
>> ERROR [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772
>> StreamSession.java:621 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>> Remote peer 192.168.0.36 failed stream session.
>> INFO [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772
>> StreamResultFuture.java:180 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>> Session with /192.168.0.36 is complete
>> WARN [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,773
>> StreamResultFuture.java:207 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>> Stream failed
>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782
>> StreamReceiveTask.java:183 - Error applying streamed data:
>> java.lang.RuntimeException: Outgoing stream handler has been closed
>> at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio
>> nRunnable.run(StreamReceiveTask.java:179) ~[apache-cassandra-2.1.14.jar:
>> 2.1.14]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> [na:1.8.0_60]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> [na:1.8.0_60]
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> [na:1.8.0_60]
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_60]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>
>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782
>> StreamSession.java:505 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>> Streaming error occurred
>> java.lang.RuntimeException: Outgoing stream handler has been closed
>> at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio
>> nRunnable.run(StreamReceiveTask.java:179) ~[apache-cassandra-2.1.14.jar:
>> 2.1.14]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> [na:1.8.0_60]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> [na:1.8.0_60]
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> [na:1.8.0_60]
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_60]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>
>> I al looking through the changes files to see if it is a bug fixed in the
>> 2.1.15
>>
>>
>>
>> Saludos
>>
>> Jean Carlo
>>
>> "The best way to predict the future is to invent it" Alan Kay
>>
>> On Fri, Aug 5, 2016 at 4:24 PM, Paulo Motta <pa...@gmail.com>
>> wrote:
>>
>>> It seems you have a streaming error, look for ERROR statement in the
>>> streaming classes before that which may give you a more specific root
>>> cause. In any case, I'd suggest you to upgrade to 2.1.15 as there were a
>>> couple of streaming fixes on this version that might help.
>>>
>>> 2016-08-05 11:15 GMT-03:00 Jean Carlo <je...@gmail.com>:
>>>
>>>>
>>>> Hi guys, Doing a repair I got this error for 2 tokenranges.
>>>>
>>>> ERROR [Thread-2499244] 2016-08-04 20:05:24,288 StorageService.java:3068
>>>> - Repair session 41e4bab0-5a63-11e6-9993-e11d93fd5b40 for range
>>>> (487410372471205090,492009442088088379] failed with error
>>>> org.apache.cassandra.exceptions.RepairExcep
>>>> tion: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>> 192.168.0.144 and /192.168.0.37
>>>> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
>>>> org.apache.cassandra.exceptions.RepairException: [repair
>>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>> 192.168.0.144 and /192.168.0.37
>>>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>>>> [na:1.8.0_60]
>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>>>> [na:1.8.0_60]
>>>> at org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:3059)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>>>> [apache-cassandra-2.1.14.jar:2.1.14]
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> [na:1.8.0_60]
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> [na:1.8.0_60]
>>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>>> Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException:
>>>> [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>> 10.234.72.144 and
>>>> /10.234.86.37
>>>> at com.google.common.base.Throwables.propagate(Throwables.java:160)
>>>> ~[guava-16.0.jar:na]
>>>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
>>>> [apache-cassandra-2.1.14.jar:2.1.14]
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> [na:1.8.0_60]
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> [na:1.8.0_60]
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> ~[na:1.8.0_60]
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> ~[na:1.8.0_60]
>>>> ... 1 common frames omitted
>>>> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
>>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>> 192.168.0.144 and /192.168.0.37
>>>> at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> at org.apache.cassandra.service.ActiveRepairService.handleMessa
>>>> ge(ActiveRepairService.java:422) ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(
>>>> RepairMessageVerbHandler.java:134) ~[apache-cassandra-2.1.14.jar:
>>>> 2.1.14]
>>>> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>> ... 3 common frames omitted
>>>>
>>>> Is it a network problem or we are having a bug ? There is nothing else
>>>> in the log of cassandra more than
>>>> Sync failed between.
>>>>
>>>> We are having cassandra 2.1.14
>>>>
>>>> Best regards
>>>>
>>>> Jean Carlo
>>>>
>>>> "The best way to predict the future is to invent it" Alan Kay
>>>>
>>>
>>>
>>
>
Re: Sync failed between in AntiEntropySessions - Repair
Posted by Paulo Motta <pa...@gmail.com>.
you need to check 192.168.0.36/10.234.86.36 for streaming ERRORS
2016-08-05 12:08 GMT-03:00 Jean Carlo <je...@gmail.com>:
> Hi Paulo
>
> I found the lines, we got an exception "Outgoing stream handler has been
> closed" these are
>
> ERROR [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772
> StreamSession.java:621 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
> Remote peer 192.168.0.36 failed stream session.
> INFO [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772
> StreamResultFuture.java:180 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
> Session with /192.168.0.36 is complete
> WARN [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,773
> StreamResultFuture.java:207 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
> Stream failed
> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782
> StreamReceiveTask.java:183 - Error applying streamed data:
> java.lang.RuntimeException: Outgoing stream handler has been closed
> at org.apache.cassandra.streaming.ConnectionHandler.
> sendMessage(ConnectionHandler.java:138) ~[apache-cassandra-2.1.14.jar:
> 2.1.14]
> at org.apache.cassandra.streaming.StreamSession.
> maybeCompleted(StreamSession.java:697) ~[apache-cassandra-2.1.14.jar:
> 2.1.14]
> at org.apache.cassandra.streaming.StreamSession.
> taskCompleted(StreamSession.java:653) ~[apache-cassandra-2.1.14.jar:
> 2.1.14]
> at org.apache.cassandra.streaming.StreamReceiveTask$
> OnCompletionRunnable.run(StreamReceiveTask.java:179)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_60]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_60]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>
> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782
> StreamSession.java:505 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
> Streaming error occurred
> java.lang.RuntimeException: Outgoing stream handler has been closed
> at org.apache.cassandra.streaming.ConnectionHandler.
> sendMessage(ConnectionHandler.java:138) ~[apache-cassandra-2.1.14.jar:
> 2.1.14]
> at org.apache.cassandra.streaming.StreamSession.
> maybeCompleted(StreamSession.java:697) ~[apache-cassandra-2.1.14.jar:
> 2.1.14]
> at org.apache.cassandra.streaming.StreamSession.
> taskCompleted(StreamSession.java:653) ~[apache-cassandra-2.1.14.jar:
> 2.1.14]
> at org.apache.cassandra.streaming.StreamReceiveTask$
> OnCompletionRunnable.run(StreamReceiveTask.java:179)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_60]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_60]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>
> I al looking through the changes files to see if it is a bug fixed in the
> 2.1.15
>
>
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
> On Fri, Aug 5, 2016 at 4:24 PM, Paulo Motta <pa...@gmail.com>
> wrote:
>
>> It seems you have a streaming error, look for ERROR statement in the
>> streaming classes before that which may give you a more specific root
>> cause. In any case, I'd suggest you to upgrade to 2.1.15 as there were a
>> couple of streaming fixes on this version that might help.
>>
>> 2016-08-05 11:15 GMT-03:00 Jean Carlo <je...@gmail.com>:
>>
>>>
>>> Hi guys, Doing a repair I got this error for 2 tokenranges.
>>>
>>> ERROR [Thread-2499244] 2016-08-04 20:05:24,288 StorageService.java:3068
>>> - Repair session 41e4bab0-5a63-11e6-9993-e11d93fd5b40 for range
>>> (487410372471205090,492009442088088379] failed with error
>>> org.apache.cassandra.exceptions.RepairExcep
>>> tion: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>> (487410372471205090,492009442088088379]] Sync failed between /
>>> 192.168.0.144 and /192.168.0.37
>>> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
>>> org.apache.cassandra.exceptions.RepairException: [repair
>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, (487410372471205090,
>>> 492009442088088379]] Sync failed between /192.168.0.144 and /
>>> 192.168.0.37
>>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>>> [na:1.8.0_60]
>>> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>>> [na:1.8.0_60]
>>> at org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:3059)
>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>>> [apache-cassandra-2.1.14.jar:2.1.14]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> [na:1.8.0_60]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> [na:1.8.0_60]
>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>> Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException:
>>> [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>> (487410372471205090,492009442088088379]] Sync failed between /
>>> 10.234.72.144 and
>>> /10.234.86.37
>>> at com.google.common.base.Throwables.propagate(Throwables.java:160)
>>> ~[guava-16.0.jar:na]
>>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
>>> [apache-cassandra-2.1.14.jar:2.1.14]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> [na:1.8.0_60]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> [na:1.8.0_60]
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> ~[na:1.8.0_60]
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> ~[na:1.8.0_60]
>>> ... 1 common frames omitted
>>> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, (487410372471205090,
>>> 492009442088088379]] Sync failed between /192.168.0.144 and /
>>> 192.168.0.37
>>> at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223)
>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>> at org.apache.cassandra.service.ActiveRepairService.handleMessa
>>> ge(ActiveRepairService.java:422) ~[apache-cassandra-2.1.14.jar:2.1.14]
>>> at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(
>>> RepairMessageVerbHandler.java:134) ~[apache-cassandra-2.1.14.jar:2.1.14]
>>> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>> ... 3 common frames omitted
>>>
>>> Is it a network problem or we are having a bug ? There is nothing else
>>> in the log of cassandra more than
>>> Sync failed between.
>>>
>>> We are having cassandra 2.1.14
>>>
>>> Best regards
>>>
>>> Jean Carlo
>>>
>>> "The best way to predict the future is to invent it" Alan Kay
>>>
>>
>>
>
Re: Sync failed between in AntiEntropySessions - Repair
Posted by Jean Carlo <je...@gmail.com>.
Hi Paulo
I found the lines, we got an exception "Outgoing stream handler has been
closed" these are
ERROR [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772
StreamSession.java:621 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
Remote peer 192.168.0.36 failed stream session.
INFO [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772
StreamResultFuture.java:180 - [Stream
#c4e79260-5a46-11e6-9993-e11d93fd5b40] Session with /192.168.0.36 is
complete
WARN [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,773
StreamResultFuture.java:207 - [Stream
#c4e79260-5a46-11e6-9993-e11d93fd5b40] Stream failed
ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782
StreamReceiveTask.java:183 - Error applying streamed data:
java.lang.RuntimeException: Outgoing stream handler has been closed
at
org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:179)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782
StreamSession.java:505 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
Streaming error occurred
java.lang.RuntimeException: Outgoing stream handler has been closed
at
org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:179)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
I al looking through the changes files to see if it is a bug fixed in the
2.1.15
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Fri, Aug 5, 2016 at 4:24 PM, Paulo Motta <pa...@gmail.com>
wrote:
> It seems you have a streaming error, look for ERROR statement in the
> streaming classes before that which may give you a more specific root
> cause. In any case, I'd suggest you to upgrade to 2.1.15 as there were a
> couple of streaming fixes on this version that might help.
>
> 2016-08-05 11:15 GMT-03:00 Jean Carlo <je...@gmail.com>:
>
>>
>> Hi guys, Doing a repair I got this error for 2 tokenranges.
>>
>> ERROR [Thread-2499244] 2016-08-04 20:05:24,288 StorageService.java:3068 -
>> Repair session 41e4bab0-5a63-11e6-9993-e11d93fd5b40 for range
>> (487410372471205090,492009442088088379] failed with error
>> org.apache.cassandra.exceptions.RepairExcep
>> tion: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>> (487410372471205090,492009442088088379]] Sync failed between /
>> 192.168.0.144 and /192.168.0.37
>> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
>> org.apache.cassandra.exceptions.RepairException: [repair
>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>> (487410372471205090,492009442088088379]] Sync failed between /
>> 192.168.0.144 and /192.168.0.37
>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> [na:1.8.0_60]
>> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> [na:1.8.0_60]
>> at org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:3059)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>> [apache-cassandra-2.1.14.jar:2.1.14]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> [na:1.8.0_60]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> [na:1.8.0_60]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>> Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException:
>> [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>> (487410372471205090,492009442088088379]] Sync failed between /
>> 10.234.72.144 and
>> /10.234.86.37
>> at com.google.common.base.Throwables.propagate(Throwables.java:160)
>> ~[guava-16.0.jar:na]
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
>> [apache-cassandra-2.1.14.jar:2.1.14]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> [na:1.8.0_60]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> [na:1.8.0_60]
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> ~[na:1.8.0_60]
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> ~[na:1.8.0_60]
>> ... 1 common frames omitted
>> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>> (487410372471205090,492009442088088379]] Sync failed between /
>> 192.168.0.144 and /192.168.0.37
>> at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at org.apache.cassandra.service.ActiveRepairService.handleMessa
>> ge(ActiveRepairService.java:422) ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(
>> RepairMessageVerbHandler.java:134) ~[apache-cassandra-2.1.14.jar:2.1.14]
>> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>> ... 3 common frames omitted
>>
>> Is it a network problem or we are having a bug ? There is nothing else in
>> the log of cassandra more than
>> Sync failed between.
>>
>> We are having cassandra 2.1.14
>>
>> Best regards
>>
>> Jean Carlo
>>
>> "The best way to predict the future is to invent it" Alan Kay
>>
>
>
Re: Sync failed between in AntiEntropySessions - Repair
Posted by Paulo Motta <pa...@gmail.com>.
It seems you have a streaming error, look for ERROR statement in the
streaming classes before that which may give you a more specific root
cause. In any case, I'd suggest you to upgrade to 2.1.15 as there were a
couple of streaming fixes on this version that might help.
2016-08-05 11:15 GMT-03:00 Jean Carlo <je...@gmail.com>:
>
> Hi guys, Doing a repair I got this error for 2 tokenranges.
>
> ERROR [Thread-2499244] 2016-08-04 20:05:24,288 StorageService.java:3068 -
> Repair session 41e4bab0-5a63-11e6-9993-e11d93fd5b40 for range
> (487410372471205090,492009442088088379] failed with error
> org.apache.cassandra.exceptions.RepairExcep
> tion: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
> (487410372471205090,492009442088088379]] Sync failed between /
> 192.168.0.144 and /192.168.0.37
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> org.apache.cassandra.exceptions.RepairException: [repair
> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, (487410372471205090,492009442088088379]]
> Sync failed between /192.168.0.144 and /192.168.0.37
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> [na:1.8.0_60]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> [na:1.8.0_60]
> at org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:3059)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> [apache-cassandra-2.1.14.jar:2.1.14]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_60]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException:
> [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
> (487410372471205090,492009442088088379]] Sync failed between /
> 10.234.72.144 and
> /10.234.86.37
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> ~[guava-16.0.jar:na]
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> [apache-cassandra-2.1.14.jar:2.1.14]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_60]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_60]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_60]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_60]
> ... 1 common frames omitted
> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv, (487410372471205090,492009442088088379]]
> Sync failed between /192.168.0.144 and /192.168.0.37
> at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:422)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(
> RepairMessageVerbHandler.java:134) ~[apache-cassandra-2.1.14.jar:2.1.14]
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-2.1.14.jar:2.1.14]
> ... 3 common frames omitted
>
> Is it a network problem or we are having a bug ? There is nothing else in
> the log of cassandra more than
> Sync failed between.
>
> We are having cassandra 2.1.14
>
> Best regards
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>