You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Nir Zilka (JIRA)" <ji...@apache.org> on 2016/11/24 10:48:00 UTC
[jira] [Comment Edited] (CASSANDRA-12905) streaming issues with 3.9
(repair)
[ https://issues.apache.org/jira/browse/CASSANDRA-12905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15692941#comment-15692941 ]
Nir Zilka edited comment on CASSANDRA-12905 at 11/24/16 10:47 AM:
------------------------------------------------------------------
Hi,
I decommissioned 4 nodes from when i opened the ticket, so total of 11 nodes currently running.
nodes list is: 10.101.123.44 , 10.101.123.46 , 10.101.123.62 , 10.101.123.36-39 , 10.101.123.50, 10.101.123.53-55
I started yesterday evening (23th) repair and catched untill now the following errors:
this is from the machine i ran the "nodetool repair" (10.101.123.55), 11 errors:
---- 1 ----
ERROR [RepairJobTask:23] 2016-11-24 00:14:18,271 RepairSession.java:290 - [repair #02b2bb40-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the following error
org.apache.cassandra.exceptions.RepairException: [repair #02b2bb40-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user, [(5374555084199702397,5375907191203787693], (5375907191203787693,5394663393980481083], (-4608172686811846999,-4606297045741692934], (343174083518476177,369373536835281005], (-789474545083012335,-787318207609633720], (-810527869609466298,-789474545083012335], (7618946177842619265,7622080038542632561], (8643901858335832297,8668152326416369637], (332360446019658863,336878679529361124], (-8627477550934015644,-8626441922324893605], (-2577622939633507215,-2575565821746779005], (336878679529361124,343174083518476177], (-3519645500583421982,-3517776702806344635], (-2583420055923926426,-2577622939633507215], (3383191969840261164,3387991562336214241], (5394663393980481083,5411354651959353112], (3387991562336214241,3390236953123836921], (-705797240279985098,-700707451972711675], (-5709906583533929520,-5707271395346631992], (1178486765398617360,1178624428305220059], (-3520515159422145033,-3519645500583421982]]] Sync failed between /10.101.123.54 and /10.101.123.50
at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:462) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:165) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:23] 2016-11-24 00:14:18,272 RepairRunnable.java:261 - Repair session 02b2bb40-b1ba-11e6-a5e3-61594f2eacfb for range [(5374555084199702397,5375907191203787693], (5375907191203787693,5394663393980481083], (-4608172686811846999,-4606297045741692934], (343174083518476177,369373536835281005], (-789474545083012335,-787318207609633720], (-810527869609466298,-789474545083012335], (7618946177842619265,7622080038542632561], (8643901858335832297,8668152326416369637], (332360446019658863,336878679529361124], (-8627477550934015644,-8626441922324893605], (-2577622939633507215,-2575565821746779005], (336878679529361124,343174083518476177], (-3519645500583421982,-3517776702806344635], (-2583420055923926426,-2577622939633507215], (3383191969840261164,3387991562336214241], (5394663393980481083,5411354651959353112], (3387991562336214241,3390236953123836921], (-705797240279985098,-700707451972711675], (-5709906583533929520,-5707271395346631992], (1178486765398617360,1178624428305220059], (-3520515159422145033,-3519645500583421982]] failed with error [repair #02b2bb40-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user, [(5374555084199702397,5375907191203787693], (5375907191203787693,5394663393980481083], (-4608172686811846999,-4606297045741692934], (343174083518476177,369373536835281005], (-789474545083012335,-787318207609633720], (-810527869609466298,-789474545083012335], (7618946177842619265,7622080038542632561], (8643901858335832297,8668152326416369637], (332360446019658863,336878679529361124], (-8627477550934015644,-8626441922324893605], (-2577622939633507215,-2575565821746779005], (336878679529361124,343174083518476177], (-3519645500583421982,-3517776702806344635], (-2583420055923926426,-2577622939633507215], (3383191969840261164,3387991562336214241], (5394663393980481083,5411354651959353112], (3387991562336214241,3390236953123836921], (-705797240279985098,-700707451972711675], (-5709906583533929520,-5707271395346631992], (1178486765398617360,1178624428305220059], (-3520515159422145033,-3519645500583421982]]] Sync failed between /10.101.123.54 and /10.101.123.50
org.apache.cassandra.exceptions.RepairException: [repair #02b2bb40-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user, [(5374555084199702397,5375907191203787693], (5375907191203787693,5394663393980481083], (-4608172686811846999,-4606297045741692934], (343174083518476177,369373536835281005], (-789474545083012335,-787318207609633720], (-810527869609466298,-789474545083012335], (7618946177842619265,7622080038542632561], (8643901858335832297,8668152326416369637], (332360446019658863,336878679529361124], (-8627477550934015644,-8626441922324893605], (-2577622939633507215,-2575565821746779005], (336878679529361124,343174083518476177], (-3519645500583421982,-3517776702806344635], (-2583420055923926426,-2577622939633507215], (3383191969840261164,3387991562336214241], (5394663393980481083,5411354651959353112], (3387991562336214241,3390236953123836921], (-705797240279985098,-700707451972711675], (-5709906583533929520,-5707271395346631992], (1178486765398617360,1178624428305220059], (-3520515159422145033,-3519645500583421982]]] Sync failed between /10.101.123.54 and /10.101.123.50
at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:462) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:165) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 2 -----
ERROR [RepairJobTask:26] 2016-11-24 03:41:30,346 RepairSession.java:290 - [repair #02ed5341-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the following error
org.apache.cassandra.exceptions.RepairException: [repair #02ed5341-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user, [(-1335941665601891325,-1321221006770475620], (-7656219523738612247,-7655796196565388324], (7609334541554158623,7611123200966944805], (4459027606150691982,4474972746587738233], (6002584461649025600,6007853196169892926], (3573722541167411217,3580784726724553369], (-5464989069710829887,-5459049126444114128], (-5459049126444114128,-5449173436278793738], (-1642684260736153313,-1631913262033950119], (-4419311717241338507,-4408863717427302952], (-4005546528790223229,-4000695816476140849], (714787271143415911,718355352027453636], (-7655796196565388324,-7646812566394513626], (-1631913262033950119,-1626800922236335484]]] Sync failed between /10.101.123.37 and /10.101.123.54
at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:462) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:165) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:26] 2016-11-24 03:41:30,347 RepairRunnable.java:261 - Repair session 02ed5341-b1ba-11e6-a5e3-61594f2eacfb for range [(-1335941665601891325,-1321221006770475620], (-7656219523738612247,-7655796196565388324], (7609334541554158623,7611123200966944805], (4459027606150691982,4474972746587738233], (6002584461649025600,6007853196169892926], (3573722541167411217,3580784726724553369], (-5464989069710829887,-5459049126444114128], (-5459049126444114128,-5449173436278793738], (-1642684260736153313,-1631913262033950119], (-4419311717241338507,-4408863717427302952], (-4005546528790223229,-4000695816476140849], (714787271143415911,718355352027453636], (-7655796196565388324,-7646812566394513626], (-1631913262033950119,-1626800922236335484]] failed with error [repair #02ed5341-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user, [(-1335941665601891325,-1321221006770475620], (-7656219523738612247,-7655796196565388324], (7609334541554158623,7611123200966944805], (4459027606150691982,4474972746587738233], (6002584461649025600,6007853196169892926], (3573722541167411217,3580784726724553369], (-5464989069710829887,-5459049126444114128], (-5459049126444114128,-5449173436278793738], (-1642684260736153313,-1631913262033950119], (-4419311717241338507,-4408863717427302952], (-4005546528790223229,-4000695816476140849], (714787271143415911,718355352027453636], (-7655796196565388324,-7646812566394513626], (-1631913262033950119,-1626800922236335484]]] Sync failed between /10.101.123.37 and /10.101.123.54
org.apache.cassandra.exceptions.RepairException: [repair #02ed5341-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user, [(-1335941665601891325,-1321221006770475620], (-7656219523738612247,-7655796196565388324], (7609334541554158623,7611123200966944805], (4459027606150691982,4474972746587738233], (6002584461649025600,6007853196169892926], (3573722541167411217,3580784726724553369], (-5464989069710829887,-5459049126444114128], (-5459049126444114128,-5449173436278793738], (-1642684260736153313,-1631913262033950119], (-4419311717241338507,-4408863717427302952], (-4005546528790223229,-4000695816476140849], (714787271143415911,718355352027453636], (-7655796196565388324,-7646812566394513626], (-1631913262033950119,-1626800922236335484]]] Sync failed between /10.101.123.37 and /10.101.123.54
at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:462) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:165) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 3 -----
ERROR [StreamReceiveTask:37] 2016-11-24 03:51:05,643 StreamSession.java:533 - [Stream #93a1a410-b1e5-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.37
ERROR [STREAM-IN-/10.101.123.37:7000] 2016-11-24 03:51:05,952 StreamSession.java:533 - [Stream #93a1a410-b1e5-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.37
java.net.SocketException: End-of-stream reached
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:69) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:292) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 4 -----
ERROR [STREAM-IN-/10.101.123.54:7000] 2016-11-24 03:57:34,878 StreamSession.java:641 - [Stream #93a574a2-b1e5-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.54 failed stream session.
ERROR [StreamReceiveTask:38] 2016-11-24 03:57:53,161 StreamSession.java:533 - [Stream #93a574a2-b1e5-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.54
----- 5 -----
ERROR [STREAM-IN-/10.101.123.53:7000] 2016-11-24 05:34:55,782 StreamSession.java:641 - [Stream #1233c743-b1e1-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.53 failed stream session.
ERROR [RepairJobTask:27] 2016-11-24 05:34:55,786 RepairSession.java:290 - [repair #02ded451-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the following error
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:642) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:494) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:298) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:27] 2016-11-24 05:34:55,787 RepairRunnable.java:261 - Repair session 02ded451-b1ba-11e6-a5e3-61594f2eacfb for range [(4336322345184841912,4366163876979388475], (-7933579975044282977,-7914796416580058518], (1839892671254449208,1841457557372605571], (1936854791639141672,1938001536070334116], (1164510091499688902,1178486765398617360], (-8158056158024454734,-8152122708824144179], (-1849807793309054700,-1843766048765777468], (-6348948024326794761,-6335446313814509922], (8533993130815635284,8557700176962818555], (-2638878446128576989,-2630418219576054523], (8564018824386122995,8570162500655087152], (8570162500655087152,8580710178918673615], (8557700176962818555,8564018824386122995], (8140631388675412405,8147625422662430664], (4213947160008781931,4214821777286722249], (-8152122708824144179,-8152031883182314791], (8582180673436608178,8586996658256378648], (4366163876979388475,4369191621839595605], (8481593436612459797,8494752028882049497], (2990658207661239276,2997421858321140810], (4207239196271256582,4213947160008781931], (6071207062879792046,6078438592707528242], (-7906539059355975104,-7901946461742072348], (-8403286291733434230,-8396456143995118801], (8046054504348116945,8048439946921860060], (-1785584523196265270,-1774397730504967785], (-5740174840775988273,-5709906583533929520], (-8166983564376894748,-8158056158024454734], (8580710178918673615,8582180673436608178], (-7914796416580058518,-7906539059355975104], (-6349776557775176357,-6348948024326794761], (-2575565821746779005,-2565520330427629056], (8295018274402639934,8299841907912314103]] failed with error Stream failed
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:642) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:494) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:298) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 6 -----
ERROR [StreamReceiveTask:36] 2016-11-24 05:45:04,687 StreamSession.java:533 - [Stream #1233c743-b1e1-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.53
java.lang.RuntimeException: Outgoing stream handler has been closed
at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:145) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:706) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:662) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:246) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 7 -----
ERROR [StreamReceiveTask:81] 2016-11-24 08:53:45,489 StreamSession.java:533 - [Stream #8708c770-b20c-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.37
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses.
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:457) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Keyspace.lambda$apply$74(Keyspace.java:466) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:22] 2016-11-24 08:53:45,493 RepairSession.java:290 - [repair #03204a20-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the following error
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:22] 2016-11-24 08:53:45,494 RepairRunnable.java:261 - Repair session 03204a20-b1ba-11e6-a5e3-61594f2eacfb for range [(7904416027548030517,7904644339187931494], (1992268616217213531,2011387685575474580], (609927302916152607,627634447892460792], (-7646812566394513626,-7644383923352462018], (-4008413746138462176,-4005546528790223229], (6817850835888923053,6818203283758294007], (6534614382736965520,6538035742767863193], (8508956293872390588,8512374867520547024], (-1922781003874961453,-1922011619066347023], (-1272836872170180568,-1271613201248661065], (1976999083225979197,1992268616217213531], (-2405793572890717179,-2402070345340440541], (6533344633441688698,6534614382736965520], (2765121196827852596,2787683730312124458], (6538035742767863193,6548726064957823711], (-7644383923352462018,-7642224985092838839], (3171894574675724789,3172900306782399653], (2197711694648372887,2206781329223605896], (6818203283758294007,6818927700000354155], (4297703982421974284,4305423041633973359], (6530186721828199637,6533344633441688698], (-8756481129378507914,-8750331462660878734], (-7642224985092838839,-7639477375842151880]] failed with error Stream failed
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [STREAM-IN-/10.101.123.37:7000] 2016-11-24 08:53:46,317 StreamSession.java:533 - [Stream #8708c770-b20c-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.37
java.net.SocketException: End-of-stream reached
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:69) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:292) ~[apache-cassandra-3.9.0.jar:3.9.0]
------ 8 ------
ERROR [StreamReceiveTask:41] 2016-11-24 09:17:14,677 StreamSession.java:533 - [Stream #1262c680-b1e1-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.54
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses.
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:457) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Keyspace.lambda$apply$74(Keyspace.java:466) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) ~[apache-cassandra-3.9.0.jar:3.9.0]
ERROR [STREAM-IN-/10.101.123.54:7000] 2016-11-24 09:17:15,017 StreamSession.java:533 - [Stream #1262c680-b1e1-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.54
java.net.SocketException: End-of-stream reached
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:69) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:292) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 9 -----
ERROR [StreamReceiveTask:72] 2016-11-24 09:47:02,248 StreamSession.java:533 - [Stream #b0813591-b204-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.54
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses.
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:457) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Keyspace.lambda$apply$74(Keyspace.java:466) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
INFO [StreamReceiveTask:72] 2016-11-24 09:47:02,249 StreamResultFuture.java:188 - [Stream #b0813591-b204-11e6-a5e3-61594f2eacfb] Session with /10.101.123.54 is complete
WARN [StreamReceiveTask:72] 2016-11-24 09:47:02,249 StreamResultFuture.java:215 - [Stream #b0813591-b204-11e6-a5e3-61594f2eacfb] Stream failed
WARN [RepairJobTask:25] 2016-11-24 09:47:02,265 RepairJob.java:153 - [repair #031881f1-b1ba-11e6-a5e3-61594f2eacfb] radius_log_user sync failed
WARN [MutationStage-1] 2016-11-24 09:47:02,266 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[MutationStage-1,5,main]: {}
java.lang.NullPointerException: null
ERROR [RepairJobTask:25] 2016-11-24 09:47:02,275 RepairSession.java:290 - [repair #031881f1-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the following error
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:25] 2016-11-24 09:47:02,276 RepairRunnable.java:261 - Repair session 031881f1-b1ba-11e6-a5e3-61594f2eacfb for range [(-1774397730504967785,-1762758309398474120], (8147625422662430664,8150788098883189595], (2403112104956782810,2404720327411004342], (6078438592707528242,6082354808100608599], (-1370323679961738923,-1365059347020395178], (7993733835100003655,8015005513539374688], (-1857508277235383784,-1849807793309054700], (-1370928608225827748,-1370323679961738923], (8150788098883189595,8164821449219626475], (8679953928760920469,8689453066878026377], (-3426998630338216696,-3420776174229205444], (3571513977484531056,3573722541167411217], (2395920478339673197,2403112104956782810], (7741141583060715116,7743151848268597969], (5294878291570266517,5298658264359247830], (3533995683509843155,3536422023436685141], (3529213866575892196,3533995683509843155], (8015005513539374688,8019140942924181546]] failed with error Stream failed
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 10 -----
ERROR [StreamReceiveTask:84] 2016-11-24 10:01:35,962 StreamSession.java:533 - [Stream #14bf2f71-b215-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.50
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses.
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:457) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Keyspace.lambda$apply$74(Keyspace.java:466) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:26] 2016-11-24 10:01:35,966 RepairSession.java:290 - [repair #032e05c0-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the following error
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:26] 2016-11-24 10:01:35,966 RepairRunnable.java:261 - Repair session 032e05c0-b1ba-11e6-a5e3-61594f2eacfb for range [(4787371764889471292,4791275868383167575], (7660338039579725082,7682519547658080680], (-7888038863789003009,-7883335201927605979], (7551090506670172891,7557064621539106213], (4203593558336671108,4204711535586868916], (7682529946942857131,7700547538244194596], (7545341063171693903,7547704657293476474], (7547704657293476474,7551090506670172891], (6833522370653895113,6851489965988389441], (1281248143248933867,1283892626126598347], (7682519547658080680,7682529946942857131], (7712069670755918582,7713757639857850164], (-2751798030355460938,-2750674055801082756], (1275546448184112943,1281248143248933867], (2787683730312124458,2789980706087291658], (7700547538244194596,7712069670755918582], (4731219540005886245,4735221596252133393], (-2753266270806071514,-2751798030355460938], (-4340445172605159261,-4330574594611691677]] failed with error Stream failed
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [STREAM-IN-/10.101.123.50:7000] 2016-11-24 10:01:36,713 StreamSession.java:533 - [Stream #14bf2f71-b215-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.50
java.net.SocketException: End-of-stream reached
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:69) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:292) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 11 -----
ERROR [StreamReceiveTask:82] 2016-11-24 11:20:34,030 StreamSession.java:533 - [Stream #873a10a2-b20c-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.44
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses.
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:457) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Keyspace.lambda$apply$74(Keyspace.java:466) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) ~[apache-cassandra-3.9.0.jar:3.9.0]
--------------------------------
errors from node 10.101.123.44 :
ERROR [STREAM-IN-/10.101.123.55:33364] 2016-11-24 11:20:34,031 StreamSession.java:641 - [Stream #873a10a2-b20c-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.55 failed stream session.
--------------------------------
errors from node 10.101.123.46 :
system.log.2016-11-23_1838:ERROR [StreamReceiveTask:183] 2016-11-23 18:36:33,983 StreamSession.java:533 - [Stream #b9289f40-b186-11e6-8990-19a24d4ea05e] Streaming error occurred on session with peer 10.101.123.53
system.log.2016-11-23_1848:ERROR [STREAM-IN-/10.101.123.53:7000] 2016-11-23 18:43:59,827 StreamSession.java:533 - [Stream #b9289f40-b186-11e6-8990-19a24d4ea05e] Streaming error occurred on session with peer 10.101.123.53
---------------------------------
errors from node 10.101.123.36 :
ERROR [StreamReceiveTask:126] 2016-11-23 20:33:23,594 StreamSession.java:533 - [Stream #b8914140-b186-11e6-81f0-a9fa8defd0bb] Streaming error occurred on session with peer 10.101.123.37
---------------------------------
errors from node 10.101.123.37 :
system.log.2016-11-24_0945:ERROR [STREAM-IN-/10.101.123.36:43970] 2016-11-23 20:33:23,595 StreamSession.java:641 - [Stream #b8914140-b186-11e6-81f0-a9fa8defd0bb] Remote peer 10.101.123.36 failed stream session.
system.log.2016-11-24_0945:ERROR [StreamReceiveTask:13] 2016-11-24 03:41:27,162 Keyspace.java:516 - Unknown exception caught while attempting to update MaterializedView! radius.radius_log_user
system.log.2016-11-24_0945:ERROR [StreamReceiveTask:13] 2016-11-24 03:41:27,169 StreamSession.java:533 - [Stream #93a1a410-b1e5-11e6-b34a-11869245b6ff] Streaming error occurred on session with peer 10.101.123.54
system.log.2016-11-24_0945:ERROR [STREAM-OUT-/10.101.123.54:7000] 2016-11-24 03:41:30,343 StreamSession.java:533 - [Stream #93a1a410-b1e5-11e6-b34a-11869245b6ff] Streaming error occurred on session with peer 10.101.123.54
system.log.2016-11-24_0945:ERROR [STREAM-IN-/10.101.123.55:46259] 2016-11-24 03:51:05,653 StreamSession.java:641 - [Stream #93a1a410-b1e5-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.55 failed stream session.
system.log.2016-11-24_0945:ERROR [StreamReceiveTask:12] 2016-11-24 03:51:18,183 Keyspace.java:516 - Unknown exception caught while attempting to update MaterializedView! radius.radius_log_user
system.log.2016-11-24_0945:ERROR [StreamReceiveTask:12] 2016-11-24 03:51:18,191 StreamSession.java:533 - [Stream #93a1a410-b1e5-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
system.log.2016-11-24_0945:ERROR [STREAM-IN-/10.101.123.55:53963] 2016-11-24 08:53:45,490 StreamSession.java:641 - [Stream #8708c770-b20c-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.55 failed stream session.
system.log.2016-11-24_0945:ERROR [StreamReceiveTask:19] 2016-11-24 09:45:19,554 StreamSession.java:533 - [Stream #8708c770-b20c-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
---------------------------------
errors from node 10.101.123.50 :
system.log:ERROR [STREAM-IN-/10.101.123.55:51504] 2016-11-24 10:01:35,962 StreamSession.java:641 - [Stream #14bf2f71-b215-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.55 failed stream session.
system.log:ERROR [StreamReceiveTask:29] 2016-11-24 10:17:21,273 StreamSession.java:533 - [Stream #14bf2f71-b215-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
system.log.2016-11-24_0943:ERROR [STREAM-IN-/10.101.123.54:40081] 2016-11-24 00:14:17,660 StreamSession.java:641 - [Stream #dadda440-b1be-11e6-91dc-6d83e11e0103] Remote peer 10.101.123.54 failed stream session.
system.log.2016-11-24_0943:ERROR [StreamReceiveTask:5] 2016-11-24 01:43:07,057 StreamSession.java:533 - [Stream #dadda440-b1be-11e6-91dc-6d83e11e0103] Streaming error occurred on session with peer 10.101.123.54
---------------------------------
errors from node 10.101.123.53 :
system.log:ERROR [StreamReceiveTask:11] 2016-11-24 05:34:55,782 StreamSession.java:533 - [Stream #1233c743-b1e1-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
system.log:ERROR [STREAM-IN-/10.101.123.55:51141] 2016-11-24 05:34:56,268 StreamSession.java:533 - [Stream #1233c743-b1e1-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
system.log:ERROR [STREAM-IN-/10.101.123.54:7000] 2016-11-24 06:47:39,392 StreamSession.java:641 - [Stream #1265acb0-b1e1-11e6-8dc2-79786d1a296a] Remote peer 10.101.123.54 failed stream session.
system.log:ERROR [StreamReceiveTask:12] 2016-11-24 07:29:10,812 StreamSession.java:533 - [Stream #1265acb0-b1e1-11e6-8dc2-79786d1a296a] Streaming error occurred on session with peer 10.101.123.54
system.log.2016-11-23_1933:ERROR [STREAM-IN-/10.101.123.46:53169] 2016-11-23 18:36:33,983 StreamSession.java:641 - [Stream #b9289f40-b186-11e6-8990-19a24d4ea05e] Remote peer 10.101.123.46 failed stream session.
---------------------------------
errors from node 10.101.123.54 :
system.log:ERROR [StreamReceiveTask:7] 2016-11-24 00:14:17,661 StreamSession.java:533 - [Stream #dadda440-b1be-11e6-91dc-6d83e11e0103] Streaming error occurred on session with peer 10.101.123.50
system.log:ERROR [STREAM-IN-/10.101.123.50:7000] 2016-11-24 00:14:18,508 StreamSession.java:533 - [Stream #dadda440-b1be-11e6-91dc-6d83e11e0103] Streaming error occurred on session with peer 10.101.123.50
system.log:ERROR [STREAM-IN-/10.101.123.37:42462] 2016-11-24 03:41:42,022 StreamSession.java:641 - [Stream #93a1a410-b1e5-11e6-b34a-11869245b6ff] Remote peer 10.101.123.37 failed stream session.
system.log:ERROR [StreamReceiveTask:18] 2016-11-24 03:57:34,863 Keyspace.java:516 - Unknown exception caught while attempting to update MaterializedView! radius.radius_log_user
system.log:ERROR [StreamReceiveTask:18] 2016-11-24 03:57:34,871 StreamSession.java:533 - [Stream #93a574a2-b1e5-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
system.log:ERROR [STREAM-IN-/10.101.123.55:42230] 2016-11-24 03:57:35,179 StreamSession.java:533 - [Stream #93a574a2-b1e5-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
system.log:ERROR [StreamReceiveTask:19] 2016-11-24 06:47:39,391 StreamSession.java:533 - [Stream #1265acb0-b1e1-11e6-8dc2-79786d1a296a] Streaming error occurred on session with peer 10.101.123.53
system.log:ERROR [STREAM-IN-/10.101.123.53:50120] 2016-11-24 06:47:40,371 StreamSession.java:533 - [Stream #1265acb0-b1e1-11e6-8dc2-79786d1a296a] Streaming error occurred on session with peer 10.101.123.53
system.log:ERROR [STREAM-IN-/10.101.123.55:51183] 2016-11-24 09:17:14,678 StreamSession.java:641 - [Stream #1262c680-b1e1-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.55 failed stream session.
system.log:ERROR [StreamReceiveTask:20] 2016-11-24 09:32:22,594 StreamSession.java:533 - [Stream #1262c680-b1e1-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
system.log:ERROR [STREAM-IN-/10.101.123.55:54255] 2016-11-24 09:47:02,249 StreamSession.java:641 - [Stream #b0813591-b204-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.55 failed stream session.
system.log.2016-11-23_1754:ERROR [STREAM-IN-/10.101.123.46:55942] 2016-11-23 17:52:43,889 StreamSession.java:641 - [Stream #b9289f40-b186-11e6-8990-19a24d4ea05e] Remote peer 10.101.123.46 failed stream session.
Thanks,
Nir.
was (Author: zilkanir):
Hi,
I decommissioned 4 nodes from when i opened the ticket, so total of 11 nodes currently running.
nodes list is: 10.101.123.44 , 10.101.123.46 , 10.101.123.62 , 10.101.123.36-39 , 10.101.123.50, 10.101.123.53-55
I started yesterday evening (23th) repair and catched untill now the following errors:
this is from the machine i ran the "nodetool repair" (10.101.123.55), 11 errors:
---- 1 ----
ERROR [RepairJobTask:23] 2016-11-24 00:14:18,271 RepairSession.java:290 - [repair #02b2bb40-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the following error
org.apache.cassandra.exceptions.RepairException: [repair #02b2bb40-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user, [(5374555084199702397,5375907191203787693], (5375907191203787693,5394663393980481083], (-4608172686811846999,-4606297045741692934], (343174083518476177,369373536835281005], (-789474545083012335,-787318207609633720], (-810527869609466298,-789474545083012335], (7618946177842619265,7622080038542632561], (8643901858335832297,8668152326416369637], (332360446019658863,336878679529361124], (-8627477550934015644,-8626441922324893605], (-2577622939633507215,-2575565821746779005], (336878679529361124,343174083518476177], (-3519645500583421982,-3517776702806344635], (-2583420055923926426,-2577622939633507215], (3383191969840261164,3387991562336214241], (5394663393980481083,5411354651959353112], (3387991562336214241,3390236953123836921], (-705797240279985098,-700707451972711675], (-5709906583533929520,-5707271395346631992], (1178486765398617360,1178624428305220059], (-3520515159422145033,-3519645500583421982]]] Sync failed between /10.101.123.54 and /10.101.123.50
at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:462) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:165) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:23] 2016-11-24 00:14:18,272 RepairRunnable.java:261 - Repair session 02b2bb40-b1ba-11e6-a5e3-61594f2eacfb for range [(5374555084199702397,5375907191203787693], (5375907191203787693,5394663393980481083], (-4608172686811846999,-4606297045741692934], (343174083518476177,369373536835281005], (-789474545083012335,-787318207609633720], (-810527869609466298,-789474545083012335], (7618946177842619265,7622080038542632561], (8643901858335832297,8668152326416369637], (332360446019658863,336878679529361124], (-8627477550934015644,-8626441922324893605], (-2577622939633507215,-2575565821746779005], (336878679529361124,343174083518476177], (-3519645500583421982,-3517776702806344635], (-2583420055923926426,-2577622939633507215], (3383191969840261164,3387991562336214241], (5394663393980481083,5411354651959353112], (3387991562336214241,3390236953123836921], (-705797240279985098,-700707451972711675], (-5709906583533929520,-5707271395346631992], (1178486765398617360,1178624428305220059], (-3520515159422145033,-3519645500583421982]] failed with error [repair #02b2bb40-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user, [(5374555084199702397,5375907191203787693], (5375907191203787693,5394663393980481083], (-4608172686811846999,-4606297045741692934], (343174083518476177,369373536835281005], (-789474545083012335,-787318207609633720], (-810527869609466298,-789474545083012335], (7618946177842619265,7622080038542632561], (8643901858335832297,8668152326416369637], (332360446019658863,336878679529361124], (-8627477550934015644,-8626441922324893605], (-2577622939633507215,-2575565821746779005], (336878679529361124,343174083518476177], (-3519645500583421982,-3517776702806344635], (-2583420055923926426,-2577622939633507215], (3383191969840261164,3387991562336214241], (5394663393980481083,5411354651959353112], (3387991562336214241,3390236953123836921], (-705797240279985098,-700707451972711675], (-5709906583533929520,-5707271395346631992], (1178486765398617360,1178624428305220059], (-3520515159422145033,-3519645500583421982]]] Sync failed between /10.101.123.54 and /10.101.123.50
org.apache.cassandra.exceptions.RepairException: [repair #02b2bb40-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user, [(5374555084199702397,5375907191203787693], (5375907191203787693,5394663393980481083], (-4608172686811846999,-4606297045741692934], (343174083518476177,369373536835281005], (-789474545083012335,-787318207609633720], (-810527869609466298,-789474545083012335], (7618946177842619265,7622080038542632561], (8643901858335832297,8668152326416369637], (332360446019658863,336878679529361124], (-8627477550934015644,-8626441922324893605], (-2577622939633507215,-2575565821746779005], (336878679529361124,343174083518476177], (-3519645500583421982,-3517776702806344635], (-2583420055923926426,-2577622939633507215], (3383191969840261164,3387991562336214241], (5394663393980481083,5411354651959353112], (3387991562336214241,3390236953123836921], (-705797240279985098,-700707451972711675], (-5709906583533929520,-5707271395346631992], (1178486765398617360,1178624428305220059], (-3520515159422145033,-3519645500583421982]]] Sync failed between /10.101.123.54 and /10.101.123.50
at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:462) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:165) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 2 -----
ERROR [RepairJobTask:26] 2016-11-24 03:41:30,346 RepairSession.java:290 - [repair #02ed5341-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the following error
org.apache.cassandra.exceptions.RepairException: [repair #02ed5341-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user, [(-1335941665601891325,-1321221006770475620], (-7656219523738612247,-7655796196565388324], (7609334541554158623,7611123200966944805], (4459027606150691982,4474972746587738233], (6002584461649025600,6007853196169892926], (3573722541167411217,3580784726724553369], (-5464989069710829887,-5459049126444114128], (-5459049126444114128,-5449173436278793738], (-1642684260736153313,-1631913262033950119], (-4419311717241338507,-4408863717427302952], (-4005546528790223229,-4000695816476140849], (714787271143415911,718355352027453636], (-7655796196565388324,-7646812566394513626], (-1631913262033950119,-1626800922236335484]]] Sync failed between /10.101.123.37 and /10.101.123.54
at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:462) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:165) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:26] 2016-11-24 03:41:30,347 RepairRunnable.java:261 - Repair session 02ed5341-b1ba-11e6-a5e3-61594f2eacfb for range [(-1335941665601891325,-1321221006770475620], (-7656219523738612247,-7655796196565388324], (7609334541554158623,7611123200966944805], (4459027606150691982,4474972746587738233], (6002584461649025600,6007853196169892926], (3573722541167411217,3580784726724553369], (-5464989069710829887,-5459049126444114128], (-5459049126444114128,-5449173436278793738], (-1642684260736153313,-1631913262033950119], (-4419311717241338507,-4408863717427302952], (-4005546528790223229,-4000695816476140849], (714787271143415911,718355352027453636], (-7655796196565388324,-7646812566394513626], (-1631913262033950119,-1626800922236335484]] failed with error [repair #02ed5341-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user, [(-1335941665601891325,-1321221006770475620], (-7656219523738612247,-7655796196565388324], (7609334541554158623,7611123200966944805], (4459027606150691982,4474972746587738233], (6002584461649025600,6007853196169892926], (3573722541167411217,3580784726724553369], (-5464989069710829887,-5459049126444114128], (-5459049126444114128,-5449173436278793738], (-1642684260736153313,-1631913262033950119], (-4419311717241338507,-4408863717427302952], (-4005546528790223229,-4000695816476140849], (714787271143415911,718355352027453636], (-7655796196565388324,-7646812566394513626], (-1631913262033950119,-1626800922236335484]]] Sync failed between /10.101.123.37 and /10.101.123.54
org.apache.cassandra.exceptions.RepairException: [repair #02ed5341-b1ba-11e6-a5e3-61594f2eacfb on radius/radius_log_user, [(-1335941665601891325,-1321221006770475620], (-7656219523738612247,-7655796196565388324], (7609334541554158623,7611123200966944805], (4459027606150691982,4474972746587738233], (6002584461649025600,6007853196169892926], (3573722541167411217,3580784726724553369], (-5464989069710829887,-5459049126444114128], (-5459049126444114128,-5449173436278793738], (-1642684260736153313,-1631913262033950119], (-4419311717241338507,-4408863717427302952], (-4005546528790223229,-4000695816476140849], (714787271143415911,718355352027453636], (-7655796196565388324,-7646812566394513626], (-1631913262033950119,-1626800922236335484]]] Sync failed between /10.101.123.37 and /10.101.123.54
at org.apache.cassandra.repair.RemoteSyncTask.syncComplete(RemoteSyncTask.java:67) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:211) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:462) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:165) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 3 -----
ERROR [StreamReceiveTask:37] 2016-11-24 03:51:05,643 StreamSession.java:533 - [Stream #93a1a410-b1e5-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.37
ERROR [STREAM-IN-/10.101.123.37:7000] 2016-11-24 03:51:05,952 StreamSession.java:533 - [Stream #93a1a410-b1e5-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.37
java.net.SocketException: End-of-stream reached
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:69) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:292) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 4 -----
ERROR [STREAM-IN-/10.101.123.54:7000] 2016-11-24 03:57:34,878 StreamSession.java:641 - [Stream #93a574a2-b1e5-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.54 failed stream session.
ERROR [StreamReceiveTask:38] 2016-11-24 03:57:53,161 StreamSession.java:533 - [Stream #93a574a2-b1e5-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.54
----- 5 -----
ERROR [STREAM-IN-/10.101.123.53:7000] 2016-11-24 05:34:55,782 StreamSession.java:641 - [Stream #1233c743-b1e1-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.53 failed stream session.
ERROR [RepairJobTask:27] 2016-11-24 05:34:55,786 RepairSession.java:290 - [repair #02ded451-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the following error
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:642) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:494) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:298) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:27] 2016-11-24 05:34:55,787 RepairRunnable.java:261 - Repair session 02ded451-b1ba-11e6-a5e3-61594f2eacfb for range [(4336322345184841912,4366163876979388475], (-7933579975044282977,-7914796416580058518], (1839892671254449208,1841457557372605571], (1936854791639141672,1938001536070334116], (1164510091499688902,1178486765398617360], (-8158056158024454734,-8152122708824144179], (-1849807793309054700,-1843766048765777468], (-6348948024326794761,-6335446313814509922], (8533993130815635284,8557700176962818555], (-2638878446128576989,-2630418219576054523], (8564018824386122995,8570162500655087152], (8570162500655087152,8580710178918673615], (8557700176962818555,8564018824386122995], (8140631388675412405,8147625422662430664], (4213947160008781931,4214821777286722249], (-8152122708824144179,-8152031883182314791], (8582180673436608178,8586996658256378648], (4366163876979388475,4369191621839595605], (8481593436612459797,8494752028882049497], (2990658207661239276,2997421858321140810], (4207239196271256582,4213947160008781931], (6071207062879792046,6078438592707528242], (-7906539059355975104,-7901946461742072348], (-8403286291733434230,-8396456143995118801], (8046054504348116945,8048439946921860060], (-1785584523196265270,-1774397730504967785], (-5740174840775988273,-5709906583533929520], (-8166983564376894748,-8158056158024454734], (8580710178918673615,8582180673436608178], (-7914796416580058518,-7906539059355975104], (-6349776557775176357,-6348948024326794761], (-2575565821746779005,-2565520330427629056], (8295018274402639934,8299841907912314103]] failed with error Stream failed
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:642) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:494) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:298) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 6 -----
ERROR [StreamReceiveTask:36] 2016-11-24 05:45:04,687 StreamSession.java:533 - [Stream #1233c743-b1e1-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.53
java.lang.RuntimeException: Outgoing stream handler has been closed
at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:145) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:706) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:662) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:246) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 7 -----
ERROR [StreamReceiveTask:81] 2016-11-24 08:53:45,489 StreamSession.java:533 - [Stream #8708c770-b20c-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.37
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses.
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:457) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Keyspace.lambda$apply$74(Keyspace.java:466) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:22] 2016-11-24 08:53:45,493 RepairSession.java:290 - [repair #03204a20-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the following error
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:22] 2016-11-24 08:53:45,494 RepairRunnable.java:261 - Repair session 03204a20-b1ba-11e6-a5e3-61594f2eacfb for range [(7904416027548030517,7904644339187931494], (1992268616217213531,2011387685575474580], (609927302916152607,627634447892460792], (-7646812566394513626,-7644383923352462018], (-4008413746138462176,-4005546528790223229], (6817850835888923053,6818203283758294007], (6534614382736965520,6538035742767863193], (8508956293872390588,8512374867520547024], (-1922781003874961453,-1922011619066347023], (-1272836872170180568,-1271613201248661065], (1976999083225979197,1992268616217213531], (-2405793572890717179,-2402070345340440541], (6533344633441688698,6534614382736965520], (2765121196827852596,2787683730312124458], (6538035742767863193,6548726064957823711], (-7644383923352462018,-7642224985092838839], (3171894574675724789,3172900306782399653], (2197711694648372887,2206781329223605896], (6818203283758294007,6818927700000354155], (4297703982421974284,4305423041633973359], (6530186721828199637,6533344633441688698], (-8756481129378507914,-8750331462660878734], (-7642224985092838839,-7639477375842151880]] failed with error Stream failed
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [STREAM-IN-/10.101.123.37:7000] 2016-11-24 08:53:46,317 StreamSession.java:533 - [Stream #8708c770-b20c-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.37
java.net.SocketException: End-of-stream reached
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:69) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:292) ~[apache-cassandra-3.9.0.jar:3.9.0]
------ 8 ------
ERROR [StreamReceiveTask:41] 2016-11-24 09:17:14,677 StreamSession.java:533 - [Stream #1262c680-b1e1-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.54
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses.
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:457) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Keyspace.lambda$apply$74(Keyspace.java:466) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) ~[apache-cassandra-3.9.0.jar:3.9.0]
ERROR [STREAM-IN-/10.101.123.54:7000] 2016-11-24 09:17:15,017 StreamSession.java:533 - [Stream #1262c680-b1e1-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.54
java.net.SocketException: End-of-stream reached
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:69) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:292) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 9 -----
ERROR [StreamReceiveTask:72] 2016-11-24 09:47:02,248 StreamSession.java:533 - [Stream #b0813591-b204-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.54
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses.
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:457) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Keyspace.lambda$apply$74(Keyspace.java:466) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
INFO [StreamReceiveTask:72] 2016-11-24 09:47:02,249 StreamResultFuture.java:188 - [Stream #b0813591-b204-11e6-a5e3-61594f2eacfb] Session with /10.101.123.54 is complete
WARN [StreamReceiveTask:72] 2016-11-24 09:47:02,249 StreamResultFuture.java:215 - [Stream #b0813591-b204-11e6-a5e3-61594f2eacfb] Stream failed
WARN [RepairJobTask:25] 2016-11-24 09:47:02,265 RepairJob.java:153 - [repair #031881f1-b1ba-11e6-a5e3-61594f2eacfb] radius_log_user sync failed
WARN [MutationStage-1] 2016-11-24 09:47:02,266 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[MutationStage-1,5,main]: {}
java.lang.NullPointerException: null
ERROR [RepairJobTask:25] 2016-11-24 09:47:02,275 RepairSession.java:290 - [repair #031881f1-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the following error
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:25] 2016-11-24 09:47:02,276 RepairRunnable.java:261 - Repair session 031881f1-b1ba-11e6-a5e3-61594f2eacfb for range [(-1774397730504967785,-1762758309398474120], (8147625422662430664,8150788098883189595], (2403112104956782810,2404720327411004342], (6078438592707528242,6082354808100608599], (-1370323679961738923,-1365059347020395178], (7993733835100003655,8015005513539374688], (-1857508277235383784,-1849807793309054700], (-1370928608225827748,-1370323679961738923], (8150788098883189595,8164821449219626475], (8679953928760920469,8689453066878026377], (-3426998630338216696,-3420776174229205444], (3571513977484531056,3573722541167411217], (2395920478339673197,2403112104956782810], (7741141583060715116,7743151848268597969], (5294878291570266517,5298658264359247830], (3533995683509843155,3536422023436685141], (3529213866575892196,3533995683509843155], (8015005513539374688,8019140942924181546]] failed with error Stream failed
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 10 -----
ERROR [StreamReceiveTask:84] 2016-11-24 10:01:35,962 StreamSession.java:533 - [Stream #14bf2f71-b215-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.50
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses.
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:457) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Keyspace.lambda$apply$74(Keyspace.java:466) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:26] 2016-11-24 10:01:35,966 RepairSession.java:290 - [repair #032e05c0-b1ba-11e6-a5e3-61594f2eacfb] Session completed with the following error
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [RepairJobTask:26] 2016-11-24 10:01:35,966 RepairRunnable.java:261 - Repair session 032e05c0-b1ba-11e6-a5e3-61594f2eacfb for range [(4787371764889471292,4791275868383167575], (7660338039579725082,7682519547658080680], (-7888038863789003009,-7883335201927605979], (7551090506670172891,7557064621539106213], (4203593558336671108,4204711535586868916], (7682529946942857131,7700547538244194596], (7545341063171693903,7547704657293476474], (7547704657293476474,7551090506670172891], (6833522370653895113,6851489965988389441], (1281248143248933867,1283892626126598347], (7682519547658080680,7682529946942857131], (7712069670755918582,7713757639857850164], (-2751798030355460938,-2750674055801082756], (1275546448184112943,1281248143248933867], (2787683730312124458,2789980706087291658], (7700547538244194596,7712069670755918582], (4731219540005886245,4735221596252133393], (-2753266270806071514,-2751798030355460938], (-4340445172605159261,-4330574594611691677]] failed with error Stream failed
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:192) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:439) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:542) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:251) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
ERROR [STREAM-IN-/10.101.123.50:7000] 2016-11-24 10:01:36,713 StreamSession.java:533 - [Stream #14bf2f71-b215-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.50
java.net.SocketException: End-of-stream reached
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:69) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:292) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
----- 11 -----
ERROR [StreamReceiveTask:82] 2016-11-24 11:20:34,030 StreamSession.java:533 - [Stream #873a10a2-b20c-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.44
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses.
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:457) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.db.Keyspace.lambda$apply$74(Keyspace.java:466) ~[apache-cassandra-3.9.0.jar:3.9.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) ~[apache-cassandra-3.9.0.jar:3.9.0]
--------------------------------
errors from node 10.101.123.46 :
system.log.2016-11-23_1838:ERROR [StreamReceiveTask:183] 2016-11-23 18:36:33,983 StreamSession.java:533 - [Stream #b9289f40-b186-11e6-8990-19a24d4ea05e] Streaming error occurred on session with peer 10.101.123.53
system.log.2016-11-23_1848:ERROR [STREAM-IN-/10.101.123.53:7000] 2016-11-23 18:43:59,827 StreamSession.java:533 - [Stream #b9289f40-b186-11e6-8990-19a24d4ea05e] Streaming error occurred on session with peer 10.101.123.53
---------------------------------
errors from node 10.101.123.36 :
ERROR [StreamReceiveTask:126] 2016-11-23 20:33:23,594 StreamSession.java:533 - [Stream #b8914140-b186-11e6-81f0-a9fa8defd0bb] Streaming error occurred on session with peer 10.101.123.37
---------------------------------
errors from node 10.101.123.37 :
system.log.2016-11-24_0945:ERROR [STREAM-IN-/10.101.123.36:43970] 2016-11-23 20:33:23,595 StreamSession.java:641 - [Stream #b8914140-b186-11e6-81f0-a9fa8defd0bb] Remote peer 10.101.123.36 failed stream session.
system.log.2016-11-24_0945:ERROR [StreamReceiveTask:13] 2016-11-24 03:41:27,162 Keyspace.java:516 - Unknown exception caught while attempting to update MaterializedView! radius.radius_log_user
system.log.2016-11-24_0945:ERROR [StreamReceiveTask:13] 2016-11-24 03:41:27,169 StreamSession.java:533 - [Stream #93a1a410-b1e5-11e6-b34a-11869245b6ff] Streaming error occurred on session with peer 10.101.123.54
system.log.2016-11-24_0945:ERROR [STREAM-OUT-/10.101.123.54:7000] 2016-11-24 03:41:30,343 StreamSession.java:533 - [Stream #93a1a410-b1e5-11e6-b34a-11869245b6ff] Streaming error occurred on session with peer 10.101.123.54
system.log.2016-11-24_0945:ERROR [STREAM-IN-/10.101.123.55:46259] 2016-11-24 03:51:05,653 StreamSession.java:641 - [Stream #93a1a410-b1e5-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.55 failed stream session.
system.log.2016-11-24_0945:ERROR [StreamReceiveTask:12] 2016-11-24 03:51:18,183 Keyspace.java:516 - Unknown exception caught while attempting to update MaterializedView! radius.radius_log_user
system.log.2016-11-24_0945:ERROR [StreamReceiveTask:12] 2016-11-24 03:51:18,191 StreamSession.java:533 - [Stream #93a1a410-b1e5-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
system.log.2016-11-24_0945:ERROR [STREAM-IN-/10.101.123.55:53963] 2016-11-24 08:53:45,490 StreamSession.java:641 - [Stream #8708c770-b20c-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.55 failed stream session.
system.log.2016-11-24_0945:ERROR [StreamReceiveTask:19] 2016-11-24 09:45:19,554 StreamSession.java:533 - [Stream #8708c770-b20c-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
---------------------------------
errors from node 10.101.123.50 :
system.log:ERROR [STREAM-IN-/10.101.123.55:51504] 2016-11-24 10:01:35,962 StreamSession.java:641 - [Stream #14bf2f71-b215-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.55 failed stream session.
system.log:ERROR [StreamReceiveTask:29] 2016-11-24 10:17:21,273 StreamSession.java:533 - [Stream #14bf2f71-b215-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
system.log.2016-11-24_0943:ERROR [STREAM-IN-/10.101.123.54:40081] 2016-11-24 00:14:17,660 StreamSession.java:641 - [Stream #dadda440-b1be-11e6-91dc-6d83e11e0103] Remote peer 10.101.123.54 failed stream session.
system.log.2016-11-24_0943:ERROR [StreamReceiveTask:5] 2016-11-24 01:43:07,057 StreamSession.java:533 - [Stream #dadda440-b1be-11e6-91dc-6d83e11e0103] Streaming error occurred on session with peer 10.101.123.54
---------------------------------
errors from node 10.101.123.53 :
system.log:ERROR [StreamReceiveTask:11] 2016-11-24 05:34:55,782 StreamSession.java:533 - [Stream #1233c743-b1e1-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
system.log:ERROR [STREAM-IN-/10.101.123.55:51141] 2016-11-24 05:34:56,268 StreamSession.java:533 - [Stream #1233c743-b1e1-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
system.log:ERROR [STREAM-IN-/10.101.123.54:7000] 2016-11-24 06:47:39,392 StreamSession.java:641 - [Stream #1265acb0-b1e1-11e6-8dc2-79786d1a296a] Remote peer 10.101.123.54 failed stream session.
system.log:ERROR [StreamReceiveTask:12] 2016-11-24 07:29:10,812 StreamSession.java:533 - [Stream #1265acb0-b1e1-11e6-8dc2-79786d1a296a] Streaming error occurred on session with peer 10.101.123.54
system.log.2016-11-23_1933:ERROR [STREAM-IN-/10.101.123.46:53169] 2016-11-23 18:36:33,983 StreamSession.java:641 - [Stream #b9289f40-b186-11e6-8990-19a24d4ea05e] Remote peer 10.101.123.46 failed stream session.
---------------------------------
errors from node 10.101.123.54 :
system.log:ERROR [StreamReceiveTask:7] 2016-11-24 00:14:17,661 StreamSession.java:533 - [Stream #dadda440-b1be-11e6-91dc-6d83e11e0103] Streaming error occurred on session with peer 10.101.123.50
system.log:ERROR [STREAM-IN-/10.101.123.50:7000] 2016-11-24 00:14:18,508 StreamSession.java:533 - [Stream #dadda440-b1be-11e6-91dc-6d83e11e0103] Streaming error occurred on session with peer 10.101.123.50
system.log:ERROR [STREAM-IN-/10.101.123.37:42462] 2016-11-24 03:41:42,022 StreamSession.java:641 - [Stream #93a1a410-b1e5-11e6-b34a-11869245b6ff] Remote peer 10.101.123.37 failed stream session.
system.log:ERROR [StreamReceiveTask:18] 2016-11-24 03:57:34,863 Keyspace.java:516 - Unknown exception caught while attempting to update MaterializedView! radius.radius_log_user
system.log:ERROR [StreamReceiveTask:18] 2016-11-24 03:57:34,871 StreamSession.java:533 - [Stream #93a574a2-b1e5-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
system.log:ERROR [STREAM-IN-/10.101.123.55:42230] 2016-11-24 03:57:35,179 StreamSession.java:533 - [Stream #93a574a2-b1e5-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
system.log:ERROR [StreamReceiveTask:19] 2016-11-24 06:47:39,391 StreamSession.java:533 - [Stream #1265acb0-b1e1-11e6-8dc2-79786d1a296a] Streaming error occurred on session with peer 10.101.123.53
system.log:ERROR [STREAM-IN-/10.101.123.53:50120] 2016-11-24 06:47:40,371 StreamSession.java:533 - [Stream #1265acb0-b1e1-11e6-8dc2-79786d1a296a] Streaming error occurred on session with peer 10.101.123.53
system.log:ERROR [STREAM-IN-/10.101.123.55:51183] 2016-11-24 09:17:14,678 StreamSession.java:641 - [Stream #1262c680-b1e1-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.55 failed stream session.
system.log:ERROR [StreamReceiveTask:20] 2016-11-24 09:32:22,594 StreamSession.java:533 - [Stream #1262c680-b1e1-11e6-a5e3-61594f2eacfb] Streaming error occurred on session with peer 10.101.123.55
system.log:ERROR [STREAM-IN-/10.101.123.55:54255] 2016-11-24 09:47:02,249 StreamSession.java:641 - [Stream #b0813591-b204-11e6-a5e3-61594f2eacfb] Remote peer 10.101.123.55 failed stream session.
system.log.2016-11-23_1754:ERROR [STREAM-IN-/10.101.123.46:55942] 2016-11-23 17:52:43,889 StreamSession.java:641 - [Stream #b9289f40-b186-11e6-8990-19a24d4ea05e] Remote peer 10.101.123.46 failed stream session.
Thanks,
Nir.
> streaming issues with 3.9 (repair)
> ----------------------------------
>
> Key: CASSANDRA-12905
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12905
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: centos 6.7 x86_64
> Reporter: Nir Zilka
> Fix For: 3.9
>
>
> Hello,
> I performed two upgrades to the current cluster (currently 15 nodes, 1 DC, private VLAN),
> first it was 2.2.5.1 and repair worked flawlessly,
> second upgrade was to 3.0.9 (with upgradesstables) and also repair worked well,
> then i upgraded 2 weeks ago to 3.9 - and the repair problems started.
> there are several errors types from the system.log (different nodes) :
> - Sync failed between /xxx.xxx.xxx.xxx and /xxx.xxx.xxx.xxx
> - Streaming error occurred on session with peer xxx.xxx.xxx.xxx Operation timed out - received only 0 responses
> - Remote peer xxx.xxx.xxx.xxx failed stream session
> - Session completed with the following error
> org.apache.cassandra.streaming.StreamException: Stream failed
> ----
> i use 3.9 default configuration with the cluster settings adjustments (3 seeds, GossipingPropertyFileSnitch).
> streaming_socket_timeout_in_ms is the default (86400000).
> i'm afraid from consistency problems while i'm not performing repair.
> Any ideas?
> Thanks,
> Nir.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)