You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Victor Trac (JIRA)" <ji...@apache.org> on 2016/06/08 21:43:20 UTC
[jira] [Created] (CASSANDRA-11982) Cassandra 3.5 cluster join fails
Victor Trac created CASSANDRA-11982:
---------------------------------------
Summary: Cassandra 3.5 cluster join fails
Key: CASSANDRA-11982
URL: https://issues.apache.org/jira/browse/CASSANDRA-11982
Project: Cassandra
Issue Type: Bug
Environment: # java -version
openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
# uname -a
Linux cassandra-dfs-10-10-160-19 4.4.11-23.53.amzn1.x86_64 #1 SMP Wed Jun 1 22:22:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
# rpm -qa datastax-ddc
datastax-ddc-3.5.0-1.noarch
Reporter: Victor Trac
In an effort to work around a failing upgradesstables/scrub (https://issues.apache.org/jira/browse/CASSANDRA-11981), I force-removed a node from the cluster, deleted its data directory, and tried to join a node to the cluster anew. After a few hours, the bootstrap thread failed:
cassandra.log
{code}
INFO 18:56:24 Redistributing index summaries
INFO 19:56:23 Saved KeyCache (17 items) in 386 ms
INFO 19:56:24 Redistributing index summaries
ERROR 20:40:27 [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Remote peer 10.10.160.18 failed stream session.
INFO 20:40:27 [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Session with /10.10.160.18 is complete
WARN 20:40:27 [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Stream failed
ERROR 20:40:27 Error while waiting on bootstrap to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na]
at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1387) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:974) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:748) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:349) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.5.0.jar:3.5.0]
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.5.0.jar:3.5.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274) ~[apache-cassandra-3.5.0.jar:3.5.0]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
Jun 08, 2016 8:40:27 PM com.google.common.util.concurrent.ExecutionList executeListener
SEVERE: RuntimeException while executing runnable com.google.common.util.concurrent.Futures$6@3592d1b5 with executor INSTANCE
java.lang.NullPointerException
at org.apache.cassandra.service.StorageService$2.onFailure(StorageService.java:1382)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213)
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189)
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636)
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489)
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
at java.lang.Thread.run(Thread.java:745)
WARN 20:40:27 Some data streaming failed. Use nodetool to check bootstrap state and resume. For more, see `nodetool help bootstrap`. IN_PROGRESS
INFO 20:40:27 Waiting for gossip to settle before accepting client requests...
INFO 20:40:35 No gossip backlog; proceeding
{code}
debug.log
{code}
DEBUG [GossipStage:1] 2016-06-08 20:40:27,370 FailureDetector.java:456 - Ignoring interval time of 2001208211 for /10.240.0.4
DEBUG [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,701 ConnectionHandler.java:269 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Received Session Failed
ERROR [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,701 StreamSession.java:635 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Remote peer 10.10.160.18 failed stream session.
DEBUG [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,708 ConnectionHandler.java:110 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Closing stream connection handler on /10.10.160.18
INFO [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,708 StreamResultFuture.java:185 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Session with /10.10.160.18 is complete
DEBUG [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,709 StreamCoordinator.java:146 - Finished connecting all sessions
WARN [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,710 StreamResultFuture.java:212 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Stream failed
ERROR [main] 2016-06-08 20:40:27,711 StorageService.java:1392 - Error while waiting on bootstrap to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na]
at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1387) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:974) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:748) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:349) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.5.0.jar:3.5.0]
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.5.0.jar:3.5.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274) ~[apache-cassandra-3.5.0.jar:3.5.0]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
WARN [main] 2016-06-08 20:40:27,777 StorageService.java:1029 - Some data streaming failed. Use nodetool to check bootstrap state and resume. For more, see `nodetool help bootstrap`. IN_PROGRESS
INFO [main] 2016-06-08 20:40:27,779 CassandraDaemon.java:639 - Waiting for gossip to settle before accepting client requests...
DEBUG [GossipStage:1] 2016-06-08 20:40:31,371 FailureDetector.java:456 - Ignoring interval time of 2256483778 for /10.10.128.18
DEBUG [GossipStage:1] 2016-06-08 20:40:31,371 FailureDetector.java:456 - Ignoring interval time of 2001387252 for /10.10.160.18
{code}
system.log
{code}
INFO [IndexSummaryManager:1] 2016-06-08 19:56:24,495 IndexSummaryRedistribution.java:74 - Redistributing index summaries
ERROR [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,701 StreamSession.java:635 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Remote peer 10.10.160.18 failed stream session.
INFO [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,708 StreamResultFuture.java:185 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Session with /10.10.160.18 is complete
WARN [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,710 StreamResultFuture.java:212 - [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Stream failed
ERROR [main] 2016-06-08 20:40:27,711 StorageService.java:1392 - Error while waiting on bootstrap to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na]
at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1387) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:974) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:748) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:349) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.5.0.jar:3.5.0]
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.5.0.jar:3.5.0]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489) ~[apache-cassandra-3.5.0.jar:3.5.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274) ~[apache-cassandra-3.5.0.jar:3.5.0]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
WARN [main] 2016-06-08 20:40:27,777 StorageService.java:1029 - Some data streaming failed. Use nodetool to check bootstrap state and resume. For more, see `nodetool help bootstrap`. IN_PROGRESS
INFO [main] 2016-06-08 20:40:27,779 CassandraDaemon.java:639 - Waiting for gossip to settle before accepting client requests...
INFO [main] 2016-06-08 20:40:35,780 CassandraDaemon.java:670 - No gossip backlog; proceeding
INFO [main] 2016-06-08 20:40:35,876 NativeTransportService.java:70 - Netty using native Epoll event loop
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)