You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (Jira)" <ji...@apache.org> on 2021/04/09 13:42:00 UTC
[jira] [Comment Edited] (CASSANDRA-16577) Node waits for schema
agreement on removed nodes
[ https://issues.apache.org/jira/browse/CASSANDRA-16577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17317992#comment-17317992 ]
Brandon Williams edited comment on CASSANDRA-16577 at 4/9/21, 1:42 PM:
-----------------------------------------------------------------------
Updated [detst|https://github.com/driftx/cassandra-dtest/tree/CASSANDRA-16577] and [patch|https://github.com/driftx/cassandra/tree/CASSANDRA-16577].
[!https://ci-cassandra.apache.org/job/Cassandra-devbranch/599/badge/icon!|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/599/pipeline]
was (Author: brandon.williams):
Updated [detst|https://github.com/driftx/cassandra-dtest/tree/CASSANDRA-16577] and [patch|https://github.com/driftx/cassandra/tree/CASSANDRA-16577].
> Node waits for schema agreement on removed nodes
> ------------------------------------------------
>
> Key: CASSANDRA-16577
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16577
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Gossip, Consistency/Bootstrap and Decommission
> Reporter: Jan Karlsson
> Assignee: Brandon Williams
> Priority: Normal
> Fix For: 4.0, 3.0.x, 3.11.x, 4.0-beta
>
>
> CASSANDRA-15158 might have introduced a bug where bootstrapping nodes wait for schema agreement from nodes that have been removed if token allocation for keyspace is enabled.
>
> It is fairly easy to reproduce with the following steps:
> {noformat}
> // Create 3 node cluster
> ccm create test --vnodes -n 3 -s -v 3.11.10
> // Remove two nodes
> ccm node2 decommission
> ccm node3 decommission
> ccm node2 remove
> ccm node3 remove
> // Create keyspace to change the schema. It works if the schema never changes.
> ccm node1 cqlsh -x "CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};"
> // Add allocate parameter
> ccm updateconf 'allocate_tokens_for_keyspace: k'
> // Add node2 again to cluster
> ccm add node2 -i 127.0.0.2 -j 7200 -r 2200
> ccm node2 start{noformat}
>
> This will cause node2 to throw exception on startup:
> {noformat}
> WARN [main] 2021-04-08 14:10:53,272 StorageService.java:941 - There are nodes in the cluster with a different schema version than us we did not merged schemas from, our version : (a5da47ec-ffe3-3111-b2f3-325f771f1539), outstanding versions -> endpoints : {8e9ec79e-5ed2-3949-8ac8-794abfee3837=[/127.0.0.3]}
> ERROR [main] 2021-04-08 14:10:53,274 CassandraDaemon.java:803 - Exception encountered during startup
> java.lang.RuntimeException: Didn't receive schemas for all known versions within the timeout
> at org.apache.cassandra.service.StorageService.waitForSchema(StorageService.java:947) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.dht.BootStrapper.allocateTokens(BootStrapper.java:206) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.dht.BootStrapper.getBootstrapTokens(BootStrapper.java:177) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1073) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:753) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:687) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:395) [apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:633) [apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:786) [apache-cassandra-3.11.10.jar:3.11.10]
> INFO [StorageServiceShutdownHook] 2021-04-08 14:10:53,279 HintsService.java:209 - Paused hints dispatch
> WARN [StorageServiceShutdownHook] 2021-04-08 14:10:53,280 Gossiper.java:1670 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
> INFO [StorageServiceShutdownHook] 2021-04-08 14:10:53,280 MessagingService.java:985 - Waiting for messaging service to quiesce
> INFO [ACCEPT-/127.0.0.2] 2021-04-08 14:10:53,281 MessagingService.java:1346 - MessagingService has terminated the accept() thread
> INFO [StorageServiceShutdownHook] 2021-04-08 14:10:53,416 HintsService.java:209 - Paused hints dispatch{noformat}
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org