You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Yixin Zou (Jira)" <ji...@apache.org> on 2021/03/09 19:50:00 UTC
[jira] [Comment Edited] (CASSANDRA-16507) Node unreachable to each
other with MUTATION message dropped
[ https://issues.apache.org/jira/browse/CASSANDRA-16507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298289#comment-17298289 ]
Yixin Zou edited comment on CASSANDRA-16507 at 3/9/21, 7:49 PM:
----------------------------------------------------------------
We did an experiment to test how cassandra behaves when some messages are dropped. Our experiment was:
Inside of cassandra/src/java/org/apache/cassandra/net/MessagingService.java, when some condition is met, return the sendOneWay function before message is sent.
We have three nodes. To mimic mutation message loss between node 1 and node 2,3. We did:
On node 1, add code : when message.verb == Verb.MUTATION && (to == node2_address || to == node3_address) return the function
On node 2, 3, add code : when message.verb == Verb.MUTATION && to == node1_address) return the function
In such a way, we tried to mimi mutation message drop between nodes. At the same time, inserted same key, different values on node 1 and node 2.
What we observed:
# nodes disconnected from each other, as shown in above ./nodetool describecluster result
# read that key on node1,2, got different values, which is inconsistency
was (Author: yixin_zou):
We did an experiment to test how cassandra behaves when some messages are dropped. Our experiment was:
Inside of cassandra/src/java/org/apache/cassandra/net/MessagingService.java, when some condition is met, return the sendOneWay function before message is sent.
We have three nodes. To mimic mutation message drop between node 1 and node 2,3. We did:
On node 1, add code : when message.verb == Verb.MUTATION && (to == node2_address || to == node3_address) return the function
On node 2, 3, add code : when message.verb == Verb.MUTATION && to == node1_address) return the function
In such a way, we tried to mimi mutation message drop between nodes. At the same time, inserted same key, different values on node 1 and node 2.
What we observed:
# nodes disconnected from each other, as shown in above ./nodetool describecluster result
# read that key on node1,2, got different values, which is inconsistency
> Node unreachable to each other with MUTATION message dropped
> ------------------------------------------------------------
>
> Key: CASSANDRA-16507
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16507
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Internode
> Reporter: Yixin Zou
> Priority: Normal
>
> Cluster setup: Ubuntu 18.04-LTS, cassandra-3.11.10. One DC, 3 nodes, 2 replication factor;
> Using an injector to filter and drop MUTATION message between one node and the other two (drop MUTATION messages sent from node 1 to node 2, 3 and drop MUTATION messages sent from node 2, 3 to node 1). At the same time, inserted same key, different values on node 1 and node 2. When insertion happened, MUTATION message drops were caught on both node 1 and node2. Then we observed the different values for that key on two nodes and observed all three nodes disconnected from each other.
>
> {code:java}
> ./nodetool describecluster
> /Cluster Information:Cluster Information:
> Name: Test Cluster Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch DynamicEndPointSnitch: enabled Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions: UNREACHABLE: [10.0.1.8, 10.0.1.7, 10.0.1.9]
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org