You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Simon Ashley (JIRA)" <ji...@apache.org> on 2015/12/17 08:46:46 UTC

[jira] [Commented] (CASSANDRA-10887) Pending range calculator gives wrong pending ranges for moves

    [ https://issues.apache.org/jira/browse/CASSANDRA-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15061671#comment-15061671 ] 

Simon Ashley commented on CASSANDRA-10887:
------------------------------------------

Ran a repro on 2.0.16 based on the description

(1)

create keyspace ks1 WITH replication = {'class': 'NetworkTopologyStrategy', 'datacenter1':3}

(2)

CREATE TABLE ks1.users (
  login varchar,
  email varchar,
  name varchar,
  login_count int,
  PRIMARY KEY (login));

(3)

INSERT INTO ks1.users (login, email, name, login_count) VALUES ('jdoe', 'jdoe@abc.com', 'Jane Doe', 1) IF NOT EXISTS;

(4)

#Run a simple lwt in a loop

cat lwtloop.sh

#!/bin/bash

#loop $1 times 

for (( a=1; a<=$1; a++ ))
do
 cqlsh -e "UPDATE ks1.users SET email = 'janedoe@abc.com' WHERE login = 'jdoe' IF email = 'jdoe@abc.com';"
 echo count: $a
done

./lwtloop.sh 1000

This outputs the following with a count

 [applied] | email
-----------+-----------------
     False | janedoe@abc.com

count: 304

(5) Initiate a nodetool move command

 nodetool -h 1.2.3.4 move -- \\-634023222112864484

(6) kill node running move against

sudo kill -9 <cassandra pid>

(7) lwtloop.sh script reports

<stdin>:1:Request did not complete within rpc_timeout.
count: 305

followed by

<stdin>:1:Unable to complete request: one or more nodes were unavailable.
count: 322

(8) Bring node back online

sudo cassandra

lwtloop.sh script now continues

[applied] | email
-----------+-----------------
     False | janedoe@abc.com

count: 587



> Pending range calculator gives wrong pending ranges for moves
> -------------------------------------------------------------
>
>                 Key: CASSANDRA-10887
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10887
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Coordination
>            Reporter: Richard Low
>            Priority: Critical
>
> My understanding is the PendingRangeCalculator is meant to calculate who should receive extra writes during range movements. However, it adds the wrong ranges for moves. An extreme example of this can be seen in the following reproduction. Create a 5 node cluster (I did this on 2.0.16 and 2.2.4) and a keyspace RF=3 and a simple table. Then start moving a node and immediately kill -9 it. Now you see a node as down and moving in the ring. Try a quorum write for a partition that is stored on that node - it will fail with a timeout. Further, all CAS reads or writes fail immediately with unavailable exception because they attempt to include the moving node twice. This is likely to be the cause of CASSANDRA-10423.
> In my example I had this ring:
> 127.0.0.1  rack1       Up     Normal  170.97 KB       20.00%              -9223372036854775808
> 127.0.0.2  rack1       Up     Normal  124.06 KB       20.00%              -5534023222112865485
> 127.0.0.3  rack1       Down   Moving  108.7 KB        40.00%              1844674407370955160
> 127.0.0.4  rack1       Up     Normal  142.58 KB       0.00%               1844674407370955161
> 127.0.0.5  rack1       Up     Normal  118.64 KB       20.00%              5534023222112865484
> Node 3 was moving to -1844674407370955160. I added logging to print the pending and natural endpoints. For ranges owned by node 3, node 3 appeared in pending and natural endpoints. The blockFor is increased to 3 so we’re effectively doing CL.ALL operations. This manifests as write timeouts and CAS unavailables when the node is down.
> The correct pending range for this scenario is node 1 is gaining the range (-1844674407370955160, 1844674407370955160). So node 1 should be added as a destination for writes and CAS for this range, not node 3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)