You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Paulo Motta (JIRA)" <ji...@apache.org> on 2017/03/29 17:30:41 UTC
[jira] [Comment Edited] (CASSANDRA-13327) Pending endpoints size
check for CAS doesn't play nicely with writes-on-replacement
[ https://issues.apache.org/jira/browse/CASSANDRA-13327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15947564#comment-15947564 ]
Paulo Motta edited comment on CASSANDRA-13327 at 3/29/17 5:29 PM:
------------------------------------------------------------------
I'm a bit confused here, did CAS unavailable happened while 127.0.0.4 was being replaced, or did 127.0.0.5 remained in JOINING/replacing state after replace was finished?
If the former, than this is expected behavior I guess? Because you had 3 normal endpoints (where 1 was down) and 2 pending endpoints (the bootstrapping and the replacing node) for the requested key so the CAS should not allowed due to CASSANDRA-8346.
If the latter than this is a bug with replace and must be fixed since the node must bump to NORMAL state after replacement is completed and the CAS should succeed. Can you reproduce this easily or have logs to understand why the replacement node did not go into NORMAL state after replacement was finished?
was (Author: pauloricardomg):
I'm a bit confused here, did CAS unavailable happened while 127.0.0.5 was being replaced, or did 127.0.0.5 remained in JOINING/replacing state after replace was finished?
If the former, than this is expected behavior I guess? Because you had 3 normal endpoints (where 1 was down) and 2 pending endpoints (the bootstrapping and the replacing node) for the requested key so the CAS should not allowed due to CASSANDRA-8346.
If the latter than this is a bug with replace and must be fixed since the node must bump to NORMAL state after replacement is completed and the CAS should succeed. Can you reproduce this easily or have logs to understand why the replacement node did not go into NORMAL state after replacement was finished?
> Pending endpoints size check for CAS doesn't play nicely with writes-on-replacement
> -----------------------------------------------------------------------------------
>
> Key: CASSANDRA-13327
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13327
> Project: Cassandra
> Issue Type: Bug
> Components: Coordination
> Reporter: Ariel Weisberg
> Assignee: Ariel Weisberg
>
> Consider this ring:
> 127.0.0.1 MR UP JOINING -7301836195843364181
> 127.0.0.2 MR UP NORMAL -7263405479023135948
> 127.0.0.3 MR UP NORMAL -7205759403792793599
> 127.0.0.4 MR DOWN NORMAL -7148113328562451251
> where 127.0.0.1 was bootstrapping for cluster expansion. Note that, due to the failure of 127.0.0.4, 127.0.0.1 was stuck trying to stream from it and making no progress.
> Then the down node was replaced so we had:
> 127.0.0.1 MR UP JOINING -7301836195843364181
> 127.0.0.2 MR UP NORMAL -7263405479023135948
> 127.0.0.3 MR UP NORMAL -7205759403792793599
> 127.0.0.5 MR UP JOINING -7148113328562451251
> It’s confusing in the ring - the first JOINING is a genuine bootstrap, the second is a replacement. We now had CAS unavailables (but no non-CAS unvailables). I think it’s because the pending endpoints check thinks that 127.0.0.5 is gaining a range when it’s just replacing.
> The workaround is to kill the stuck JOINING node, but Cassandra shouldn’t unnecessarily fail these requests.
> It also appears like required participants is bumped by 1 during a host replacement so if the replacing host fails you will get unavailables and timeouts.
> This is related to the check added in CASSANDRA-8346
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)