[jira] [Commented] (CASSANDRA-15433) Pending ranges are not recalculated on keyspace creation

["Alex Petrov (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17015172#comment-17015172 ] 

Alex Petrov commented on CASSANDRA-15433:
-----------------------------------------

Thank you for the patch.

Following up an offline discussion, I've submitted a patch that you can use to create a test for this patch (see dependent jira) that could also help to simplify further development of features related to bootstrap and ring movements.

While what you are suggesting seems to mitigate the problem I'm not fully certain that this is what we need to do. First, pending range tasks will be triggered for _all_ keyspaces, which is definitely not what we want to do. From looking at the code, it seems like for newly created keyspaces, we do not calculate pending ranges at all, so I'd recompute them for one of the keyspaces (namely, for freshly created one). 

Moreover, keyspaces diff is more than just created keyspaces. It also contains created and altered ones, so triggering recompute without a regard to what exactly happened to which keyspace is also not what we'd like.

To summarise, I'd say we need to add a test for bootstrap + keyspace creation; technically it'll also cover moving and leaving nodes since we also use pending ranges there. And we need to make sure we recompute pending ranges only for the keyspaces that were created. 

> Pending ranges are not recalculated on keyspace creation
> --------------------------------------------------------
>
>                 Key: CASSANDRA-15433
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-15433
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Cluster/Membership
>            Reporter: Josh Snyder
>            Assignee: Sumanth Pasupuleti
>            Priority: Normal
>
> When a node begins bootstrapping, Cassandra recalculates pending tokens for each keyspace that exists when the state change is observed (in StorageService:handleState*). When new keyspaces are created, we do not recalculate pending ranges (around Schema:merge). As a result, writes for new keyspaces are not received by nodes in BOOT or BOOT_REPLACE modes. When bootstrapping finishes, the node which just bootstrapped will not have data for the newly created keyspace.
> Consider a ring with bootstrapped nodes A, B, and C. Node D is pending, and when it finishes bootstrapping, C will cede ownership of some ranges to D. A quorum write is acknowledged by C and A. B missed the write, and the coordinator didn't send it to D at all. When D finishes bootstrapping, the quorum B+D will not contain the mutation.
> Steps to reproduce:
> # Join a node in BOOT mode
> # Create a keyspace
> # Send writes to that keyspace
> # On the joining node, observe that {{nodetool cfstats}} records zero writes to the new keyspace
> I have observed this directly in Cassandra 3.0, and based on my reading the code, I believe it affects up through trunk.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org