[jira] [Commented] (CASSANDRA-14083) Avoid invalidating disk boundaries unnecessarily

["Paulo Motta (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272918#comment-16272918 ] 

Paulo Motta commented on CASSANDRA-14083:
-----------------------------------------

After doing the trivial change of only invalidating disk boundaries when the replication settings change, {{disk_balance_test.py:TestDiskBalance.disk_balance_bootstrap_test}} started failing with imbalanced disks.

After investigation, it turned out that when the node starts bootstrapping, it doesn't have any information about itself on {{TokenMetadata}}, so the disk boundaries will be empty. When the node adds itself to gossip, the cached ring version does not change, so the disk boundaries is never invalidated what affects the disk balance.

This test was not failing before this change because during keyspace initialization, the disk boundaries were being invalidated by [Keyspace.setMetatada|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/Keyspace.java#L187], and properly reloaded with the correct boundaries during streaming - but if some consumer read the disk boundaries before it was set by the bootstrap process, it would cache an older version.

My simple fix simply [invalidates the cached ring|https://github.com/pauloricardomg/cassandra/commit/fb66c3c451caec936447929f45be3c5f90725a48] after the node is added as bootstrapping to gossip, but this will also invalidate cached rings unnecessarily only to invalidate the disk boundaries. Perhaps we could decouple the cached ring version from the actual ring version which takes into account pending node changes (bootstrapping, leaving)?

Patch:
* [3.11|https://github.com/pauloricardomg/cassandra/tree/3.11-14083]

Since this depends on CASSANDRA-13948, I will wait until that is committed before setting this as PA.

> Avoid invalidating disk boundaries unnecessarily
> ------------------------------------------------
>
>                 Key: CASSANDRA-14083
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14083
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Paulo Motta
>            Assignee: Paulo Motta
>
> We currently invalidate disk boundaries whenever [instantiating a new replication strategy|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/Keyspace.java#L359], but this is done whenever [updating keyspace settings|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/Keyspace.java#L187]. 
> Computing new boundaries is expensive and unnecessarily invalidating them will cause {{DiskBoundaries}} consumers to also invalidate their work unnecessarily. For instance, after CASSANDRA-13948 the {{CompactionStrategyManager}} will reload all compaction strategies when the boundaries are invalidated.
> In this case, we should only invalidate the disk boundaries when the replication settings change to avoid doing unnecessary work.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org