[jira] [Created] (CASSANDRA-19154) Allow for easy and safe ways to mutate metadata for operators

["Alex Petrov (Jira)" <ji...@apache.org>]

Alex Petrov created CASSANDRA-19154:
---------------------------------------

             Summary: Allow for easy and safe ways to mutate metadata for operators
                 Key: CASSANDRA-19154
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-19154
             Project: Cassandra
          Issue Type: Improvement
            Reporter: Alex Petrov


Currently, we have two scenarios when the cluster can be put in a (very) bad state by a trivial mistake:
1. When forcing a snapshot, operator may accidentally force a lower _epoch_. If a lower epoch is forced, we will have a log that contains two epochs that are in different periods:
{epoch=23, period=11, kind=FORCE_SNAPSHOT}
{epoch=23, period=8, kind=CUSTOM}
2. When forcing a snapshot, operator may accidentally force a lower _period_, which seems to be even more dangerous. In this case, we will create a new period that immediately follows the last period in the log, but the cluster metadata that is forced in this epoch will have the period that was forced. All subsequent attempts to commit in such cluster will be unsuccesful, since the highest epoch of the period in question will not match the one that is subsequent to the metadata epoch.


Both cases could be fixed by always forcing the epoch that is subsequent to the one that is being forced. It is debatable, if we should disallow commits with +1000 epochs that are seemingly safe altogether, since they seem to be a safeguard in cases when there are ongoing changes in the cluster. But if we could make a clusterwide “pause” to guarantee we remain on the same epoch, probably we can do even that.

Additionally, we should consider switching from using IF NOT EXISTS for new period creation to some form of UPDATE that would detect the problem we have here. That said, I am not fully certain if we will have NULLs in the conditional for non-existing partitions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org