[jira] [Updated] (CASSANDRA-19152) Improve operator tooling for recovering the cluster metadata log

["Alex Petrov (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CASSANDRA-19152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alex Petrov updated CASSANDRA-19152:
------------------------------------
      Workflow: Copy of Cassandra Default Workflow  (was: Copy of Cassandra Bug Workflow)
    Issue Type: Improvement  (was: Bug)

> Improve operator tooling for recovering the cluster metadata log
> ----------------------------------------------------------------
>
>                 Key: CASSANDRA-19152
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-19152
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Alex Petrov
>            Priority: Normal
>
> There are a handful of ways that the CMS can become permanently unavailable, in which case operators will need to intervene to recover it.
> This may be due to misconfiguration, for example only running with a single instance or having all CMS members in the same DC and losing that DC.
> In these cases, we need to be able to manually promote a non-CMS instance to the CMS, something like:
> 1. disable replication everywhere in the cluster to avoid interference from for example a flapping CMS instance
> 2. dump cluster metadata to a file, modify this cluster metadata to make one of the live instances in the cluster the CMS
> 3. stop the instance from #1, start with -Dcassandra.unsafe_boot_with_cluster_metadata=/tmp/cm.dump
> Perhaps an offline tool to dump/export cluster metadata that could work by just reading from the local table without actually starting the node itself. Such a tool should probably take an epoch param, so we could add a mode/other tool to list the local log table in order to select that epoch and then export it so it can be used to force the cluster to a desired state. Something along the lines of:
> > bin/show_metadata_log
> {code}
> 0 PRE_INIT
> 1 INITIALIZE
> 2 UNSAFE_JOIN 127.0.0.1
> 3 UNSAFE_JOIN 127.0.0.2
> ...
> > bin/reconstruct_metadata --epoch 2 --output /tmp/recovery.dump
> > bin/cassandra -Dcassandra.boot_with_metadata=/tmp/recovery.dump
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org