[jira] [Created] (HDDS-16924) SCM that installed a Ratis snapshot in process keeps serving checkpoints from the closed DB, so downloads from it fail once it is leader

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-16924:
---------------------------------

             Summary: SCM that installed a Ratis snapshot in process keeps serving checkpoints from the closed DB, so downloads from it fail once it is leader
                 Key: HDDS-16924
                 URL: https://issues.apache.org/jira/browse/HDDS-16924
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: MC-4-serve-scm-checkpoints-from-current-db-store.patch, TestBugMC4CheckpointServiceClosedStore.java

h3. Mechanism
{{InterSCMGrpcService}} takes the transaction info table and an {{SCMDBCheckpointProvider}} from {{scm.getScmMetadataStore().getStore()}} once, in its constructor, and the service is created once per process in the {{SCMHAManagerImpl}} constructor. {{SCMHAManagerImpl.installCheckpoint}} closes that store ({{stopServices}}, which runs {{SCMMetadataStoreImpl.stop}}) and {{startServices}} opens a new one and points the managers at it, but the gRPC service is not updated. From then on every {{InterSCMGrpcService.download}} reads the closed RocksDB and fails with {{RocksDatabaseException: Rocks Database is closed}}. Only a restart of that SCM clears it.

The javadoc of {{SCMHAManagerImpl.reloadSCMState}} says that all classes which use or store the metadata manager should be updated with the new instance; this one is missed.

h3. Trigger
# An SCM installs a checkpoint in process. With default settings this happens on every SCM added with {{--bootstrap}}: Ratis (3.3.1) sends a snapshot notification to a bootstrapping follower once, independent of log purging, and the new SCM downloads and installs the leader's checkpoint.
# That SCM becomes leader before its next restart (failover or {{transferLeadership}}).
# Something downloads a checkpoint from it.

h3. Impact
* Every checkpoint download from that leader fails with gRPC {{UNKNOWN}}, and the leader logs "Error streaming SCM DB checkpoint" with "Rocks Database is closed" (run).
* Consumers of that download: an SCM being bootstrapped while this SCM is leader (see HDDS-16899 for how a bootstrap behaves when the download fails), Recon's SCM DB snapshot sync ({{StorageContainerServiceProviderImpl.getSCMDBSnapshot}} downloads through {{InterSCMGrpcClient}} from the leader), and, with {{ozone.scm.ha.ratis.log.purge.enabled}}, any follower that lags behind the purged log. These are code reading; the run shows the failing download by the call a follower makes in {{SCMStateMachine.notifyInstallSnapshotFromLeader}}.
* It lasts until that SCM restarts. Losing leadership does not clear it: downloads then go to another SCM, and they fail again whenever this SCM is leader before a restart.
* The SCM's own data is not affected. An SCM bootstrapped while this one is leader is subject to HDDS-16899 and can end up without DB state that is not in the Raft log.

{{SCMDBCheckpointServlet}} ({{/dbCheckpoint}}) stores the store the same way, but Jetty initializes it on its first request, so it is affected only if it served a request before the install, and no Ozone component uses SCM's {{/dbCheckpoint}}. It is not changed by the patch (code reading only). The same pattern in the OM checkpoint servlets is HDDS-16878. {{SCMDBCheckpointServlet.init}} passes the store to the same {{DBCheckpointServlet.initialize}}, so the SCM servlet can follow the fix made there.

h3. Reproduction
PASS on the unmodified source, default configuration. [^TestBugMC4CheckpointServiceClosedStore.java] ({{hadoop-ozone/integration-test}}) starts three SCMs with one held back, starts the held back SCM C (it bootstraps and installs the leader's checkpoint in process, and its original store is closed), checks that a download from leader A works, transfers leadership to C and downloads from C through the follower's {{SCMSnapshotProvider}}. The download from C fails and C logs "Rocks Database is closed". A passing test means the defect is present. With the patch the download from C succeeds and the test fails at its {{assertThatThrownBy}}.

h3. Patch
[^MC-4-serve-scm-checkpoints-from-current-db-store.patch], against b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at 52812559dad.

{{InterSCMGrpcService.download}} looks up the current store on each request and builds the transaction info table and the {{SCMDBCheckpointProvider}} from it, so the service holds no reference to a store that an install can close. Nothing changes for an SCM that never installed a checkpoint.

Covered by an assertion added to {{TestSCMInstallSnapshot.testInstallCheckPoint}}: after the in process install the SCM serves a checkpoint, and that checkpoint carries the installed transaction index. Without the change the download fails with the error above. With it {{TestSCMInstallSnapshot}}, {{TestSCMInstallSnapshotWithHA}}, {{TestSCMSnapshot}}, {{TestScmHAFinalization}}, {{TestSecretKeySnapshot}} and the server-scm {{TestInterSCMGrpcProtocolService}}, {{TestInterSCMGrpcClient}} and {{TestSCMSnapshotProvider}} pass, and checkstyle is clean.

Found by TLA+ model checking and code review of the SCM Ratis snapshot and checkpoint install paths under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org