[jira] [Updated] (HDDS-16863) OM that stops during snapshot install after the DB swap restarts pre-finalized although its DB is finalized

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16863:
------------------------------
    Attachment: TestBugMC1PreFinalizedAfterInterruptedInstall.java

> OM that stops during snapshot install after the DB swap restarts pre-finalized although its DB is finalized
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16863
>                 URL: https://issues.apache.org/jira/browse/HDDS-16863
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Major
>         Attachments: MC-1-finalize-om-from-db-on-startup.patch, TestBugMC1PreFinalizedAfterInterruptedInstall.java
>
>
> h3. Mechanism
> The OM keeps its apparent version in two places: the VERSION file ({{OMStorage}}) and {{#LAYOUTVERSION}} in the OM DB meta table. On startup {{OMVersionManager}} is built from the VERSION file only. {{OzoneManager.instantiateServices}} then calls {{RatisBasedVersionManager.validateDBVersion}}, which only logs at INFO when the DB differs ("That is expected if this component has never been finalized to a newer version."). {{RatisBasedVersionManager.finalizeFromSnapshotIfRequired}}, the only code that brings the VERSION file up to the DB, runs only when {{instantiateServices}} is called from {{reloadOMState}} during a snapshot install.
> {{OzoneManager.installCheckpoint}} first replaces the DB with the leader's checkpoint and only then writes the VERSION file, once per version, from {{finalizeFromSnapshotIfRequired}}. If the OM process stops in between, the DB is finalized and the VERSION file is not. That happens on a crash, or on any exception that sends {{installCheckpoint}} to {{exitSystem}}, for example a failed VERSION file write. The window runs from the end of the checkpoint file move (the transient DB marker is gone, so the DB opens normally on restart) through opening the new DB and building the services, to the last VERSION write.
> After a restart nothing repairs this:
> * Startup only logs the mismatch.
> * The Raft log is replayed from the DB's transaction index, which is the checkpoint index, so the {{CompleteFinalizeUpgrade}} entry below it is never applied on this OM.
> * {{OMUpgradeFinalizeService}} only acts when the finalization in progress marker is set, and the finalized checkpoint does not have it.
> The OM runs pre-finalized until a later snapshot install on that node finalizes it by chance, or until finalize is run again and that OM later leads (see Impact).
> h3. Trigger
> # An OM is down or behind while the cluster is finalized, and the leader purges its log past it.
> # The OM installs a snapshot from the leader. The checkpoint carries the finalized version.
> # The OM process stops after the DB swap and before the VERSION file is written: a crash (kill, OOM, host failure) or an error such as a full or failing metadata disk, which leads to {{exitSystem}}.
> # The OM is restarted and rejoins without another snapshot install. With default settings ({{ozone.om.ratis.log.purge.gap}} 1,000,000) it catches up from the Raft log.
> I ran the VERSION write failure variant (read-only {{om/current}}) on a mini cluster. The crash variant is from code, not run: it leaves the same files on disk.
> The same state also arises without a snapshot install when {{om.db}} is replaced by hand with a checkpoint from a finalized peer, the recovery that the comment next to the inconsistent DB marker error in {{OmMetadataManagerImpl}} recommends. The gap exists in every release since 1.2.0, where HDDS-4908 added the snapshot-only finalize under the same {{withNewSnapshot}} condition, so an OM left in this state by an older release is carried into a newer one (from code, not run).
> SCM has the same gap for a crash during its snapshot install. A reload failure there restores the DB backup, so only the crash variant applies. This patch does not cover SCM.
> h3. Impact
> * The OM silently runs pre-finalized on top of a finalized DB. {{getVersionForClient}}, which is advertised to clients in {{ServiceInfo}}, reports the last pre-ZDU client version instead of the software version (run).
> * While that OM is leader, the version-gated request checks and the version stamped into requests are evaluated against the old version, and {{queryUpgradeStatus}} reports the OM as UNFINALIZED and the cluster as IN_PROGRESS (from code, not run).
> * Running finalize again sets the finalization marker only on the stuck OM, because each replica checks {{needsFinalization}} when it applies the request. The stuck OM then finalizes once it is next leader. Until then its meta table differs from its peers by that marker (from code, not run).
> * No data loss. Followers apply the requests as the leader stamped them.
> h3. Reproduction
> PASS on the unmodified commit, 3 of 3 runs, about 35 s. [^TestBugMC1PreFinalizedAfterInterruptedInstall.java] ({{hadoop-ozone/integration-test}}) uses a three OM HA mini cluster with one OM down, set up as in {{TestOMUpgradeFinalization.testFinalizationFromSnapshot}}. It finalizes the two running OMs, purges the log and makes the down OM's {{om/current}} read only. When that OM starts it installs the snapshot, the VERSION write fails after the DB swap and {{installCheckpoint}} calls {{exitSystem}}. A test exit manager parks the exiting thread and the test stops the OM. The directory is made writable again and the OM is restarted as a new {{OzoneManager}} on the same storage.
> Result: {{apparent version INITIAL_VERSION (0), VERSION file 0, DB version 100, finalization marker null, DB transaction 1#106, applied index (t:1, i:106), leader index 106}}, one snapshot install in total, and the startup log line "Version file has different apparent version (INITIAL_VERSION (0)) than DB (ZDU (100)). That is expected if this component has never been finalized to a newer version." A passing test means the defect is present.
> The test does not write after the restart. With its 5 entry purge gap the leader would purge past the OM again, and the resulting second snapshot install finalizes it.
> h3. Patch
> [^MC-1-finalize-om-from-db-on-startup.patch], against a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at b0aa6475b78.
> On startup (the {{withNewSnapshot}} false branch of {{instantiateServices}}) the OM now calls a new {{RatisBasedVersionManager.finalizeToDBVersionIfRequired}}. It finalizes only up to the version recorded in the DB, never to the software version of the binary, and logs a warning when it does. It is a no-op unless the DB records a version the VERSION file does not allow. On a normal restart the DB is at or below the VERSION file, since {{OMCompleteFinalizeUpgradeRequest}} writes the VERSION file before its DB batch. It runs before the Ratis server is created, so there is no concurrent apply and no log replay yet.
> The bound uses {{ComponentVersionManager.finalizeUpgrade(lastVersion)}}. HDDS-16862 adds the same helper, with an identical hunk, to bound the snapshot install path, while this patch bounds the startup path. The two patches merge cleanly in either order ({{git merge}}, or {{git apply --3way}} for the patch files).
> {{QuotaRepairUpgradeAction}} now skips the repair, with a warning like the follower branch, when the OM has no Ratis server yet. Without that, a VERSION file below {{QUOTA}} makes the startup finalize fail with an NPE in {{checkLeaderStatus}} and the OM does not start. A follower already skips this repair. If the pending mark from HDDS-16861 lands, this startup skip should set it too. The patches for HDDS-16860 and HDDS-16861 change the same few lines of {{QuotaRepairUpgradeAction.execute}} as this one, so whichever lands later needs a trivial rebase.
> Covered by {{testFinalizationFromDBOnRestart}} in the existing {{TestOMUpgradeFinalization}}. On one OM it checks four cases:
> * An OM without a DB version stays pre-finalized.
> * A DB at {{HBASE_SUPPORT}} over a VERSION file at {{INITIAL_VERSION}} finalizes the OM to {{HBASE_SUPPORT}} and no further.
> * A DB behind the VERSION file changes nothing.
> * A DB at the software version finalizes the OM fully.
> Failures:
> * Without the change: {{expected: <HBASE_SUPPORT (7)> but was: <INITIAL_VERSION (0)>}}.
> * With an unbounded startup finalize: {{expected: <HBASE_SUPPORT (7)> but was: <ZDU (100)>}}.
> * Without the quota repair change: the NPE above.
> With the patch, these pass and checkstyle is clean:
> * {{TestOMUpgradeFinalization}} (4 tests)
> * the OM upgrade, state machine and Ratis server unit suites, plus suites that start an OM through {{OmTestManagers}} (171 tests)
> * the SCM and datanode version manager suites (40 tests)
> * {{TestOzoneManagerRestart}}, {{TestUpgradeFinalizationWithOldClients}}, {{TestOMRatisSnapshots}}, {{TestOMBucketLayoutUpgrade}}, {{TestOMInstallSnapshotDuringBootstrapping}} and {{TestOmStartupInvalidApparentVersion}} (22 tests)
> With the patch the reproduction fails, because the restarted OM finalizes in its constructor ({{expected: INITIAL_VERSION (0) but was: ZDU (100)}}).
> Found by TLA+ model checking and code review of the OM upgrade finalization paths under HDDS-15926, on commit a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org