[jira] [Created] (HDDS-16848) Hard lease auto commit built before a later hsync shrinks the key below the length OM acknowledged and deletes the acknowledged block

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-16848:
---------------------------------

             Summary: Hard lease auto commit built before a later hsync shrinks the key below the length OM acknowledged and deletes the acknowledged block
                 Key: HDDS-16848
                 URL: https://issues.apache.org/jira/browse/HDDS-16848
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: MC-1-reject-commit-shorter-than-hsynced-length.patch, TestBugMC1StaleAutoCommit.java

h3. Mechanism
{{OmMetadataManagerImpl.getExpiredOpenKeys}} builds the hard lease auto commit of an hsync'ed key from the committed key entry ({{keyTable}} or {{fileTable}}) as it is at scan time: its {{dataSize}} and its block list. {{OpenKeyCleanupService}} submits the {{CommitKey}} requests after the scan, one after the other. {{OMKeyCommitRequest.validateAndUpdateCache}} and {{OMKeyCommitRequestWithFSO}} then apply the request with no check that the key is still in the scanned state. The only checks related to the hsync state are that the open key exists, that it has no {{DELETED_HSYNC_KEY}} or {{OVERWRITTEN_HSYNC_KEY}} marker and that it has no {{LEASE_RECOVERY}} flag.

If the writer allocates a block and hsyncs between the scan and the apply, the stale commit sets the key length back to the scan time value, treats the new block as allocated but uncommitted ({{OmKeyInfo.updateLocationInfoList}}), moves it to {{deletedTable}} through {{wrapUncommittedBlocksAsPseudoKey}}, and removes the open key.

h3. Trigger
{{ozone.fs.hsync.enabled}} and {{ozone.hbase.enhancements.allowed}} are true (both default false). Any bucket layout.
# A writer writes into a block and hsyncs.
# Its open key reaches {{ozone.om.lease.hard.limit}} (default 7d). The modification time of the open key moves only on block allocation and on the first hsync, so this happens to a writer that was silent at OM for 7 days, and also to a live writer that stayed inside one block for 7 days (HDDS-16825).
# A cleanup run scans the open key table and builds the auto commit.
# Before that commit is applied, the writer moves to a new block, writes and hsyncs. OM acknowledges the longer length.
# The auto commit is applied.

The window in step 4 is the rest of the scan, the {{DeleteOpenKeys}} request of the same run, the auto commits submitted before this one and its own Ratis round trip. It is narrow, from one Ratis round trip to seconds on a large open key table (argued from code, not measured on a cluster).

h3. Impact
* The key is committed with the scan time length. Data OM had acknowledged to the writer is no longer readable.
* The acknowledged block is queued in {{deletedTable}}, so {{KeyDeletingService}} sends it to SCM for deletion. Nothing restores it.
* The open key is gone, but the stock client only calls OM on the first hsync of each block, so further hsyncs into the deleted block keep succeeding against the datanodes until the next block allocation or close fails with {{KEY_NOT_FOUND}} (argued from client code, not run).

h3. Reproduction
PASS, deterministic, unmodified source, for OBJECT_STORE and FILE_SYSTEM_OPTIMIZED buckets. [^TestBugMC1StaleAutoCommit.java] ({{hadoop-ozone/ozone-manager}}) runs one real OM through {{OzoneManagerProtocol}} with a 200 ms hard lease limit. It performs the two halves of {{OpenKeyCleanupTask.call}} itself, the real {{KeyManager.getExpiredOpenKeys}} scan and the submission of its {{CommitKey}} to the OM Ratis server built as the service builds it, and issues the writer's allocate block and hsync between them, so no thread has to be paused. Output on the OBS bucket: "Acknowledged length 3000 with blocks [13046302476, 13046302821]; after the auto commit: length 1000 with blocks [13046302476], blocks queued for deletion [13046302821]". The FSO bucket gives the same result. A passing test means the defect is present. Physical deletion on datanodes was not run.

h3. Patch
[^MC-1-reject-commit-shorter-than-hsynced-length.patch], against a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at b0aa6475b78.

A commit that is neither an hsync nor a recovery commit, sent by the client that holds the hsync'ed key, is rejected with {{INVALID_REQUEST}} when its length is smaller than the length of the hsync'ed key in the key table. A stale auto commit is exactly this case. A correct client never sends it, because {{KeyOutputStream}} only sends growing lengths and the final close is never shorter than an earlier hsync. Hsync and recovery commits are not changed, and the next cleanup run commits from fresh state if the lease is still expired. Both {{OMKeyCommitRequest}} and {{OMKeyCommitRequestWithFSO}} get the check.

The check changes what the apply writes, so it is enabled by a new {{OzoneManagerVersion.HSYNC_COMMIT_VALIDATION}} (101) through {{OMVersionManager.isAllowed}}. Until the cluster is finalized to that version every OM applies the commit as before, so OMs on different software versions never apply the same log entry differently, and the defect remains until finalization. The version number is a placeholder: it would be the first OM version after the 2.x baseline, so a cluster at version 100 reports that finalization is needed after upgrading, and other open fixes may claim the same number. Which version (or a shared one) to use is for review to decide.

A stricter alternative is to carry the scanned {{updateID}} in {{CommitKeyRequest}} and compare it on apply. It would also reject a stale commit of equal length (which cannot lose acknowledged data), but it needs a new request field and a change to the cleanup service, and it changes what the apply writes just as the length check does. The length check uses only state the apply already reads and rejects only the case that loses acknowledged data. Reading the key again in the service before each submit would only narrow the window.

Covered by {{testCommitShorterThanHsyncedKey}} in the existing {{TestOMKeyCommitRequest}}, inherited by {{TestOMKeyCommitRequestWithFSO}}, once with the version finalized (the stale commit is rejected, a full close is accepted) and once before finalization (the stale commit is applied as before). Without the change the finalized case fails for both layouts with "expected: <INVALID_REQUEST> but was: <OK>". With it the commit request, commit response, recover lease, key rename, key delete, metadata manager, open key cleanup, OM storage and OM version suites pass (222 tests), {{TestHSync}} passes on a mini cluster (42 tests, with both patches), and checkstyle is clean. With the patch the reproduction fails at its first assertion because the auto commit is rejected and the key keeps 3000 bytes.

Found by TLA+ model checking and code review of the OM open key cleanup and hsync lease recovery paths under HDDS-15926, on commit a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing; the closest are HDDS-16764 (the same staleness on the {{DeleteOpenKeys}} path) and HDDS-16825 (how a live writer reaches the hard limit). The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org