[jira] [Created] (HDDS-16768) Single key delete of an hsync'ed FSO file persists DELETED_HSYNC_KEY under the wrong open table key

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-16768:
---------------------------------

             Summary: Single key delete of an hsync'ed FSO file persists DELETED_HSYNC_KEY under the wrong open table key
                 Key: HDDS-16768
                 URL: https://issues.apache.org/jira/browse/HDDS-16768
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: MC-1-persist-deleted-hsync-marker-under-fso-open-key.patch

h3. Mechanism
{{OMKeyDeleteRequestWithFSO.validateAndUpdateCache}} flags the hsync writer's open entry with {{DELETED_HSYNC_KEY}} in the cache under the FSO key, {{getOpenFileName(volumeId, bucketId, parentId, fileName, hsyncClientId)}} ({{:174}}). {{OMKeyDeleteResponseWithFSO.addToDBBatch}} persists that flagged entry under the non FSO key, {{getOpenKey(volume, bucket, key, hsyncClientId)}} ({{:117-118}}).

The real FSO open row in RocksDB therefore stays unflagged, and a stray {{/volume/bucket/key/clientId}} row is added to the open file table.

h3. Trigger
On an FSO bucket with hsync enabled: create a file, hsync, delete the file non recursively ({{FileSystem.delete(file, false)}}), and let the delete flush and its cache epoch evict (automatic, milliseconds).

h3. Impact
Two consequences of the same root cause, both reproduced:
* *Deleted file comes back.* When the writer then closes or hsyncs, its {{CommitKey}} is accepted instead of failing with {{KEY_NOT_FOUND}}. The deleted file is back in the file table and survives an OM restart, while its earlier blocks are already queued for deletion by the delete. This is a regression of HDDS-9930 for FSO single key delete, introduced when HDDS-10650 replaced the removal of the open key with the {{DELETED_HSYNC_KEY}} marker.
* *Open key cleanup stops.* If the writer never closes, {{OmMetadataManagerImpl.getExpiredOpenKeys}} reaches the unflagged hsync'ed open row after the lease hard limit, looks up its {{fileTable}} row, gets null, and dereferences it. {{OpenKeyCleanupService}} catches only {{IOException}}, so the FSO task of every run that reaches the row dies with {{NullPointerException}} ({{OmMetadataManagerImpl.java:1516}}) and submits nothing. The row is never removed, so a restart does not help, and expired FSO open keys are no longer cleaned up or auto committed.

h3. Reproduction
PASS, no injected timing, through the public FileSystem API on a single OM mini cluster for the first consequence, and with FileSystem calls plus the real cleanup service for the second. Both also reproduce at OM level through the real write path without a mini cluster. Divergence between OMs is argued from the same mechanism, not run.

h3. Suggested fix
* In {{OMKeyDeleteResponseWithFSO.addToDBBatch}}, persist {{deletedOpenKeyInfo}} under the key the request used for the cache ({{getOpenFileName(...)}}), or pass the request's {{dbOpenKey}} into the response.
* Harden {{getExpiredOpenKeys}}: when the {{fileTable}} lookup returns null for an hsync'ed open row, treat the row as an orphan and add it to the delete list. This also heals rows already persisted.
* The sibling sites {{OmKeysDeleteRequestWithFSO}} and {{OMDirectoriesPurgeRequestWithFSO}} with their responses were checked and persist under the right key.
* Existing clusters may hold stray {{/volume/bucket/key/clientId}} rows in the open file table. Those carry the marker, so the cleanup service removes them once they expire, provided the scan is not stopped first by an unflagged row.

h3. Patch
[^MC-1-persist-deleted-hsync-marker-under-fso-open-key.patch], against 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at 9c6bee807c8.

Two changes:
* Root cause: {{OMKeyDeleteResponseWithFSO.addToDBBatch}} builds the open key with {{getOpenFileName(volumeId, bucketId, parentObjectID, fileName, hsyncClientId)}}, the key the request used for the cache.
* Scan hardening: in {{getExpiredOpenKeys}}, an expired hsync'ed open key whose key table entry is missing is added to the delete list instead of being dereferenced. This is what lets a cluster that already holds unflagged rows recover. It is separable from the root cause change if a reviewer prefers two patches.

Covered by {{testCleanupDeletedHsyncKey}} and {{testCleanupHsyncKeyWithoutCommittedFile}} in the existing {{TestOpenKeyCleanupService}}. Without the change the first fails because the writer's commit after the delete is accepted ("Expected OMException to be thrown, but nothing was thrown"), and the scan throws the {{NullPointerException}} above. The cleanup service, key delete, open key delete, directory purge and {{TestOmMetadataManager}} unit suites (143 tests) and {{TestHSync}} (42 tests) pass, checkstyle is clean.

Found by TLA+ model checking and code review of the OM FSO namespace mutation paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org