[jira] [Created] (HDDS-16879) RevokedSTSTokenCleanupService keeps the metadata manager closed by a checkpoint install, so every cleanup run fails while that OM is leader

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-16879:
---------------------------------

             Summary: RevokedSTSTokenCleanupService keeps the metadata manager closed by a checkpoint install, so every cleanup run fails while that OM is leader
                 Key: HDDS-16879
                 URL: https://issues.apache.org/jira/browse/HDDS-16879
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: CR-2-sts-token-cleanup-read-current-metadata-manager.patch, TestBugCR2StsCleanupClosedMetadataManager.java

h3. Mechanism
{{RevokedSTSTokenCleanupService}} stores {{ozoneManager.getMetadataManager()}} in a final field in its constructor, and the service is created once in {{OzoneManager.start}}. {{OzoneManager.installCheckpoint}} stops the old metadata manager, which closes its store, and {{reloadOMState}} creates a new {{OmMetadataManagerImpl}}. The service is not stopped, recreated or updated on that path, so each later run iterates {{s3RevokedStsTokenTable}} of the closed store.

The service is not in a release yet (HDDS-14094), so only master is affected. The OM checkpoint servlets have the same problem, see HDDS-16878.

h3. Trigger
# An OM installs a checkpoint from the leader through {{installSnapshotFromLeader}}.
# That OM becomes leader (election or leadership transfer). The service only runs on the leader.

h3. Impact
* While that OM is leader, every run fails with {{RocksDatabaseException: Rocks Database is closed}}, logs "Failure while scanning s3RevokedStsTokenTable" at ERROR with a stack trace, and deletes nothing, until the OM restarts. With the default {{ozone.om.sts.token.cleanup.service.interval}} of 3h that is one ERROR every 3 hours. The scan fails on an empty table too, so the ERROR appears whether or not STS is used.
* Expired revocation entries stay in the table while that OM leads. They belong to tokens past their maximum lifetime, and revocation checks read the current table, so no authentication result changes. Another OM that has not installed a checkpoint since it started cleans up the backlog once it becomes leader (code reading).

h3. Reproduction
PASS on the unmodified source, deterministic. [^TestBugCR2StsCleanupClosedMetadataManager.java] ({{hadoop-ozone/integration-test}}) starts three OMs with one held back and the cleanup interval set to 1s, advances the leader past the Ratis log purge gap, starts the late OM so that Ratis installs a checkpoint on it, and transfers leadership to it. One revocation entry 13 hours old is then written directly into the new leader's table, because a real revoke cannot be back dated; it only makes the missed cleanup visible. Output:
{noformat}
After omNode-3 installed a checkpoint and became leader: scan failures logged = 4, 'Rocks Database is closed' in log = true, expired entry still present = true
ERROR service.RevokedSTSTokenCleanupService (RevokedSTSTokenCleanupService.java:call(187)) - Failure while scanning s3RevokedStsTokenTable.  It will be retried in the next interval
org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Rocks Database is closed
	at org.apache.hadoop.hdds.utils.db.RocksDatabase.acquire(RocksDatabase.java:417)
	at org.apache.hadoop.hdds.utils.db.RocksDatabase.newIterator(RocksDatabase.java:809)
{noformat}
A passing test means the defect is present. With the patch no scan fails, the service logs "Found and removed 1 revoked STS token entries." and the test fails at its first defect assertion.

h3. Patch
[^CR-2-sts-token-cleanup-read-current-metadata-manager.patch], against a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at 6818707698c.

The service no longer keeps the metadata manager and reads {{ozoneManager.getMetadataManager()}} at the start of each run, as {{OMUpgradeFinalizeService}} already does. Nothing changes for an OM that never installed a checkpoint, and no lifecycle code is added to the install path.

Covered by the new {{TestRevokedSTSTokenCleanupService#scansCurrentMetadataManagerAfterItIsReplaced}}: after the service is created the OM returns a new metadata manager, and one run must clean the expired entry of the new table and not the one of the original table. Without the change it submits the original table's entry. With it {{TestRevokedSTSTokenCleanupService}}, {{TestS3RevokeSTSTokenRequest}} and {{TestS3SecurityUtil}} pass (38 tests) and checkstyle is clean.

Found by code review of the OM Ratis snapshot (checkpoint) install paths, as part of the TLA+ verification effort under HDDS-15926, on commit a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org