[jira] [Created] (HDDS-17179) OM skips the secret key expiry check before symmetric token signing is finalized, so a runtime finalize issues tokens that fail early

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-17179:
---------------------------------

             Summary: OM skips the secret key expiry check before symmetric token signing is finalized, so a runtime finalize issues tokens that fail early
                 Key: HDDS-17179
                 URL: https://issues.apache.org/jira/browse/HDDS-17179
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: CR-2-check-secret-key-expiry-before-symmetric-token-finalization.patch, TestBugCR2FinalizeSkipsSecretKeyGuard.java

h3. Mechanism
HDDS-13343 added a check to {{OzoneManager.createDelegationTokenSecretManager}}: OM refuses to start unless {{hdds.secret.key.rotate.duration}} + {{ozone.manager.delegation.token.max-lifetime}} + {{ozone.manager.delegation.remover.scan.interval}} is at most {{hdds.secret.key.expiry.duration}}, so that no SCM secret key expires while a token it signed can still be renewed. The check only runs when {{DELEGATION_TOKEN_SYMMETRIC_SIGN}} is already finalized at construction; an OM that is not finalized only checks the certificate grace period.

{{OzoneDelegationTokenSecretManager.createToken}} checks {{isAllowed(DELEGATION_TOKEN_SYMMETRIC_SIGN)}} on every call, and finalization ({{finalizeupgrade}}, then {{OMCompleteFinalizeUpgradeRequest}} and {{OMVersionManager.finalizeUpgrade}}) does not restart OM and does not run the check. So an OM that started before finalization with a configuration the check would refuse signs delegation tokens with SCM secret keys from the moment finalization completes, and the check only fires at the next OM start.

h3. Trigger
# A secure cluster that has not finalized {{DELEGATION_TOKEN_SYMMETRIC_SIGN}} (layout 8, added in 2.0.0; for example a cluster upgraded from 1.4.x) runs this version with a configuration that breaks the rule. The default {{hdds.secret.key.expiry.duration}} became 9d in 2.1.0; before that it was 7d, so an {{ozone-site.xml}} that sets it explicitly to the old default (7d with a 7d token lifetime and 1d rotation) is enough. With default values the rule holds and nothing happens.
# An administrator runs {{ozone admin om finalizeupgrade}}. OM finalizes without a restart.
# A client gets a delegation token, now signed with the current SCM secret key.
# SCM removes that key when it expires, while the token's max date is still in the future (for a token signed just before the next rotation, rotation period plus token lifetime minus key expiry, so 1d with the values above).
# Renewing the token fails ("Tampered/Invalid token", caused by "Secret verify key ... not found"), and the token cannot recover. Authenticating with it goes through the same signature check in {{OzoneDelegationTokenSecretManager.retrievePassword}} (code reading, not run).
# The next OM start fails with the check's {{IllegalArgumentException}}. A checkpoint install from the leader fails the same way, see Impact.

h3. Impact
Tokens issued between finalization and the next OM restart lose up to the last day of their lifetime with the configuration above (computed from the formula; the scaled run shows the same effect), which breaks long-running jobs that renew their tokens up to the max lifetime. The OM then refuses to start until the configuration is fixed. An OM that has finalized also terminates when it next installs a checkpoint from the leader: {{reloadOMState}} calls {{instantiateServices(true)}}, which runs the same check, and {{installCheckpoint}} exits the process when the reload throws (code reading, not run). Requires an explicit non-compliant configuration on a cluster that finalizes this layout feature at runtime; default configurations are not affected.

Affected versions: 2.1.0 and later (2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1), the releases that contain the check from HDDS-13343. 2.0.0 has layout 8 but no check at all.

h3. Reproduction
PASS on the unmodified source. [^TestBugCR2FinalizeSkipsSecretKeyGuard.java] ({{hadoop-ozone/integration-test}}) starts a secure SCM (no datanodes, safe mode off) and a secure OM with MiniKdc, with the OM storage initialized at {{HBASE_SUPPORT}} (layout 7) through the existing {{OMStorage.TESTING_INIT_APPARENT_VERSION_KEY}} hook, and scaled timings (rotate 10s, expiry 20s, token max lifetime 45s, remover interval 1s). It then calls {{finalizeUpgrade}}, gets a token, waits until SCM has removed the signing key, renews the token and restarts the OM. Output on the unmodified source:
{noformat}
OM started pre-finalized, apparent version HBASE_SUPPORT (7), symmetric DT allowed false
pre-finalize token secretKeyId='null' (RSA signed)
OM finalized at runtime, apparent version GET_KEY_INFO_REJECTS_OBS (103)
symmetric token issued=1792500182109 maxDate=1792500227109 signedBy=f068b740-... keyExpiry=1792500196294 (key expires 30815 ms before maxDate)
SCM removed key f068b740-... at 1792500196962, token maxDate 1792500227109 is still 30147 ms away
(OM log) SCMSecurityException: Secret verify key f068b740-... not found for token (...)
renew of unexpired token rejected at 1792500196983 (30126 ms before maxDate, ...): TOKEN_ERROR_OTHER ... Delegation token renewal failed ...
control token signed by 4ee85a92-... renewed, new expiry 1792500241984
OM restart after runtime finalization refused: Secret key expiry duration hdds.secret.key.expiry.duration should be greater than value of (...)
{noformat}
A passing test means the defect is present. With the patch the first OM start already fails with the check's message.

h3. Patch
[^CR-2-check-secret-key-expiry-before-symmetric-token-finalization.patch], against b94df213f20ec0d675f079b57a4566e4e64e4060. It also applies to master at df42f7f60a2.

The secret key check now runs at every OM start, whether or not {{DELEGATION_TOKEN_SYMMETRIC_SIGN}} is finalized; the certificate grace period check still runs only before finalization. This changes behaviour for one setup only: an OM that is not finalized and has a configuration that breaks the rule now refuses to start (with the same message as after finalization) instead of starting and failing later. The {{ozone.manager.delegation.token.max-lifetime}} description in {{ozone-default.xml}} now says that a secure Ozone Manager fails to start if the rule does not hold. The same check also runs in the checkpoint reload, so with the patch such an OM refuses at start instead of exiting during a later install.

HDDS-16958 (open) changes the same check and the same {{ozone-default.xml}} description. Each patch applies to b94df213f20ec0d675f079b57a4566e4e64e4060 on its own; this one does not apply on top of the patch attached to HDDS-16958 ({{OzoneManager.java}} and {{ozone-default.xml}} conflict, the test applies), so whichever lands second needs a small rebase, and the {{ozone-default.xml}} clause here can be dropped if HDDS-16958 lands first.

Rejected alternatives: checking in {{OMFinalizeUpgradeRequestBase.preExecute}} covers only the leader's configuration and not an OM that finalizes from a leader snapshot; an upgrade action for the layout feature runs during Ratis apply, and if it threw on one OM that OM would stay at the previous layout while its peers move on.

Covered by an extension of {{assertSecureOmInitFailsWithInvalidTokenAndSecretKeyConfig}} in the existing {{TestSecureOzoneCluster}}: OM initialization with the invalid configuration is refused when the OM is not finalized (new) and when it is (existing), and an OM that is not finalized still starts with a valid configuration. Without the change it fails at the case that is not finalized ({{Expected java.lang.IllegalArgumentException to be thrown, but nothing was thrown.}}). With it {{TestSecureOzoneCluster}} (9 tests), {{TestDelegationToken}}, {{TestSecureOzoneManager}} and {{TestOMUpgradeFinalization}} (7 tests) on mini clusters and the OM unit suites for the finalize requests, the finalize service, the delegation token secret manager and the bucket and key managers (83 tests) pass, and checkstyle is clean.

h3. Release note
A secure Ozone Manager that has not finalized {{DELEGATION_TOKEN_SYMMETRIC_SIGN}} now refuses to start if {{hdds.secret.key.rotate.duration}} + {{ozone.manager.delegation.token.max-lifetime}} + {{ozone.manager.delegation.remover.scan.interval}} exceeds {{hdds.secret.key.expiry.duration}}, as a finalized one already did. This hits secure clusters upgraded from 1.4.x or older and not yet finalized whose {{ozone-site.xml}} sets the expiry explicitly (for example to the old 7d default) or raises the token max lifetime or rotate duration; raise {{hdds.secret.key.expiry.duration}} on SCM and OM (9d covers the defaults) before upgrading.

Found by code review of the OM delegation token secret manager and request paths, as part of the TLA+ verification effort under HDDS-15926, on commit b94df213f20ec0d675f079b57a4566e4e64e4060. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org