[jira] [Updated] (HDDS-16958) OM delegation token and secret key duration guard ignores hdds.secret.key.rotate.check.duration

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16958:
------------------------------
    Attachment: CR-1-include-rotation-check-interval-in-delegation-token-guard.patch

> OM delegation token and secret key duration guard ignores hdds.secret.key.rotate.check.duration
> -----------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16958
>                 URL: https://issues.apache.org/jira/browse/HDDS-16958
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Minor
>         Attachments: CR-1-include-rotation-check-interval-in-delegation-token-guard.patch, TestBugCR1TokenOutlivesSecretKey.java
>
>
> h3. Mechanism
> HDDS-13343 added a startup guard in {{OzoneManager.createDelegationTokenSecretManager}}: OM refuses to start unless {{hdds.secret.key.rotate.duration}} + {{ozone.manager.delegation.token.max-lifetime}} + {{ozone.manager.delegation.remover.scan.interval}} <= {{hdds.secret.key.expiry.duration}}. The formula assumes that a secret key signs tokens for exactly the rotate duration. It can sign for up to two {{hdds.secret.key.rotate.check.duration}} longer:
> * SCM rotates only when {{SecretKeyManagerService}} runs, every rotation check interval, so the rotation is up to one interval late.
> * {{DefaultSecretKeySignerClient}} on OM polls SCM for a new key at the rotate duration of its current key and then every rotation check interval, so OM keeps signing with the old key for up to one more interval.
> With R = rotate duration, C = rotation check duration, L = token max lifetime and E = key expiry duration, the last token signed with a key can have a max date of up to key creation + R + 2C + L, while SCM may remove the key at the first rotation after key creation + E. If E < R + 2C + L, the key can be gone before the token's max date. An OM that has not cached the key (after a failover, or because it never verified a token of that key) then rejects the token: {{verifySignature}} logs "Secret verify key ... not found" and the client gets "Tampered/Invalid token.". An OM restarted after the removal skips the token when it loads its tokens.
> h3. Trigger
> The guard already forces E >= R + L + remover scan interval, so this needs both of the following:
> # The remover scan interval is shorter than two rotation check intervals.
> # The expiry duration is within two rotation check intervals of R + L.
> The default settings (R = 1d, C = 10m, E = 9d, L = 7d, scan interval 1h) are not affected, and neither is any expiry duration that the guard accepts as long as the rotation check interval is at most half of the remover scan interval. The two conditions are necessary, not sufficient: SCM removes an expired key only at a rotation, so whether a key is hit depends on how the two timers line up. The check in the patch is the worst case bound. The reproduction below uses a rotation check interval close to the rotate duration.
> h3. Impact
> A delegation token signed shortly before OM switches to a new key is rejected at most two rotation check intervals before its max date, on an OM that does not have the key cached. Only configurations whose expiry duration is within two rotation check intervals of R + L are affected, which the guard allows only when the remover scan interval is shorter than two rotation check intervals. Block and container tokens were not examined.
> h3. Reproduction
> PASS in about two minutes, unmodified source. [^TestBugCR1TokenOutlivesSecretKey.java] ({{hadoop-ozone/integration-test}}) starts a Kerberos secured SCM and OM with R = 18s, C = 15s, E = 89s, L = 70s and a scan interval of 1s, which the guard accepts (18 + 70 + 1 = 89). It requests tokens until OM has switched keys twice, keeps the last token of the previous key and waits until SCM has removed that key. In the run SCM removed the key 18.3s before the max date of the token, and the token was then rejected with "Tampered/Invalid token." 18.2s before its max date, while a token of the current key was accepted. A passing test means the defect is present.
> h3. Patch
> [^CR-1-include-rotation-check-interval-in-delegation-token-guard.patch], against 025266b52a6dcaf39508607cad736799c40907e8. It also applies to master at a65a32e7c4b.
> The guard gets a second condition, R + L + 2C <= E, with its own message, and the descriptions of {{ozone.manager.delegation.token.max-lifetime}} and {{hdds.secret.key.expiry.duration}} in {{ozone-default.xml}} name it. The existing condition is unchanged. The default settings and the compose and test configurations in the tree pass, except {{TestSecretKeySnapshot}}, which sits exactly on the old minimum (R = 30s, C = 1s, E = 61s, L = 30s, scan interval 1s) and gets a token lifetime that is one check interval shorter.
> Compatibility: OM refuses to start on a configuration that violates the new condition, like the HDDS-13343 check next to it. The configurations that start today and are refused after the change are those with R + L + scan interval <= E < R + L + 2C, for example the default settings with C = 1h and E = 8d1h. The new condition is a worst case bound, sufficient but not necessary, so it also refuses configurations in which the timers never line up. Logging a warning instead of refusing to start is the alternative if reviewers prefer it. Adding 2C to the existing sum (R + L + scan interval + 2C <= E) was considered and not chosen, because it would also refuse configurations in which no unexpired token can lose its key.
> Covered by a second case in the existing guard check of {{TestSecureOzoneCluster.testSecureScmAndOmStartupAndAccessControl}}: E = 8d1h with C = 1h passes the old condition and is now refused. Without the change the test fails because OM starts. With it {{TestSecureOzoneCluster}} (9 tests), {{TestSecretKeySnapshot}}, {{TestDelegationToken}} and {{TestBlockTokens}} pass and checkstyle is clean. With the patch the reproduction fails when it creates the OM, with the new message.
> Found by code review of the SCM secret key rotation and distribution paths, as part of the TLA+ verification effort under HDDS-15926, on commit 025266b52a6dcaf39508607cad736799c40907e8. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org