[jira] [Created] (HDDS-16959) OM drops persisted delegation tokens from memory when the secret key fetch from SCM fails during token load

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-16959:
---------------------------------

             Summary: OM drops persisted delegation tokens from memory when the secret key fetch from SCM fails during token load
                 Key: HDDS-16959
                 URL: https://issues.apache.org/jira/browse/HDDS-16959
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: CR-4-keep-persisted-delegation-tokens-when-secret-key-fetch-fails.patch, TestBugCR4OmRestartDuringScmKeyInit.java

h3. Mechanism
The OM DB stores only the identifier and the renew date of a delegation token. When {{OzoneDelegationTokenSecretManager}} is created, {{loadTokenSecretState}} calls {{addPersistedDelegationToken}} for every stored token, which fetches the token's secret key from SCM with {{SecretKeyClient.getSecretKey}} and recomputes the password. {{loadTokenSecretState}} catches every exception from that call, logs "exception while loading delegation token from DB... ignored to continue startup" and goes on to the next token.

HDDS-13234 added that catch for a secret key that SCM has already removed, in which case {{getSecretKey}} returns null and the token can never be verified again. A fetch that fails ({{SCMSecurityException}} "Error fetching secret key ...") ends in the same catch. Before HDDS-13234 the same failure aborted the OM start. That change is in 2.1.0 and later releases (its fix versions are 2.0.1 and 2.1.0). The token is then missing from {{currentTokens}} although it is unexpired, still in the DB, and its key still exists on SCM. Nothing loads it later, so the OM rejects it until the OM process is restarted.

Two things make the failed fetch reachable during a normal start:
* The token load runs in the {{OzoneManager}} constructor ({{instantiateServices}}), before {{startSecretManager}} starts the signer client. Only the signer client's initial fetch waits for SCM to leave {{SECRET_KEY_NOT_INITIALIZED}}.
* {{SecretKeyProtocolClientSideTranslatorPB.handleError}} turns the error status into an exception after the RPC has returned, so the SCM failover retry policy does not retry it.

h3. Trigger
# Secure cluster, at least one delegation token issued.
# An SCM is restarted and becomes leader. Until {{SecretKeyManagerService}} has run {{SecretKeyManager.checkAndInitialize}} on it, the secret key RPC answers {{SECRET_KEY_NOT_INITIALIZED}}.
# An OM starts in that window. The window is normally short: SCM initializes the keys as soon as it is leader ready, and if that attempt fails the next one comes with the next rotation check ({{hdds.secret.key.rotate.check.duration}}, default 10m). Its natural width was not measured.
# The token is renewed or used for authentication on that OM.

Any other failure of the fetch at load time has the same effect, for example SCM not reachable within the client retries (same catch, not run). {{OzoneManager.reloadOMState}} (Ratis snapshot installation on a follower) and {{OzoneManager.restart}} build the secret manager the same way, so they are exposed too (read from the code, not run).

h3. Impact
* Every token loaded while the fetch fails is rejected by that OM: renewal fails with "Delegation token renewal failed" and the OM logs "token ... can't be found in cache", {{retrievePassword}} throws {{InvalidToken}} with the same message, so token authentication fails. Jobs that hold delegation tokens fail although their tokens are valid.
* The state lasts until that OM is restarted while SCM can serve the key. The only trace at load time is one ERROR line per token.
* In OM HA every OM loads its tokens on its own, and tokens are only checked on the leader. An OM that lost its tokens as a follower shows it when it becomes leader (read from the code, the reproduction uses one OM).

h3. Reproduction
PASS in about one minute, unmodified source. [^TestBugCR4OmRestartDuringScmKeyInit.java] ({{hadoop-ozone/integration-test}}) runs a Kerberos mini cluster with three SCMs and one OM. A client gets a delegation token, a follower SCM is restarted, leadership is transferred to it and the OM is restarted before that SCM has initialized its secret keys. The test holds the monitor of the new leader's {{SecretKeyManager}} during the OM restart, which only delays the synchronized {{checkAndInitialize}} so that the window is hit on every run. Afterwards the token is unexpired, present in the OM DB and its key is served by SCM, but {{renewDelegationToken}} fails and {{retrievePassword}} throws {{InvalidToken}}. After a second OM restart the same token renews. A passing test means the defect is present.

h3. Patch
[^CR-4-keep-persisted-delegation-tokens-when-secret-key-fetch-fails.patch], against 025266b52a6dcaf39508607cad736799c40907e8. It also applies to master at a65a32e7c4b.

{{addPersistedDelegationToken}} now tells the two cases apart. If the fetch throws {{SCMSecurityException}}, the token is kept in {{currentTokens}} as an entry of its own type ({{TokenInfoWithoutPassword}}) and one WARN line without a stack trace is logged. {{validateToken}} computes the password of such an entry from its secret key when the token is used, and throws {{InvalidToken}} if the key still cannot be fetched or SCM reports it as gone. Only the load creates such an entry, and every later update of the token replaces it with a regular one. The value {{validateToken}} computes is the one the existing signature check requires the stored password to have, so the accepted tokens are the same as after a successful load. The HDDS-13234 behaviour for a key that SCM no longer has (null result) is unchanged, and OM startup does not wait for SCM any longer than today.

Retrying the fetch during the load was considered. It would hold the OM constructor (and a snapshot installation) for as long as SCM does not answer, and would still need a decision on what to do when the retries run out.

Covered by {{testReloadTokenWhenSecretKeyFetchFails}} in the existing {{TestOzoneDelegationTokenSecretManager}}: the fetch fails during the reload and on first use, then SCM reports the key as missing, then serves it. Without the change the test fails because the token "can't be found in cache". With the change {{TestOzoneDelegationTokenSecretManager}} (17 tests, including {{testExpiredSecretKey}} from HDDS-13234), the token identifier, delegation token request and response and double buffer unit tests (19 tests), {{TestDelegationToken}} (2 tests), {{TestSecureOzoneCluster}} (9 tests) and {{TestSecretKeySnapshot}} pass and checkstyle is clean. With the patch the reproduction fails at the assertion that expects the renewal to fail, because the renewal succeeds.

Found by code review of the SCM secret key rotation and distribution paths, as part of the TLA+ verification effort under HDDS-15926, on commit 025266b52a6dcaf39508607cad736799c40907e8. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org