[jira] [Updated] (HDDS-16961) ozone admin om fetch-key does not make OM fetch the current secret key from SCM, it prints the key OM already has

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16961:
------------------------------
    Attachment: TestBugCR11FetchKeyDoesNotReachSigner.java

> ozone admin om fetch-key does not make OM fetch the current secret key from SCM, it prints the key OM already has
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16961
>                 URL: https://issues.apache.org/jira/browse/HDDS-16961
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Minor
>         Attachments: CR-11-let-om-fetch-key-reach-secret-key-signer-client.patch, TestBugCR11FetchKeyDoesNotReachSigner.java
>
>
> h3. Mechanism
> {{ozone admin om fetch-key}} is described as "CLI command to force OM to fetch the latest secret key from SCM". {{OzoneManager.refetchSecretKey}} serves it with two calls on its secret key client: {{refetchSecretKey()}}, then {{getCurrentSecretKey().getId()}}, which the command prints as "Current Secret Key ID".
> Since HDDS-8829 that client is a {{DefaultSecretKeyClient}}, the composition of a signer client and a verifier client. It passes {{getCurrentSecretKey}}, {{start}} and {{stop}} on to {{DefaultSecretKeySignerClient}}, but not {{refetchSecretKey}}. So the empty default of {{SecretKeySignerClient.refetchSecretKey}} runs, SCM is not asked, and the command reports the ID of the key OM already holds, with exit code 0.
> When HDDS-8677 added the command, OM held a {{DefaultSecretKeySignerClient}} directly, whose {{refetchSecretKey}} does ask SCM. The command worked in 1.4.x and has had no effect since 2.0.0.
> h3. Trigger
> # Secure cluster.
> # SCM gets a new current secret key that OM does not have yet, for example through {{ozone admin scm rotate --force}}.
> # {{ozone admin om fetch-key}} is run.
> h3. Impact
> * The command reports success and a key ID, but OM keeps signing with its old key until its own poller fetches the new one, which happens once the old key is older than {{hdds.secret.key.rotate.duration}} (1d). OM cannot be made to switch earlier, which is what the command is for, short of restarting it.
> * The printed ID is not SCM's current key, without any hint.
> * Nothing else is affected. The old key stays valid on SCM until it expires, so tokens that OM issues in the meantime are accepted.
> h3. Reproduction
> PASS, deterministic, unmodified source, under 1 second. [^TestBugCR11FetchKeyDoesNotReachSigner.java] ({{hadoop-hdds/framework}}) creates the client the way OM does, with a mock SCM that has a new current key after the first fetch, and makes the two calls of {{OzoneManager.refetchSecretKey}}. The key is unchanged and SCM was called once (by {{start}}). A second test is the control: the same sequence on {{DefaultSecretKeySignerClient}} alone returns the new key. A passing test means the defect is present.
> On a secure mini cluster the command itself was run through {{TestBlockTokensCLI.testFetchKeyOMAdminCommand}} with a forced rotation on SCM added before it (this is the regression test of the patch): without the fix it fails with "expected: <3f7f03d9-...> but was: <e13b7dfc-...>", the first being SCM's current key and the second the ID that the command printed.
> h3. Patch
> [^CR-11-let-om-fetch-key-reach-secret-key-signer-client.patch], against 025266b52a6dcaf39508607cad736799c40907e8. It also applies to master at 4389803780f.
> {{DefaultSecretKeyClient.refetchSecretKey}} now passes the call on to the signer client, as it does for the other signer methods. The verifier side is not touched. {{OzoneManager.refetchSecretKey}} rethrows the cause of the {{UncheckedIOException}} of the signer client, so that a failed fetch from SCM is returned to the caller as an error.
> {{TestBlockTokensCLI.testFetchKeyOMAdminCommand}} now rotates the key on SCM first, so that the printed ID can only match if OM really fetched it. The existing test compared the printed ID with SCM's current key without a rotation in between, which is why it kept passing. {{TestOMAdminRpcAuthorization}} gets a test in which the fetch fails and the caller has to receive the error.
> With the patch {{TestBlockTokensCLI}} (3 tests), {{TestOMAdminRpcAuthorization}} (5 tests) and {{TestBlockTokens}} (5 tests) pass on a mini cluster, the secret key and token unit suites of {{hadoop-hdds/framework}} (38 tests) pass, and checkstyle is clean. Without the two main changes the changed test and the new test fail. With the patch the reproduction fails at its assertion that the key is unchanged.
> Found by code review of the SCM secret key rotation and distribution paths, as part of the TLA+ verification effort under HDDS-15926, on commit 025266b52a6dcaf39508607cad736799c40907e8. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org