[jira] [Created] (HDDS-16858) Tenant requests release the authorizer write lock only in their apply, so a request that is not applied on this OM leaves the lock held and later tenant operations time out

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-16858:
---------------------------------

             Summary: Tenant requests release the authorizer write lock only in their apply, so a request that is not applied on this OM leaves the lock held and later tenant operations time out
                 Key: HDDS-16858
                 URL: https://issues.apache.org/jira/browse/HDDS-16858
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: MC-1-release-tenant-authorizer-lock-on-ratis-submit-failure.patch

h3. Mechanism
Each of the six tenant requests ({{OMTenantCreateRequest}}, {{OMTenantDeleteRequest}}, {{OMTenantAssignUserAccessIdRequest}}, {{OMTenantRevokeUserAccessIdRequest}}, {{OMTenantAssignAdminRequest}}, {{OMTenantRevokeAdminRequest}}) takes the OM wide authorizer write lock in {{preExecute}} through {{AuthorizerLock.tryWriteLockInOMRequest}}, changes Ranger, and releases the lock only in the {{finally}} of its {{validateAndUpdateCache}}. {{AuthorizerLockImpl}} records a single stamp and holder thread for this lock.

{{OMExecutionFlow.submitExecutionToRatis}} has no step that releases the lock when the request is not applied on this OM. When {{OzoneManagerRatisServer.submitRequest}} throws {{ServiceException}} (for example from {{NotLeaderException}}, {{LeaderSteppingDownException}} or {{ResourceUnavailableException}}), it calls nothing, and when it returns a failed response it calls only {{OMClientRequest.handleRequestFailure}}, which is empty. The tenant requests stopped overriding {{handleRequestFailure}} when the lock was introduced in HDDS-6701.

The OM is then left with the stamp recorded and no tenant request in flight. Each later tenant request on that OM fails in {{preExecute}} with {{TIMEOUT}} ("Timed out acquiring authorizer write lock. Another multi-tenancy request is in-progress. Try again later"), and the Ranger background sync cannot take its read lock either (from {{OMRangerBGSyncService}}, not run). Because {{unlockWriteInOMRequest}} releases whatever stamp is recorded, the lock is freed only when this OM applies a tenant log entry or restarts.

h3. Impact
* Tenant operations (create, delete, user assign and revoke, admin assign and revoke) fail with {{TIMEOUT}} on the affected OM while it is leader, until it applies a tenant log entry or restarts.
* The Ranger background sync makes no progress on that OM in the meantime (code reading, not run).
* No data loss. Key, bucket and S3 operations do not use this lock.

h3. Workaround
Move leadership to another OM ({{ozone admin om transfer}}) and run a tenant operation there that succeeds. Once the affected OM applies that entry, its lock is released. Restarting the affected OM also releases it.

h3. Patch
[^MC-1-release-tenant-authorizer-lock-on-ratis-submit-failure.patch], against a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at b0aa6475b78.

The lock is released whatever the submit outcome. {{OMExecutionFlow}} calls a new hook, {{OMClientRequest.afterSubmit}}, in a {{finally}} after the submit to Ratis. It is a no op by default, and the six tenant requests override it to call the new {{AuthorizerLock.unlockWriteInOMRequestIfHeldByCurrentThread}}, which releases the lock only if the current handler thread is its recorded holder. If the request was applied, its apply has already cleared the holder before the reply was completed, so the call does nothing and a lock taken meanwhile by the next tenant request is left alone. Non tenant requests are not affected, and there is no protocol, config or apply change.

The issue has no standalone reproduction. The regression tests are in the existing {{TestOMTenantDeleteRequest}}: {{lockNotHeldAfterSubmitWhateverTheOutcome}} (each submit outcome), {{responseAfterApplyKeepsLockOfNextRequest}} (the request was applied and the next request took the lock before the response was returned) and {{tenantRequestReleasesOnlyOwnLockAfterSubmit}} (each of the six tenant requests). Without the change the first and third fail because the lock is still held; the second guards against an unconditional unlock. With it the tenant and S3 secret request suites, {{TestOMExecutionFlow}}, {{TestAuthorizerLockImpl}}, the multi tenant manager suites and the OM Ratis suites (165 tests) pass, as do {{TestMultiTenantVolume}} and {{TestOzoneTenantShell}} on a mini cluster (11 tests). Checkstyle is clean.

HDDS-7184 reported the same symptom ("Timed out acquiring authorizer write lock" with no multi-tenancy request in progress) and was resolved in 2023 as probably fixed by HDDS-7755, after the failing test stopped failing. This lock leak is another way to reach that symptom.

Suggested priority: Major. Tenant operations stop on one OM until it applies a tenant entry from another leader or restarts. No data is lost and the key and S3 paths are not affected.

Found by TLA+ model checking and code review of the OM multi-tenancy and Ranger sync paths under HDDS-15926, on commit a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and apache/ozone pull requests before filing; no existing issue covers it. Related: HDDS-6701 (introduced the lock and removed the tenant {{handleRequestFailure}} overrides), HDDS-7184 (same symptom, resolved). The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org