[jira] [Assigned] (HDDS-16596) Isolate synchronous Ratis-commit and SCM RPC waits from the OM client handler pool so a transient bucket-lock convoy cannot escalate to full-pool exhaustion

["Andrey Yarovoy (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrey Yarovoy reassigned HDDS-16596:
-------------------------------------

    Assignee: Andrey Yarovoy

> Isolate synchronous Ratis-commit and SCM RPC waits from the OM client handler pool so a transient bucket-lock convoy cannot escalate to full-pool exhaustion
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16596
>                 URL: https://issues.apache.org/jira/browse/HDDS-16596
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: OM
>            Reporter: Andrey Yarovoy
>            Assignee: Andrey Yarovoy
>            Priority: Major
>
> h2. Problem
> The OM services client RPCs on a single shared IPC handler pool ({{{}ozone.om.handler.count.key{}}}). Write requests block their handler thread synchronously for the duration of the Ratis commit — {{OzoneManagerRatisServer.submitRequestToRatis}} parks the handler in {{CompletableFuture.get()}} until the write is replicated — and outbound SCM calls ({{{}allocateBlock{}}}, container/pipeline location) likewise park their handler for the whole round trip inside {{RetryInvocationHandler}} / {{{}Client.getRpcResponse{}}}.
> Because these waits sit on the same pool that serves reads, the pool has no reserved headroom. Under sustained write load a large fraction of handlers are permanently parked on commit, so the pool runs at near-zero idle at rest. When a transient contention event then arrives — for example a bucket read/write-lock convoy that would otherwise drain in microseconds — the arriving requests consume the few remaining handlers and the localized stall escalates into a full-pool outage affecting unrelated buckets and read RPCs.
> h2. Evidence
> On the heavy loaded cluster with mixed IO patterns per-thread analysis of OM stack dumps from a saturated OM (200-thread handler pool) versus a healthy OM under comparable client load:
> The two profiles carry the same peak lock-convoy magnitude (145 vs 133 threads), so the convoy itself is not the differentiator. The differentiator is headroom: the saturated OM held ~93% of its handlers in commit waits with zero idle, so a convoy had nothing to absorb it; the healthy OM held ~93% of handlers idle and absorbed convoys of similar size without a pool-wide stall.
> h2. Proposed direction
> Decouple the blocking waits from the client-facing handler pool so a transient convoy cannot consume the whole pool:
>  * Complete the Ratis commit asynchronously (release the handler once the request is submitted, respond on commit) rather than blocking the handler in {{{}get(){}}}, *or* run commit waits on a bounded executor separate from the handler pool.
>  * Isolate outbound SCM RPCs onto a bounded pool with backpressure so a slow SCM cannot exhaust client handlers.
>  * As a floor, ensure read RPCs retain reserved capacity independent of in-flight write commits ({{{}ozone.om.read.threadpool{}}}, default 10, is small relative to the handler pool).
> Each has correctness/backpressure implications (in-flight accounting, retry/failover, ordering) and should be scoped as design subtasks.
> h2. Testing
>  * A concurrency benchmark that saturates the write path and confirms read-RPC latency and idle-handler count stay bounded while commits are in flight.
>  * A fault-injection test with a deliberately slow downstream (Ratis or SCM) asserting the handler pool retains headroom rather than reaching full occupancy.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org