[jira] [Created] (HDDS-17325) GrpcOmTransport has no call deadline and ignores interrupts, so a stuck OM blocks the client indefinitely

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-17325:
---------------------------------

             Summary: GrpcOmTransport has no call deadline and ignores interrupts, so a stuck OM blocks the client indefinitely
                 Key: HDDS-17325
                 URL: https://issues.apache.org/jira/browse/HDDS-17325
             Project: Apache Ozone
          Issue Type: Bug
          Components: Ozone Client
            Reporter: Siyao Meng
            Assignee: Siyao Meng


When {{ozone.om.transport.class}} selects the gRPC transport (not the default since HDDS-9446, but used for S3 Gateway in some deployments), {{GrpcOmTransport}}:
* calls {{submitRequest}} on a blocking stub without a deadline ({{withDeadlineAfter}}), and there is no config key for a call timeout. If the OM accepts the request but never answers (e.g. a stuck handler), the call blocks with no limit, and the failover and retry logic is never reached. A half-open connection is only closed by TCP, after 15 minutes or more.
* sleeps between retries inside {{catch (Exception e)}}, which only logs, so an interrupt is lost and the retry continues.

For S3 Gateway, each request that reaches the stuck OM holds a gateway thread until the gateway is restarted.

Proposed fix: set a per-call deadline from a new config key (a {{StatusRuntimeException}} already goes through the retry and failover policy), and restore the interrupt flag and stop retrying when the thread is interrupted.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org