You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2026/10/22 11:34:00 UTC
[jira] [Updated] (HDDS-17321) Bound client retry and timeout waits to avoid long silent hangs
[ https://issues.apache.org/jira/browse/HDDS-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siyao Meng updated HDDS-17321:
------------------------------
Description:
Several Ozone clients wait far longer than needed, often with nothing printed, because a retry wait grows without bound, a timeout is retried many times on the same server, an error that will not change is retried, or a timeout setting is read in the wrong unit. For example, a CLI command started before a single OM is ready keeps sleeping long after OM is up, and an unreachable OM or SCM costs about 15 minutes per connection attempt.
This epic groups fixes that keep these waits bounded and in line with the configured values, without changing behavior on the normal path.
h3. Fixes
* HDDS-17302 OM client retry wait time grows without bound with a single OM
* HDDS-17310 OM and SCM clients retry a connect timeout on the same server for about 15 minutes
* HDDS-17312 gRPC client read and EC write timeouts are truncated to whole seconds
* HDDS-17314 Make hdds.heartbeat.recon.interval control the Datanode to Recon heartbeat interval
* HDDS-17316 Cap the wait between retries when loading the initial secret key from SCM
* HDDS-17318 SCM client retries non-retriable server errors, safemode wait ignores its timeout
* HDDS-17320 SCM client does not recognize RetriableWithNoFailoverException returned by SCM
h3. Open
* HDDS-17322 Datanode heartbeat to an unreachable SCM or Recon can block about 15 minutes on connect timeout retries
* HDDS-17323 OM admin client nests two retry loops and retries AccessControlException
* HDDS-17324 Ozone CLI waits forever for the CA list when SCM has fewer CA certificates than configured SCM nodes
* HDDS-17325 GrpcOmTransport has no call deadline and ignores interrupts, so a stuck OM blocks the client indefinitely
* HDDS-17326 Client keeps waiting for ALL_COMMITTED on a pipeline with a failed follower
* HDDS-17327 Topology-aware read tries an unresponsive nearest datanode first for every ReadChunk
* HDDS-17328 EC read waits about 2 minutes on an unresponsive datanode before switching to reconstruction
* HDDS-17329 Ratis client write can take about 2 minutes although hdds.ratis.client.request.write.timeout is 70s
* HDDS-11343 (existing) OM "leader not ready" wait is not capped, and its counter is shared by all threads using one client
Related: HDDS-16382, HDDS-4409
was:
Several Ozone clients wait far longer than needed, often with nothing printed, because a retry wait grows without bound, a timeout is retried many times on the same server, an error that will not change is retried, or a timeout setting is read in the wrong unit. For example, a CLI command started before a single OM is ready keeps sleeping long after OM is up, and an unreachable OM or SCM costs about 15 minutes per connection attempt.
This epic groups fixes that keep these waits bounded and in line with the configured values, without changing behavior on the normal path.
h3. Fixes
* HDDS-17302 OM client retry wait time grows without bound with a single OM
* HDDS-17310 OM and SCM clients retry a connect timeout on the same server for about 15 minutes
* HDDS-17312 gRPC client read and EC write timeouts are truncated to whole seconds
* HDDS-17314 Make hdds.heartbeat.recon.interval control the Datanode to Recon heartbeat interval
* HDDS-17316 Cap the wait between retries when loading the initial secret key from SCM
* HDDS-17318 SCM client retries non-retriable server errors, safemode wait ignores its timeout
* HDDS-17320 SCM client does not recognize RetriableWithNoFailoverException returned by SCM
h3. Candidates
* OM "leader not ready" wait is not capped, and its counter is shared by all threads using one client
* Datanode heartbeat client to SCM still retries a connect timeout 45 times (related: HDDS-17269)
* OM admin client nests two retry loops and retries AccessControlException
* CA list fetch ({{HAUtils.getCAListWithRetry}}) retries forever
* Data path: a dead follower adds a watch timeout to every flush (related: HDDS-10978), and reads retry a hung nearest replica first (related: HDDS-16419)
Related: HDDS-16382, HDDS-4409
> Bound client retry and timeout waits to avoid long silent hangs
> ---------------------------------------------------------------
>
> Key: HDDS-17321
> URL: https://issues.apache.org/jira/browse/HDDS-17321
> Project: Apache Ozone
> Issue Type: Epic
> Components: Ozone Client
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> Several Ozone clients wait far longer than needed, often with nothing printed, because a retry wait grows without bound, a timeout is retried many times on the same server, an error that will not change is retried, or a timeout setting is read in the wrong unit. For example, a CLI command started before a single OM is ready keeps sleeping long after OM is up, and an unreachable OM or SCM costs about 15 minutes per connection attempt.
> This epic groups fixes that keep these waits bounded and in line with the configured values, without changing behavior on the normal path.
> h3. Fixes
> * HDDS-17302 OM client retry wait time grows without bound with a single OM
> * HDDS-17310 OM and SCM clients retry a connect timeout on the same server for about 15 minutes
> * HDDS-17312 gRPC client read and EC write timeouts are truncated to whole seconds
> * HDDS-17314 Make hdds.heartbeat.recon.interval control the Datanode to Recon heartbeat interval
> * HDDS-17316 Cap the wait between retries when loading the initial secret key from SCM
> * HDDS-17318 SCM client retries non-retriable server errors, safemode wait ignores its timeout
> * HDDS-17320 SCM client does not recognize RetriableWithNoFailoverException returned by SCM
> h3. Open
> * HDDS-17322 Datanode heartbeat to an unreachable SCM or Recon can block about 15 minutes on connect timeout retries
> * HDDS-17323 OM admin client nests two retry loops and retries AccessControlException
> * HDDS-17324 Ozone CLI waits forever for the CA list when SCM has fewer CA certificates than configured SCM nodes
> * HDDS-17325 GrpcOmTransport has no call deadline and ignores interrupts, so a stuck OM blocks the client indefinitely
> * HDDS-17326 Client keeps waiting for ALL_COMMITTED on a pipeline with a failed follower
> * HDDS-17327 Topology-aware read tries an unresponsive nearest datanode first for every ReadChunk
> * HDDS-17328 EC read waits about 2 minutes on an unresponsive datanode before switching to reconstruction
> * HDDS-17329 Ratis client write can take about 2 minutes although hdds.ratis.client.request.write.timeout is 70s
> * HDDS-11343 (existing) OM "leader not ready" wait is not capped, and its counter is shared by all threads using one client
> Related: HDDS-16382, HDDS-4409
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org