You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ptlrs (via GitHub)" <gi...@apache.org> on 2026/06/10 16:52:53 UTC
[PR] HDDS-15444. Adjusted Ratis client retry, timeout configs to improve failure responsiveness [ozone]
ptlrs opened a new pull request, #10482:
URL: https://github.com/apache/ozone/pull/10482
## What changes were proposed in this pull request?
Some of the configuration values for the ratis client are too long such that they make the client appear non-responsive.
This jira updates some of the configurations to make the client be more responsive to failures on the server side
- Exponential Backoff (for TimeoutIOException on writes):
- Base sleep: 4s to 1s — faster initial retry
- Max sleep cap: 40s to 5s — don't wait long between retries on a dead leader
- Max retries: unlimited (Integer.MAX_VALUE) to 2 — if 2 retries fail, the leader is dead. We will let Ozone allocate a new pipeline
- Multilinear Random Retry (for generic/other exceptions):
- Policy: 5s×5, 10s×5, 15s×5, 20s×5, 25s×5, 60s×10 (~16 min, 35 retries) to 5s×6 (~30s total) — fail fast instead of hanging
- Watch Timeout (waiting for ALL_COMMITTED replication):
- Overall watch timeout: 3 min to 30s
- Watch RPC timeout: 180s to 30s — aligned with server-side watch timeout (also 30s). There's no point waiting longer than the server
- Write Timeout (overall budget for write retries):
- Write request timeout: 5 min to 70s — enough for one RPC (60s) + buffer. This prevents retries from dragging on
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15444
## How was this patch tested?
CI: https://github.com/ptlrs/ozone/actions/runs/27291093969
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15444. Adjusted Ratis client retry, timeout configs to improve failure responsiveness [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10482:
URL: https://github.com/apache/ozone/pull/10482#discussion_r3393436390
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/conf/RatisClientConfig.java:
##########
@@ -70,31 +70,31 @@ public class RatisClientConfig {
private String multilinearPolicy;
@Config(key = "hdds.ratis.client.exponential.backoff.base.sleep",
- defaultValue = "4s",
+ defaultValue = "1s",
type = ConfigType.TIME,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "Specifies base sleep for exponential backoff retry policy."
- + " With the default base sleep of 4s, the sleep duration for ith"
- + " retry is min(4 * pow(2, i), max_sleep) * r, where r is "
+ + " With the default base sleep of 1s, the sleep duration for ith"
+ + " retry is min(1 * pow(2, i), max_sleep) * r, where r is "
+ "random number in the range [0.5, 1.5).")
- private Duration exponentialPolicyBaseSleep = Duration.ofSeconds(4);
+ private Duration exponentialPolicyBaseSleep = Duration.ofSeconds(1);
@Config(key = "hdds.ratis.client.exponential.backoff.max.sleep",
- defaultValue = "40s",
+ defaultValue = "5s",
type = ConfigType.TIME,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "The sleep duration obtained from exponential backoff "
+ "policy is limited by the configured max sleep. Refer "
+ "dfs.ratis.client.exponential.backoff.base.sleep for further "
+ "details.")
- private Duration exponentialPolicyMaxSleep = Duration.ofSeconds(40);
+ private Duration exponentialPolicyMaxSleep = Duration.ofSeconds(5);
@Config(key = "hdds.ratis.client.exponential.backoff.max.retries",
- defaultValue = "2147483647",
+ defaultValue = "2",
type = ConfigType.INT,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "Client's max retry value for the exponential backoff policy.")
- private int exponentialPolicyMaxRetries = Integer.MAX_VALUE;
+ private int exponentialPolicyMaxRetries = 2;
Review Comment:
So all the changes combined theoretically:
1. max total retry time widow would be 2 * max_sleep * 1.5 = 15 second.
2. min total retry time window would be 1 * 2 * 0.5 + 1 * 4 * 0.5 = 3 second.
Just curious if the 3 to 15 seconds time window is enough to decide if the leader is not reachable after the initial timeout?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15444. Adjusted Ratis client retry, timeout configs to improve failure responsiveness [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10482:
URL: https://github.com/apache/ozone/pull/10482#discussion_r3393455898
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/conf/RatisClientConfig.java:
##########
@@ -70,31 +70,31 @@ public class RatisClientConfig {
private String multilinearPolicy;
@Config(key = "hdds.ratis.client.exponential.backoff.base.sleep",
- defaultValue = "4s",
+ defaultValue = "1s",
type = ConfigType.TIME,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "Specifies base sleep for exponential backoff retry policy."
- + " With the default base sleep of 4s, the sleep duration for ith"
- + " retry is min(4 * pow(2, i), max_sleep) * r, where r is "
+ + " With the default base sleep of 1s, the sleep duration for ith"
+ + " retry is min(1 * pow(2, i), max_sleep) * r, where r is "
+ "random number in the range [0.5, 1.5).")
- private Duration exponentialPolicyBaseSleep = Duration.ofSeconds(4);
+ private Duration exponentialPolicyBaseSleep = Duration.ofSeconds(1);
@Config(key = "hdds.ratis.client.exponential.backoff.max.sleep",
- defaultValue = "40s",
+ defaultValue = "5s",
type = ConfigType.TIME,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "The sleep duration obtained from exponential backoff "
+ "policy is limited by the configured max sleep. Refer "
+ "dfs.ratis.client.exponential.backoff.base.sleep for further "
+ "details.")
- private Duration exponentialPolicyMaxSleep = Duration.ofSeconds(40);
+ private Duration exponentialPolicyMaxSleep = Duration.ofSeconds(5);
@Config(key = "hdds.ratis.client.exponential.backoff.max.retries",
- defaultValue = "2147483647",
+ defaultValue = "2",
type = ConfigType.INT,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "Client's max retry value for the exponential backoff policy.")
- private int exponentialPolicyMaxRetries = Integer.MAX_VALUE;
+ private int exponentialPolicyMaxRetries = 2;
Review Comment:
Oh Sorry I might be confused myself:
I guess what retries here control when to retry and there is another request timeout. So update the number as:
1. max total retry time widow would be 2 * max_sleep * 1.5 + 2 * write_request_time_out = 2 * 5 * 1.5 + 2 * 70 = 155 seconds.
2. min total retry time window would be 1 * 2 * 0.5 + 1 * 4 * 0.5 + 2 * write_request_time_out = 143 second.
After the initial timeout, there are around 143 to 155 seconds before giving up, which seems to be sufficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15444. Adjusted Ratis client retry, timeout configs to improve failure responsiveness [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10482:
URL: https://github.com/apache/ozone/pull/10482#discussion_r3393455898
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/conf/RatisClientConfig.java:
##########
@@ -70,31 +70,31 @@ public class RatisClientConfig {
private String multilinearPolicy;
@Config(key = "hdds.ratis.client.exponential.backoff.base.sleep",
- defaultValue = "4s",
+ defaultValue = "1s",
type = ConfigType.TIME,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "Specifies base sleep for exponential backoff retry policy."
- + " With the default base sleep of 4s, the sleep duration for ith"
- + " retry is min(4 * pow(2, i), max_sleep) * r, where r is "
+ + " With the default base sleep of 1s, the sleep duration for ith"
+ + " retry is min(1 * pow(2, i), max_sleep) * r, where r is "
+ "random number in the range [0.5, 1.5).")
- private Duration exponentialPolicyBaseSleep = Duration.ofSeconds(4);
+ private Duration exponentialPolicyBaseSleep = Duration.ofSeconds(1);
@Config(key = "hdds.ratis.client.exponential.backoff.max.sleep",
- defaultValue = "40s",
+ defaultValue = "5s",
type = ConfigType.TIME,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "The sleep duration obtained from exponential backoff "
+ "policy is limited by the configured max sleep. Refer "
+ "dfs.ratis.client.exponential.backoff.base.sleep for further "
+ "details.")
- private Duration exponentialPolicyMaxSleep = Duration.ofSeconds(40);
+ private Duration exponentialPolicyMaxSleep = Duration.ofSeconds(5);
@Config(key = "hdds.ratis.client.exponential.backoff.max.retries",
- defaultValue = "2147483647",
+ defaultValue = "2",
type = ConfigType.INT,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "Client's max retry value for the exponential backoff policy.")
- private int exponentialPolicyMaxRetries = Integer.MAX_VALUE;
+ private int exponentialPolicyMaxRetries = 2;
Review Comment:
Oh Sorry I might be confused myself:
I guess what retries here control when to retry and there is another request time out. So update the number is:
1. max total retry time widow would be 2 * max_sleep * 1.5 + 2 * write_request_time_out = 2 * 5 * 1.5 + 2 * 70 = 155 seconds.
2. min total retry time window would be 1 * 2 * 0.5 + 1 * 4 * 0.5 + 2 * write_request_time_out = 143 second.
After the initial timeout, there are around 143 to 155 seconds before giving up, which seems to be sufficient.
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/conf/RatisClientConfig.java:
##########
@@ -70,31 +70,31 @@ public class RatisClientConfig {
private String multilinearPolicy;
@Config(key = "hdds.ratis.client.exponential.backoff.base.sleep",
- defaultValue = "4s",
+ defaultValue = "1s",
type = ConfigType.TIME,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "Specifies base sleep for exponential backoff retry policy."
- + " With the default base sleep of 4s, the sleep duration for ith"
- + " retry is min(4 * pow(2, i), max_sleep) * r, where r is "
+ + " With the default base sleep of 1s, the sleep duration for ith"
+ + " retry is min(1 * pow(2, i), max_sleep) * r, where r is "
+ "random number in the range [0.5, 1.5).")
- private Duration exponentialPolicyBaseSleep = Duration.ofSeconds(4);
+ private Duration exponentialPolicyBaseSleep = Duration.ofSeconds(1);
@Config(key = "hdds.ratis.client.exponential.backoff.max.sleep",
- defaultValue = "40s",
+ defaultValue = "5s",
type = ConfigType.TIME,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "The sleep duration obtained from exponential backoff "
+ "policy is limited by the configured max sleep. Refer "
+ "dfs.ratis.client.exponential.backoff.base.sleep for further "
+ "details.")
- private Duration exponentialPolicyMaxSleep = Duration.ofSeconds(40);
+ private Duration exponentialPolicyMaxSleep = Duration.ofSeconds(5);
@Config(key = "hdds.ratis.client.exponential.backoff.max.retries",
- defaultValue = "2147483647",
+ defaultValue = "2",
type = ConfigType.INT,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "Client's max retry value for the exponential backoff policy.")
- private int exponentialPolicyMaxRetries = Integer.MAX_VALUE;
+ private int exponentialPolicyMaxRetries = 2;
Review Comment:
So all the changes combined theoretically, for write requests:
1. max total retry time widow would be 2 * max_sleep * 1.5 = 2 * 5 * 1.5 = 15 second.
2. min total retry time window would be 1 * 2 * 0.5 + 1 * 4 * 0.5 = 3 second.
Just curious if the 3 to 15 seconds time window is enough to decide if the leader is not reachable after the initial timeout?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15444. Adjusted Ratis client retry, timeout configs to improve failure responsiveness [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10482:
URL: https://github.com/apache/ozone/pull/10482#discussion_r3393382334
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/conf/RatisClientConfig.java:
##########
@@ -70,31 +70,31 @@ public class RatisClientConfig {
private String multilinearPolicy;
@Config(key = "hdds.ratis.client.exponential.backoff.base.sleep",
- defaultValue = "4s",
+ defaultValue = "1s",
type = ConfigType.TIME,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "Specifies base sleep for exponential backoff retry policy."
- + " With the default base sleep of 4s, the sleep duration for ith"
- + " retry is min(4 * pow(2, i), max_sleep) * r, where r is "
+ + " With the default base sleep of 1s, the sleep duration for ith"
+ + " retry is min(1 * pow(2, i), max_sleep) * r, where r is "
Review Comment:
Do we know where is the `max_sleep` defined? What is its default value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15444. Adjusted Ratis client retry, timeout configs to improve failure responsiveness [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10482:
URL: https://github.com/apache/ozone/pull/10482#discussion_r3393436390
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/conf/RatisClientConfig.java:
##########
@@ -70,31 +70,31 @@ public class RatisClientConfig {
private String multilinearPolicy;
@Config(key = "hdds.ratis.client.exponential.backoff.base.sleep",
- defaultValue = "4s",
+ defaultValue = "1s",
type = ConfigType.TIME,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "Specifies base sleep for exponential backoff retry policy."
- + " With the default base sleep of 4s, the sleep duration for ith"
- + " retry is min(4 * pow(2, i), max_sleep) * r, where r is "
+ + " With the default base sleep of 1s, the sleep duration for ith"
+ + " retry is min(1 * pow(2, i), max_sleep) * r, where r is "
+ "random number in the range [0.5, 1.5).")
- private Duration exponentialPolicyBaseSleep = Duration.ofSeconds(4);
+ private Duration exponentialPolicyBaseSleep = Duration.ofSeconds(1);
@Config(key = "hdds.ratis.client.exponential.backoff.max.sleep",
- defaultValue = "40s",
+ defaultValue = "5s",
type = ConfigType.TIME,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "The sleep duration obtained from exponential backoff "
+ "policy is limited by the configured max sleep. Refer "
+ "dfs.ratis.client.exponential.backoff.base.sleep for further "
+ "details.")
- private Duration exponentialPolicyMaxSleep = Duration.ofSeconds(40);
+ private Duration exponentialPolicyMaxSleep = Duration.ofSeconds(5);
@Config(key = "hdds.ratis.client.exponential.backoff.max.retries",
- defaultValue = "2147483647",
+ defaultValue = "2",
type = ConfigType.INT,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "Client's max retry value for the exponential backoff policy.")
- private int exponentialPolicyMaxRetries = Integer.MAX_VALUE;
+ private int exponentialPolicyMaxRetries = 2;
Review Comment:
So all the changes combined theoretically:
1. max total retry time widow would be 2 * max_sleep * 1.5 = 15 second.
2. min total retry time window would be 1 * 2 * 0.5 + 1 * 4 * 0.5 = 3 second.
Just curious if the 3 to 15 seconds time window enough to decide if the leader is not reachable after the initial timeout?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15444. Adjusted Ratis client retry, timeout configs to improve failure responsiveness [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #10482:
URL: https://github.com/apache/ozone/pull/10482
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15444. Adjusted Ratis client retry, timeout configs to improve failure responsiveness [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10482:
URL: https://github.com/apache/ozone/pull/10482#discussion_r3393436390
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/conf/RatisClientConfig.java:
##########
@@ -70,31 +70,31 @@ public class RatisClientConfig {
private String multilinearPolicy;
@Config(key = "hdds.ratis.client.exponential.backoff.base.sleep",
- defaultValue = "4s",
+ defaultValue = "1s",
type = ConfigType.TIME,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "Specifies base sleep for exponential backoff retry policy."
- + " With the default base sleep of 4s, the sleep duration for ith"
- + " retry is min(4 * pow(2, i), max_sleep) * r, where r is "
+ + " With the default base sleep of 1s, the sleep duration for ith"
+ + " retry is min(1 * pow(2, i), max_sleep) * r, where r is "
+ "random number in the range [0.5, 1.5).")
- private Duration exponentialPolicyBaseSleep = Duration.ofSeconds(4);
+ private Duration exponentialPolicyBaseSleep = Duration.ofSeconds(1);
@Config(key = "hdds.ratis.client.exponential.backoff.max.sleep",
- defaultValue = "40s",
+ defaultValue = "5s",
type = ConfigType.TIME,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "The sleep duration obtained from exponential backoff "
+ "policy is limited by the configured max sleep. Refer "
+ "dfs.ratis.client.exponential.backoff.base.sleep for further "
+ "details.")
- private Duration exponentialPolicyMaxSleep = Duration.ofSeconds(40);
+ private Duration exponentialPolicyMaxSleep = Duration.ofSeconds(5);
@Config(key = "hdds.ratis.client.exponential.backoff.max.retries",
- defaultValue = "2147483647",
+ defaultValue = "2",
type = ConfigType.INT,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "Client's max retry value for the exponential backoff policy.")
- private int exponentialPolicyMaxRetries = Integer.MAX_VALUE;
+ private int exponentialPolicyMaxRetries = 2;
Review Comment:
So all the changes combined theoretically:
1. max total retry time widow would be 2 * max_sleep * 1.5 = 2 * 5 * 1.5 = 15 second.
2. min total retry time window would be 1 * 2 * 0.5 + 1 * 4 * 0.5 = 3 second.
Just curious if the 3 to 15 seconds time window is enough to decide if the leader is not reachable after the initial timeout?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15444. Adjusted Ratis client retry, timeout configs to improve failure responsiveness [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10482:
URL: https://github.com/apache/ozone/pull/10482#discussion_r3393401625
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/conf/RatisClientConfig.java:
##########
@@ -70,31 +70,31 @@ public class RatisClientConfig {
private String multilinearPolicy;
@Config(key = "hdds.ratis.client.exponential.backoff.base.sleep",
- defaultValue = "4s",
+ defaultValue = "1s",
type = ConfigType.TIME,
tags = { OZONE, CLIENT, PERFORMANCE },
description = "Specifies base sleep for exponential backoff retry policy."
- + " With the default base sleep of 4s, the sleep duration for ith"
- + " retry is min(4 * pow(2, i), max_sleep) * r, where r is "
+ + " With the default base sleep of 1s, the sleep duration for ith"
+ + " retry is min(1 * pow(2, i), max_sleep) * r, where r is "
Review Comment:
oh it is `hdds.ratis.client.exponential.backoff.max.sleep` beflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org