You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Peter Lee (Jira)" <ji...@apache.org> on 2025/03/05 18:11:00 UTC
[jira] [Assigned] (HDDS-11343) S3G randomly selects retry time when retrying OM
[ https://issues.apache.org/jira/browse/HDDS-11343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Lee reassigned HDDS-11343:
--------------------------------
Assignee: Peter Lee
> S3G randomly selects retry time when retrying OM
> ------------------------------------------------
>
> Key: HDDS-11343
> URL: https://issues.apache.org/jira/browse/HDDS-11343
> Project: Apache Ozone
> Issue Type: Improvement
> Components: s3gateway
> Affects Versions: 1.4.0
> Reporter: JiangHua Zhu
> Assignee: Peter Lee
> Priority: Major
>
> When S3G retries OM, the rpc sleep time is the product of {ozone.client.wait.between.retries.millis} and {ozone.client.failover.max.attempts}.
> sleeping time = {ozone.client.wait.between.retries.millis}*{ozone.client.failover.max.attempts}
> When retry=1, sleep time = 2000ms.
> When retry=2, sleep time = 4000ms.
> ......
> Here are some S3Gateway logs:
> {code:java}
> 2024-08-19 10:04:01,978 [Thread-13636] INFO org.apache.hadoop.io.retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.net.ConnectException: Call From xxx.xxx.xxx.xxx/xx.xx.xx.xx to xxx.xxx.xxx.xxx:9862 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused, while invoking $Proxy124.submitRequest over nodeId=null,nodeAddress=xxx.xxx.xxx.xxx:9862. Trying to failover after sleeping for 24000ms. Current retry count: 0.
> {code}
> {code:java}
> 2024-08-19 10:05:51,985 [Thread-14359] INFO org.apache.hadoop.io.retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.net.ConnectException: Call From xxx.xxx.xxx.xxx/xxx.xxx.xxx.xxx to xx.xx.xx.xx:9862 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused, while invoking $Proxy124.submitRequest over nodeId=null,nodeAddress=xxx.xxx.xxx.xxx:9862. Trying to failover after sleeping for 134000ms. Current retry count: 0.
> {code}
> We see that the sleep time reaches tens of seconds or even longer. Each increase here is very expensive.
> We can optimize as much as possible without changing the linear probability. When choosing the sleep time, you can get a random value from the calculated value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org