You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/07/21 13:33:23 UTC
[GitHub] [hadoop-ozone] maobaolong opened a new pull request #1231: HDDS-3994. Make retry policy can be set by configuration.
maobaolong opened a new pull request #1231:
URL: https://github.com/apache/hadoop-ozone/pull/1231
## What changes were proposed in this pull request?
Write object when met exception can be slower than before, and i realized HDDS-3350 using a newer retry policy replace the existing retry policy, and the old configure items has been removed, so the behavior has been changed, i guess that is a behavior incompatible, so i trying to use a policy framework to let users can specify it to which policy they want, default to the `RetryLimitedPolicy`
## What is the link to the Apache JIRA
HDDS-3994
## How was this patch tested?
Set configure key `dfs.ratis.client.request.retry.policy` to `org.apache.hadoop.hdds.ratis.retrypolicy.RetryLimitedPolicyCreator` or `org.apache.hadoop.hdds.ratis.retrypolicy.RequestTypeDependentRetryPolicyCreator`
set the OrderedAsync logLevel to TRACE by execute the following command.
curl 'http://0.0.0.0:19878/logLevel?log=org.apache.ratis.client.impl.OrderedAsync&level=TRACE'
put a file to s3g and see the behavior of the retry policy by view the s3g log.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on a change in pull request #1231: HDDS-3994. Make retry policy can be set by configuration.
Posted by GitBox <gi...@apache.org>.
maobaolong commented on a change in pull request #1231:
URL: https://github.com/apache/hadoop-ozone/pull/1231#discussion_r463928974
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
##########
@@ -112,6 +112,18 @@
public static final String
DFS_CONTAINER_RATIS_LEADER_PENDING_BYTES_LIMIT_DEFAULT = "1GB";
+ public static final String DFS_RATIS_CLIENT_REQUEST_RETRY_POLICY =
+ "dfs.ratis.client.request.retry.policy";
+
+ public static final String DFS_RATIS_CLIENT_REQUEST_MAX_RETRIES_KEY =
+ "dfs.ratis.client.request.max.retries";
+ public static final int DFS_RATIS_CLIENT_REQUEST_MAX_RETRIES_DEFAULT = 180;
+ public static final String DFS_RATIS_CLIENT_REQUEST_RETRY_INTERVAL_KEY =
+ "dfs.ratis.client.request.retry.interval";
+ public static final TimeDuration
+ DFS_RATIS_CLIENT_REQUEST_RETRY_INTERVAL_DEFAULT =
+ TimeDuration.valueOf(1000, TimeUnit.MILLISECONDS);
+
Review comment:
Yeah，Sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on a change in pull request #1231: HDDS-3994. Make retry policy can be set by configuration.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on a change in pull request #1231:
URL: https://github.com/apache/hadoop-ozone/pull/1231#discussion_r463539752
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
##########
@@ -112,6 +112,18 @@
public static final String
DFS_CONTAINER_RATIS_LEADER_PENDING_BYTES_LIMIT_DEFAULT = "1GB";
+ public static final String DFS_RATIS_CLIENT_REQUEST_RETRY_POLICY =
+ "dfs.ratis.client.request.retry.policy";
+
+ public static final String DFS_RATIS_CLIENT_REQUEST_MAX_RETRIES_KEY =
+ "dfs.ratis.client.request.max.retries";
+ public static final int DFS_RATIS_CLIENT_REQUEST_MAX_RETRIES_DEFAULT = 180;
+ public static final String DFS_RATIS_CLIENT_REQUEST_RETRY_INTERVAL_KEY =
+ "dfs.ratis.client.request.retry.interval";
+ public static final TimeDuration
+ DFS_RATIS_CLIENT_REQUEST_RETRY_INTERVAL_DEFAULT =
+ TimeDuration.valueOf(1000, TimeUnit.MILLISECONDS);
+
Review comment:
Can we add these configs to RatisClientConfig class? I would also suggest renaming some of the configs to match the existing configs like hdds.ratis.client.retrylimited.retry.interval or sth similar. This would make it easier to distinguish between the configs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on pull request #1231: HDDS-3994. Make retry policy can be set by configuration.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on pull request #1231:
URL: https://github.com/apache/hadoop-ozone/pull/1231#issuecomment-667900513
> it make the old config key cannot works from then on, so are you sure we should modify the key name of limitedretrypolicy?
@maobaolong I think we should rename these configs before GA. The old config names are very abstract and seem like applies to all retry policies.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1231: HDDS-3994. Make retry policy can be set by configuration.
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1231:
URL: https://github.com/apache/hadoop-ozone/pull/1231#issuecomment-668350729
@lokeshj1703
> I think we should rename these configs before GA. The old config names are very abstract and seem like applies to all retry policies.
OK, this fine. Thank you for your review and suggestion, now the CI checks have been passed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1231: HDDS-3994. Make retry policy can be set by configuration.
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1231:
URL: https://github.com/apache/hadoop-ozone/pull/1231#issuecomment-662792198
@ChenSammi Please take a look, thank you very mush.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong closed pull request #1231: HDDS-3994. Make retry policy can be set by configuration.
Posted by GitBox <gi...@apache.org>.
maobaolong closed pull request #1231:
URL: https://github.com/apache/hadoop-ozone/pull/1231
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 closed pull request #1231: HDDS-3994. Make retry policy can be set by configuration.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 closed pull request #1231:
URL: https://github.com/apache/hadoop-ozone/pull/1231
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1231: HDDS-3994. Make retry policy can be set by configuration.
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1231:
URL: https://github.com/apache/hadoop-ozone/pull/1231#issuecomment-666949820
@xiaoyuyao @lokeshj1703 Thanks for the discussion and glad to make consensual for us. Now, I change the default policy to the newer RequestTypeDependentRetryPolicy, PTAL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1231: HDDS-3994. Make retry policy can be set by configuration.
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1231:
URL: https://github.com/apache/hadoop-ozone/pull/1231#issuecomment-667481634
@lokeshj1703 Thanks for your review. I put the config into ratisclientconfig, and changed the names of limtedretrypolicy ralated configs, it make the old config key cannot works from then on, so are you sure we should modify the key name of limitedretrypolicy?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on pull request #1231: HDDS-3994. Make retry policy can be set by configuration.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on pull request #1231:
URL: https://github.com/apache/hadoop-ozone/pull/1231#issuecomment-668488849
@maobaolong Thanks for the contribution! I have committed the PR to master branch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1231: HDDS-3994. Make retry policy can be set by configuration.
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1231:
URL: https://github.com/apache/hadoop-ozone/pull/1231#issuecomment-662281038
@lokeshj1703 Please take a look at this PR, thank you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong closed pull request #1231: HDDS-3994. Make retry policy can be set by configuration.
Posted by GitBox <gi...@apache.org>.
maobaolong closed pull request #1231:
URL: https://github.com/apache/hadoop-ozone/pull/1231
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org