You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/12/08 09:56:09 UTC
[GitHub] [ozone] lokeshj1703 opened a new pull request #1675: HDDS-4563. Increase default value for key and block deletion limit.
lokeshj1703 opened a new pull request #1675:
URL: https://github.com/apache/ozone/pull/1675
## What changes were proposed in this pull request?
Currently ozone.key.deleting.limit.per.task defaults to 1000 and hdds.scm.block.deletion.per-interval.max defaults to 10000. These config values can be increased to 20000 based on the latest block deletion results.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4563
## How was this patch tested?
Existing UT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #1675: HDDS-4563. Increase default value for key and block deletion limit.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #1675:
URL: https://github.com/apache/ozone/pull/1675#discussion_r541561230
##########
File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
##########
@@ -85,7 +85,7 @@ private OMConfigKeys() {
public static final String OZONE_KEY_DELETING_LIMIT_PER_TASK =
"ozone.key.deleting.limit.per.task";
- public static final int OZONE_KEY_DELETING_LIMIT_PER_TASK_DEFAULT = 1000;
+ public static final int OZONE_KEY_DELETING_LIMIT_PER_TASK_DEFAULT = 20000;
Review comment:
Should also be changed in https://github.com/apache/ozone/blob/19dd94d594cfab0308b778dba367ecc62ed0f47e/hadoop-hdds/common/src/main/resources/ozone-default.xml#L396-L397
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on a change in pull request #1675: HDDS-4563. Increase default value for key and block deletion limit.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on a change in pull request #1675:
URL: https://github.com/apache/ozone/pull/1675#discussion_r542265423
##########
File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
##########
@@ -85,7 +85,7 @@ private OMConfigKeys() {
public static final String OZONE_KEY_DELETING_LIMIT_PER_TASK =
"ozone.key.deleting.limit.per.task";
- public static final int OZONE_KEY_DELETING_LIMIT_PER_TASK_DEFAULT = 1000;
+ public static final int OZONE_KEY_DELETING_LIMIT_PER_TASK_DEFAULT = 20000;
Review comment:
Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on pull request #1675: HDDS-4563. Increase default value for key and block deletion limit.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on pull request #1675:
URL: https://github.com/apache/ozone/pull/1675#issuecomment-748830042
@adoroszlai @bharatviswa504 Thanks for reviewing the PR! I have committed it to master branch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on pull request #1675: HDDS-4563. Increase default value for key and block deletion limit.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on pull request #1675:
URL: https://github.com/apache/ozone/pull/1675#issuecomment-746129085
@adoroszlai I have updated these links in the PR description as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on pull request #1675: HDDS-4563. Increase default value for key and block deletion limit.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on pull request #1675:
URL: https://github.com/apache/ozone/pull/1675#issuecomment-744336023
@adoroszlai Thanks for reviewing the PR! I have updated the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on pull request #1675: HDDS-4563. Increase default value for key and block deletion limit.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on pull request #1675:
URL: https://github.com/apache/ozone/pull/1675#issuecomment-746127846
@adoroszlai The results were uploaded in HDDS-4344.
https://issues.apache.org/jira/browse/HDDS-4344?focusedCommentId=17237974&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17237974
https://issues.apache.org/jira/browse/HDDS-4344?focusedCommentId=17242233&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17242233
With 20000 keys/min deletion rate at OM, minimum time for OM to delete 1 million blocks is 1000000/20000 = 50 minutes. HDFS take around 60-70 minutes to delete 1 million blocks in a 10 node cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai edited a comment on pull request #1675: HDDS-4563. Increase default value for key and block deletion limit.
Posted by GitBox <gi...@apache.org>.
adoroszlai edited a comment on pull request #1675:
URL: https://github.com/apache/ozone/pull/1675#issuecomment-746056144
Thanks for fixing the config test.
> These config values can be increased to 20000 based on the latest block deletion results.
@lokeshj1703 Can you please share these results or the rationale for the change? I think it would be useful for admins to understand what affects these settings and how they can tune it based on the actual environment Ozone is deployed to. Otherwise 1000/10000 and 20000 are just opaque numbers (at least to me).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #1675: HDDS-4563. Increase default value for key and block deletion limit.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #1675:
URL: https://github.com/apache/ozone/pull/1675#issuecomment-746056144
> These config values can be increased to 20000 based on the latest block deletion results.
@lokeshj1703 Can you please share these results or the rationale for the change? I think it would be useful for admins to understand what affects these settings and how they can tune it based on the actual environment Ozone is deployed to. Otherwise 1000/10000 and 20000 are just opaque numbers (at least to me).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 closed pull request #1675: HDDS-4563. Increase default value for key and block deletion limit.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 closed pull request #1675:
URL: https://github.com/apache/ozone/pull/1675
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on pull request #1675: HDDS-4563. Increase default value for key and block deletion limit.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on pull request #1675:
URL: https://github.com/apache/ozone/pull/1675#issuecomment-748827233
> Question: SCM sending 20000 block deletes in a single RPC Call will it have any effect on client writes?
@bharatviswa504 Block deletion happens in background right now for datanode so the deletions do not happen instantaneously. Datanode would have to store these 20000 blocks in container rocksdb which would lead to some disk usage.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org