You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/01/08 09:24:09 UTC
[GitHub] [hadoop-ozone] bshashikant opened a new pull request #420:
HDDS-2854. Add a config for the write chunk executor queue limit.
bshashikant opened a new pull request #420: HDDS-2854. Add a config for the write chunk executor queue limit.
URL: https://github.com/apache/hadoop-ozone/pull/420
## What changes were proposed in this pull request?
Added a config for the write chunk executor blocking queue limit
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2854
## How was this patch tested?
Simple config. Already unit tests in place to test the config.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] sonarcloud[bot] commented on issue #420: HDDS-2854.
Add a config for the write chunk executor queue limit.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on issue #420: HDDS-2854. Add a config for the write chunk executor queue limit.
URL: https://github.com/apache/hadoop-ozone/pull/420#issuecomment-571972749
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=hadoop-ozone&pullRequest=420&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=hadoop-ozone&pullRequest=420&metric=new_coverage&view=list)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=hadoop-ozone&pullRequest=420&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=hadoop-ozone&pullRequest=420&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#420: HDDS-2854. Add a config for the write chunk executor queue limit.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #420: HDDS-2854. Add a config for the write chunk executor queue limit.
URL: https://github.com/apache/hadoop-ozone/pull/420#discussion_r364177580
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
##########
@@ -48,6 +48,12 @@
= "dfs.container.ratis.num.write.chunk.threads";
public static final int DFS_CONTAINER_RATIS_NUM_WRITE_CHUNK_THREADS_DEFAULT
= 60;
+ public static final String
+ DFS_CONTAINER_RATIS_NUM_WRITE_CHUNK_EXECUTOR_QUEUE_LIMIT
+ = "dfs.container.ratis.num.write.chunk.executor.queue.limit";
Review comment:
`num` stands for "number of ..." (eg. in `dfs.container.ratis.num.write.chunk.threads` it is "number of ... threads"). It seems to be unnecessary in this property name, "... queue limit" seems better than "number of ... queue limit".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#420: HDDS-2854. Add a config for the write chunk executor queue limit.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #420: HDDS-2854. Add a config for the write chunk executor queue limit.
URL: https://github.com/apache/hadoop-ozone/pull/420#discussion_r364175660
##########
File path: hadoop-hdds/common/src/main/resources/ozone-default.xml
##########
@@ -203,6 +203,15 @@
will use for writing chunks (60 by default).
</description>
</property>
+ <property>
+ <name>dfs.container.ratis.num.write.chunk.executor.queue.limit</name>
+ <value>60</value>
Review comment:
```suggestion
<value>1024</value>
```
to be consistent with the code and the description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] sonarcloud[bot] removed a comment on issue #420:
HDDS-2854. Add a config for the write chunk executor queue limit.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on issue #420: HDDS-2854. Add a config for the write chunk executor queue limit.
URL: https://github.com/apache/hadoop-ozone/pull/420#issuecomment-571972749
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=hadoop-ozone&pullRequest=420&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=hadoop-ozone&pullRequest=420&metric=new_coverage&view=list)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=hadoop-ozone&pullRequest=420&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=hadoop-ozone&pullRequest=420&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bshashikant merged pull request #420: HDDS-2854. Add
a config for the write chunk executor queue limit.
Posted by GitBox <gi...@apache.org>.
bshashikant merged pull request #420: HDDS-2854. Add a config for the write chunk executor queue limit.
URL: https://github.com/apache/hadoop-ozone/pull/420
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bshashikant commented on issue #420: HDDS-2854. Add
a config for the write chunk executor queue limit.
Posted by GitBox <gi...@apache.org>.
bshashikant commented on issue #420: HDDS-2854. Add a config for the write chunk executor queue limit.
URL: https://github.com/apache/hadoop-ozone/pull/420#issuecomment-572069299
Thanks @adoroszlai and @lokeshj1703 for the review. As per discussion with @mukul1987 , it was suggested that the data blocking queue size should be same as the number of max pending requests we can handle on the leader . So i removed the additional config and used the same value as that of max pending requests here. With proper tuning these 2 values, we can see batching happening during log sync.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] sonarcloud[bot] commented on issue #420: HDDS-2854.
Add a config for the write chunk executor queue limit.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on issue #420: HDDS-2854. Add a config for the write chunk executor queue limit.
URL: https://github.com/apache/hadoop-ozone/pull/420#issuecomment-572080935
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=hadoop-ozone&pullRequest=420&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=hadoop-ozone&pullRequest=420&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=hadoop-ozone&pullRequest=420&metric=new_coverage&view=list)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=hadoop-ozone&pullRequest=420&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=hadoop-ozone&pullRequest=420&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on issue #420: HDDS-2854. Add
a config for the write chunk executor queue limit.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on issue #420: HDDS-2854. Add a config for the write chunk executor queue limit.
URL: https://github.com/apache/hadoop-ozone/pull/420#issuecomment-572043639
@bshashikant Thanks for working on this! The changes look good to me. I have no additional comments apart from attila's comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org