You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/01/16 10:38:24 UTC
[GitHub] [hadoop] MingXiangLi opened a new pull request #3889: HDFS-15180.Split FsDatasetImpl lock to block Pool level
MingXiangLi opened a new pull request #3889:
URL: https://github.com/apache/hadoop/pull/3889
- DataNode performance bottleneck is mainly in datasetLock.DatasetLock not only influence performance and also influence stable.
- We have split datasetLock to block pool level and volume level.And have stable run in 30000+ DataNode server in our cluster over than one year.It has big improvement for our client read write performance and stable.
- This pr is split lock to block pool level.Another pr will split to volume level when this pr is merge.
Design doc
https://drive.google.com/file/d/1eaE8vSEhIli0H3j2eDiPJNYuKAC0MFgu/view?usp=sharing
https://issues.apache.org/jira/browse/HDFS-15382
https://issues.apache.org/jira/browse/HDFS-15382
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3889: HDFS-15180.Split FsDatasetImpl lock to block Pool level
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3889:
URL: https://github.com/apache/hadoop/pull/3889#issuecomment-1013950055
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 17s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 13 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 4s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 52s | | trunk passed |
| +1 :green_heart: | compile | 24m 15s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 45s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 20s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 21s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 37s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 24s | | the patch passed |
| +1 :green_heart: | compile | 23m 40s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javac | 23m 40s | [/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3889/1/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 6 new + 1862 unchanged - 5 fixed = 1868 total (was 1867) |
| +1 :green_heart: | compile | 20m 45s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 20m 45s | [/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3889/1/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 6 new + 1738 unchanged - 5 fixed = 1744 total (was 1743) |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 48s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3889/1/artifact/out/results-checkstyle-root.txt) | root: The patch generated 14 new + 554 unchanged - 5 fixed = 568 total (was 559) |
| +1 :green_heart: | mvnsite | 3m 12s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| -1 :x: | javadoc | 1m 9s | [/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3889/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt) | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04. |
| +1 :green_heart: | javadoc | 3m 21s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | spotbugs | 3m 45s | [/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3889/1/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html) | hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| +1 :green_heart: | shadedclient | 26m 30s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 35s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 371m 23s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | | The patch does not generate ASF License warnings. |
| | | 610m 51s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Should org.apache.hadoop.hdfs.server.common.NoLockManager$NoLock be a _static_ inner class? At NoLockManager.java:inner class? At NoLockManager.java:[lines 29-41] |
| | org.apache.hadoop.hdfs.server.datanode.DataSetLockManager.lockLeakCheck() makes inefficient use of keySet iterator instead of entrySet iterator At DataSetLockManager.java:keySet iterator instead of entrySet iterator At DataSetLockManager.java:[line 258] |
| | Should org.apache.hadoop.hdfs.server.datanode.DataSetLockManager$TrackLog be a _static_ inner class? At DataSetLockManager.java:inner class? At DataSetLockManager.java:[lines 99-128] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3889/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3889 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 92f3788f2864 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 21ab47acc058481402dba6e0116adec068abaa13 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3889/1/testReport/ |
| Max. process+thread count | 1976 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3889/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao closed pull request #3889: HDFS-15180.Split FsDatasetImpl lock to block Pool level
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao closed pull request #3889:
URL: https://github.com/apache/hadoop/pull/3889
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3889: HDFS-15180.Split FsDatasetImpl lock to block Pool level
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #3889:
URL: https://github.com/apache/hadoop/pull/3889#issuecomment-1019431899
Close this PR. Please trace https://issues.apache.org/jira/browse/HDFS-15382 or https://github.com/apache/hadoop/pull/3900 to follow the new progress if interested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] MingXiangLi commented on pull request #3889: HDFS-15180.Split FsDatasetImpl lock to block Pool level
Posted by GitBox <gi...@apache.org>.
MingXiangLi commented on pull request #3889:
URL: https://github.com/apache/hadoop/pull/3889#issuecomment-1015149020
> 感谢@MingXiangLi的作品。现在是一个很大的进步。 快速浏览一下，我认为最好将其拆分为不同的独立JIRA和PR，并且合并更顺畅。 cc @jojochuang @sodonnel @goiri @ayushtkn你介意看看并提供一些其他建议吗？谢谢。
Good advice. Add sub task to this pr. https://github.com/apache/hadoop/pull/3900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3889: HDFS-15180.Split FsDatasetImpl lock to block Pool level
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #3889:
URL: https://github.com/apache/hadoop/pull/3889#issuecomment-1014523383
Thanks @MingXiangLi for your works. It is a great progress now.
Take a quick glance, I think it is better to split it to different separate JIRA and PR, and more smooth to merge.
cc @jojochuang @sodonnel @goiri @ayushtkn Would you mind to take a look and give some other suggestions? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org