You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/04/14 14:30:10 UTC
[GitHub] [hadoop] tomscut opened a new pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
tomscut opened a new pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907
JIRA: [HDFS-15975](https://issues.apache.org/jira/browse/HDFS-15975)
When counting some indicators, we can use LongAdder instead of AtomicLong to improve performance. The long value is not an atomic snapshot in LongAdder, but I think we can tolerate that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tasanuma commented on pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
tasanuma commented on pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#issuecomment-821784876
@tomscut I think this is a good optimization. Would you mind creating another PR for branch-3.3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a change in pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
tomscut commented on a change in pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#discussion_r614496017
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
##########
@@ -607,11 +608,11 @@ public long getCacheCapacity() {
}
public long getNumBlocksFailedToCache() {
- return numBlocksFailedToCache.get();
+ return numBlocksFailedToCache.longValue();
}
public long getNumBlocksFailedToUncache() {
- return numBlocksFailedToUncache.get();
+ return numBlocksFailedToUncache.longValue();
}
public long getNumBlocksCached() {
Review comment:
> Is there any reason not to use `LongAdder` for `numBlocksCached`?
I'm sorry I missed this, I'll fix it right away. Thank you @tamaashu for your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#discussion_r613786951
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
##########
@@ -771,7 +771,7 @@ private void printStatistics(boolean force) {
.append(" Total time for transactions(ms): ")
.append(totalTimeTransactions)
.append(" Number of transactions batched in Syncs: ")
- .append(numTransactionsBatchedInSync.get())
+ .append(numTransactionsBatchedInSync.longValue())
Review comment:
I see, Thanks @tomscut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
tomscut commented on pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#issuecomment-819592967
> Nice one. For metrics and statistics, `LongAdder` provides much better throughput.
Thanks @virajjasani for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a change in pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
tomscut commented on a change in pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#discussion_r613783014
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
##########
@@ -771,7 +771,7 @@ private void printStatistics(boolean force) {
.append(" Total time for transactions(ms): ")
.append(totalTimeTransactions)
.append(" Number of transactions batched in Syncs: ")
- .append(numTransactionsBatchedInSync.get())
+ .append(numTransactionsBatchedInSync.longValue())
Review comment:
At present, it is printed once a minute. I have not changed this frequency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a change in pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
tomscut commented on a change in pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#discussion_r613783259
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
##########
@@ -771,7 +771,7 @@ private void printStatistics(boolean force) {
.append(" Total time for transactions(ms): ")
.append(totalTimeTransactions)
.append(" Number of transactions batched in Syncs: ")
- .append(numTransactionsBatchedInSync.get())
+ .append(numTransactionsBatchedInSync.longValue())
Review comment:
> How frequently this gets printed compared to `longSync()`?
At present, it is printed once a minute. I have not changed this frequency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tasanuma commented on a change in pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
tasanuma commented on a change in pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#discussion_r614222329
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
##########
@@ -607,11 +608,11 @@ public long getCacheCapacity() {
}
public long getNumBlocksFailedToCache() {
- return numBlocksFailedToCache.get();
+ return numBlocksFailedToCache.longValue();
}
public long getNumBlocksFailedToUncache() {
- return numBlocksFailedToUncache.get();
+ return numBlocksFailedToUncache.longValue();
}
public long getNumBlocksCached() {
Review comment:
Is there any reason not to use `LongAdder` for `numBlocksCached`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
tomscut commented on pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#issuecomment-820175087
> Left few nits and one question
Thanks @virajjasani for your good advice, I fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#issuecomment-821037653
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 21s | | trunk passed |
| +1 :green_heart: | compile | 20m 38s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 17m 58s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 5s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 9s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 8m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 30s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 55s | | the patch passed |
| +1 :green_heart: | compile | 20m 3s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 20m 3s | | the patch passed |
| +1 :green_heart: | compile | 18m 2s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 18m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 42s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/3/artifact/out/results-checkstyle-root.txt) | root: The patch generated 6 new + 242 unchanged - 5 fixed = 248 total (was 247) |
| +1 :green_heart: | mvnsite | 4m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 0s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 6s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 8m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 35s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 23s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 35s | | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 235m 13s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 19s | | The patch does not generate ASF License warnings. |
| | | 453m 10s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2907 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 4ad4d1f8986a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c894c8e77e51ff22b7a2a6c645a152558a303095 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/3/testReport/ |
| Max. process+thread count | 3608 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#issuecomment-820500567
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 24s | | trunk passed |
| +1 :green_heart: | compile | 21m 58s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 19m 30s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 53s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 46s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 8m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 41s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 4s | | the patch passed |
| +1 :green_heart: | compile | 22m 37s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 22m 37s | | the patch passed |
| +1 :green_heart: | compile | 19m 52s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 19m 52s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 39s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/2/artifact/out/results-checkstyle-root.txt) | root: The patch generated 5 new + 242 unchanged - 5 fixed = 247 total (was 247) |
| +1 :green_heart: | mvnsite | 4m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 50s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 4s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 9m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 46s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 21s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 37s | | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 244m 34s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 9s | | The patch does not generate ASF License warnings. |
| | | 474m 18s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2907 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 4aef76f54001 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fbb20ec4f413e59c2b24204ab916a6b341426d44 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/2/testReport/ |
| Max. process+thread count | 3238 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#issuecomment-821314734
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 16s | | trunk passed |
| +1 :green_heart: | compile | 20m 46s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 18m 23s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 2s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 7s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 8m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 26s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 51s | | the patch passed |
| +1 :green_heart: | compile | 20m 12s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 20m 12s | | the patch passed |
| +1 :green_heart: | compile | 18m 14s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 18m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 46s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/4/artifact/out/results-checkstyle-root.txt) | root: The patch generated 5 new + 242 unchanged - 5 fixed = 247 total (was 247) |
| +1 :green_heart: | mvnsite | 4m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 58s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 59s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 8m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 39s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 33s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 31s | | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 242m 3s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not generate ASF License warnings. |
| | | 461m 25s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.TestViewDistributedFileSystem |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2907 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 2f8c2108fda1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 699e8185d1ea421e6f153918f01530fbe71ef490 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/4/testReport/ |
| Max. process+thread count | 3089 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
tomscut commented on pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#issuecomment-821837006
> @tomscut I think this is a good optimization. Would you mind creating another PR for branch-3.3?
Thanks @tasanuma for the merge. I will create PR for branch-3.3 later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tasanuma commented on a change in pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
tasanuma commented on a change in pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#discussion_r614693047
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
##########
@@ -607,11 +608,11 @@ public long getCacheCapacity() {
}
public long getNumBlocksFailedToCache() {
- return numBlocksFailedToCache.get();
+ return numBlocksFailedToCache.longValue();
}
public long getNumBlocksFailedToUncache() {
- return numBlocksFailedToUncache.get();
+ return numBlocksFailedToUncache.longValue();
}
public long getNumBlocksCached() {
Review comment:
@tomscut Now we can remove `import java.util.concurrent.atomic.AtomicLong` in FsDatasetCache.java. Others look good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
tomscut commented on pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#issuecomment-819959935
Those failed unit tests were unrelated to the change. And they work fine locally.
Hi @goiri @tasanuma , could you please review the code? Thanks very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tasanuma commented on pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
tasanuma commented on pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#issuecomment-820568403
> Nice one. For metrics and statistics, LongAdder provides much better throughput.
Agreed with @virajjasani. Thanks for working on this, @tomscut.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#discussion_r613775043
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableCounterLong.java
##########
@@ -32,11 +32,11 @@
@InterfaceStability.Evolving
public class MutableCounterLong extends MutableCounter {
- private AtomicLong value = new AtomicLong();
+ private LongAdder value = new LongAdder();
Review comment:
nit: this can be `final`
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
##########
@@ -771,7 +771,7 @@ private void printStatistics(boolean force) {
.append(" Total time for transactions(ms): ")
.append(totalTimeTransactions)
.append(" Number of transactions batched in Syncs: ")
- .append(numTransactionsBatchedInSync.get())
+ .append(numTransactionsBatchedInSync.longValue())
Review comment:
How frequently this gets printed compared to `longSync()`?
##########
File path: hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
##########
@@ -141,21 +141,21 @@ public static OpType fromSymbol(String symbol) {
public static final String NAME = "DFSOpsCountStatistics";
- private final Map<OpType, AtomicLong> opsCount = new EnumMap<>(OpType.class);
+ private final Map<OpType, LongAdder> opsCount = new EnumMap<>(OpType.class);
public DFSOpsCountStatistics() {
super(NAME);
for (OpType opType : OpType.values()) {
- opsCount.put(opType, new AtomicLong(0));
+ opsCount.put(opType, new LongAdder());
}
}
public void incrementOpCounter(OpType op) {
- opsCount.get(op).addAndGet(1);
+ opsCount.get(op).increment();
}
private class LongIterator implements Iterator<LongStatistic> {
- private Iterator<Entry<OpType, AtomicLong>> iterator =
+ private Iterator<Entry<OpType, LongAdder>> iterator =
Review comment:
good to declare this `final`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a change in pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
tomscut commented on a change in pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#discussion_r614688538
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
##########
@@ -607,11 +608,11 @@ public long getCacheCapacity() {
}
public long getNumBlocksFailedToCache() {
- return numBlocksFailedToCache.get();
+ return numBlocksFailedToCache.longValue();
}
public long getNumBlocksFailedToUncache() {
- return numBlocksFailedToUncache.get();
+ return numBlocksFailedToUncache.longValue();
}
public long getNumBlocksCached() {
Review comment:
I fixed that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#issuecomment-819897893
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 59s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 32s | | trunk passed |
| +1 :green_heart: | compile | 28m 48s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 24m 6s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 5m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 8s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 13s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 10m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 42s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 21s | | the patch passed |
| +1 :green_heart: | compile | 20m 20s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 20m 20s | | the patch passed |
| +1 :green_heart: | compile | 18m 4s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 18m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 45s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/1/artifact/out/results-checkstyle-root.txt) | root: The patch generated 5 new + 242 unchanged - 5 fixed = 247 total (was 247) |
| +1 :green_heart: | mvnsite | 4m 10s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 0s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 3s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 8m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 41s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 25s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 36s | | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 246m 51s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 12s | | The patch does not generate ASF License warnings. |
| | | 495m 56s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2907 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux d880014b6357 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9d78a92e7028adeacc66fa9b9aeeca32a87f702a |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/1/testReport/ |
| Max. process+thread count | 3362 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2907/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tasanuma commented on pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
tasanuma commented on pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#issuecomment-821784149
Merged it. Thanks for your contribution, @tomscut. Thanks for your reviews, @virajjasani.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a change in pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
tomscut commented on a change in pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907#discussion_r614696047
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetCache.java
##########
@@ -607,11 +608,11 @@ public long getCacheCapacity() {
}
public long getNumBlocksFailedToCache() {
- return numBlocksFailedToCache.get();
+ return numBlocksFailedToCache.longValue();
}
public long getNumBlocksFailedToUncache() {
- return numBlocksFailedToUncache.get();
+ return numBlocksFailedToUncache.longValue();
}
public long getNumBlocksCached() {
Review comment:
Thank you @tasanuma for your careful review. I will be more careful when I submit the code later. : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tasanuma merged pull request #2907: HDFS-15975. Use LongAdder instead of AtomicLong
Posted by GitBox <gi...@apache.org>.
tasanuma merged pull request #2907:
URL: https://github.com/apache/hadoop/pull/2907
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org