You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/07/13 16:02:29 UTC
[GitHub] [hadoop] amahussein opened a new pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
amahussein opened a new pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200
- Backporting HDFS-15160 to branch-3.2
- First: [HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell](https://issues.apache.org/jira/browse/HDFS-15150)
- `git cherry-pick -x d7c136b9ed6d99e1b03f5b89723b3a20df359ba8`
- main conflicts:
- `FsDatasetImpl.java.deepCopyReplica`
- `FoldedTreeSet` is not used in branch-3.2
- testing:
- ```
mvn test -Dtest=TestWriteToReplica,TestReplicaMap,\
TestProvidedImpl,TestFsVolumeList,TestInterDatanodeProtocol
```
- Second: [HDFS-15160](https://issues.apache.org/jira/browse/HDFS-15160)
- `git cherry-pick -x 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718`
- main conflicts:
- `ReplicaMap.java`
- `FsDatasetImpl.java.deepCopyReplica`
- `FsDatasetImpl.java.addVolume` different code. I passed the red and write locks to `tempVolumeMap`
- `DirectoryScanner.scan()` conflict in the entire loop.
- [HDFS-15415. Reduce locking in Datanode DirectoryScanner](https://issues.apache.org/jira/browse/HDFS-15415) Which is already committed to branch-2.10 removes what [HDFS-15160](https://issues.apache.org/jira/browse/HDFS-15160) has added. So, I skipped that.
- Third: [HDFS-15457. TestFsDatasetImpl fails intermittently (#2407)](https://issues.apache.org/jira/browse/HDFS-15457) and [HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig (#2679)](https://issues.apache.org/jira/browse/HDFS-15818)
- `git cherry-pick -x 98097b8f19789605b9697f6a959da57261e0fe19 9434c1eccc255a25ea5e11f6d8c9e1f83996d6b4`
- main conflicts:
- Java imports
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-918879503
Committed to branch-3.2.3.
cherry-pick to branch-3.2.3, build at local and run related unit tests passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-906060946
Thanks @amahussein for your works. I trigger Yetus and it seems works well. Failed unit test `TestDirectoryScanner ` looks not related to changes. AND verify this feature with pseudo cluster, it also seems work well.
Just notice that this PR include different cherry-pick commits together. I am not sure if it is traced gracefully when check in.
@brahmareddybattula do you have any idea here? If not I would like give my +1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-910738631
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 9 new or modified test files. |
|||| _ branch-3.2 Compile Tests _ |
| +0 :ok: | mvndep | 3m 35s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 4s | | branch-3.2 passed |
| +1 :green_heart: | compile | 16m 26s | | branch-3.2 passed |
| +1 :green_heart: | checkstyle | 2m 43s | | branch-3.2 passed |
| +1 :green_heart: | mvnsite | 2m 49s | | branch-3.2 passed |
| +1 :green_heart: | javadoc | 2m 18s | | branch-3.2 passed |
| +1 :green_heart: | spotbugs | 5m 35s | | branch-3.2 passed |
| +1 :green_heart: | shadedclient | 17m 14s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 6s | | the patch passed |
| +1 :green_heart: | compile | 16m 37s | | the patch passed |
| -1 :x: | javac | 16m 37s | [/results-compile-javac-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/8/artifact/out/results-compile-javac-root.txt) | root generated 1 new + 1587 unchanged - 1 fixed = 1588 total (was 1588) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 2m 27s | | root: The patch generated 0 new + 569 unchanged - 1 fixed = 569 total (was 570) |
| +1 :green_heart: | mvnsite | 2m 45s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 20s | | the patch passed |
| +1 :green_heart: | spotbugs | 5m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 4s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 16m 31s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/8/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | unit | 181m 22s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not generate ASF License warnings. |
| | | 323m 19s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.namenode.TestFsck |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3200 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 12f0180fa72b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.2 / 7bfebdabc6807ed558ff9db88a4c73b96dd557fc |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/8/testReport/ |
| Max. process+thread count | 3299 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/8/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] amahussein commented on a change in pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
amahussein commented on a change in pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#discussion_r703761237
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
##########
@@ -201,16 +201,16 @@ public Block getStoredBlock(String bpid, long blkid)
* The deepCopyReplica call doesn't use the datasetock since it will lead the
* potential deadlock with the {@link FsVolumeList#addBlockPool} call.
*/
+ @SuppressWarnings("unchecked")
@Override
public Set<? extends Replica> deepCopyReplica(String bpid)
throws IOException {
- Set<? extends Replica> replicas = null;
+ Set<? extends Replica> replicas;
Review comment:
Thanks @brahmareddybattula
I think we can apply some changes to trunk given that the type casting is unchecked and causes some Javac warnings.
Perhaps we can file another refactoring jira to address these issues on trunk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-909552788
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 8m 8s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 9 new or modified test files. |
|||| _ branch-3.2 Compile Tests _ |
| +0 :ok: | mvndep | 3m 50s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 44s | | branch-3.2 passed |
| +1 :green_heart: | compile | 16m 34s | | branch-3.2 passed |
| +1 :green_heart: | checkstyle | 2m 28s | | branch-3.2 passed |
| +1 :green_heart: | mvnsite | 2m 40s | | branch-3.2 passed |
| +1 :green_heart: | javadoc | 2m 16s | | branch-3.2 passed |
| +1 :green_heart: | spotbugs | 4m 59s | | branch-3.2 passed |
| +1 :green_heart: | shadedclient | 14m 51s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 57s | | the patch passed |
| +1 :green_heart: | compile | 14m 24s | | the patch passed |
| -1 :x: | javac | 14m 24s | [/results-compile-javac-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/6/artifact/out/results-compile-javac-root.txt) | root generated 1 new + 1587 unchanged - 1 fixed = 1588 total (was 1588) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 2m 33s | | root: The patch generated 0 new + 569 unchanged - 1 fixed = 569 total (was 570) |
| +1 :green_heart: | mvnsite | 2m 44s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 19s | | the patch passed |
| +1 :green_heart: | spotbugs | 5m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 20s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 15m 54s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 174m 57s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | asflicense | 1m 3s | | The patch does not generate ASF License warnings. |
| | | 316m 39s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3200 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 8ac03118cb9f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.2 / 29a98320b76d60afb481f729456fbb9db1c37287 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/6/testReport/ |
| Max. process+thread count | 3189 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/6/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao merged pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao merged pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] amahussein edited a comment on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
amahussein edited a comment on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-909634740
After rebasing, it looks like there are javac checks added to the build that failed it.
Although this is a part of the original code, I decided to address it so that we can pass the build and move forward with the PR.
I also tested the TestDirectoryScanner and It passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on a change in pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#discussion_r702617962
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
##########
@@ -201,16 +201,16 @@ public Block getStoredBlock(String bpid, long blkid)
* The deepCopyReplica call doesn't use the datasetock since it will lead the
* potential deadlock with the {@link FsVolumeList#addBlockPool} call.
*/
+ @SuppressWarnings("unchecked")
@Override
public Set<? extends Replica> deepCopyReplica(String bpid)
throws IOException {
- Set<? extends Replica> replicas = null;
+ Set<? extends Replica> replicas;
Review comment:
can this change relate to trunk also..? Even this can be emptyset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on a change in pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#discussion_r707624080
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
##########
@@ -201,16 +201,16 @@ public Block getStoredBlock(String bpid, long blkid)
* The deepCopyReplica call doesn't use the datasetock since it will lead the
* potential deadlock with the {@link FsVolumeList#addBlockPool} call.
*/
+ @SuppressWarnings("unchecked")
@Override
public Set<? extends Replica> deepCopyReplica(String bpid)
throws IOException {
- Set<? extends Replica> replicas = null;
+ Set<? extends Replica> replicas;
Review comment:
can please raise jira to track this..?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-909105949
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 11m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 9 new or modified test files. |
|||| _ branch-3.2 Compile Tests _ |
| +0 :ok: | mvndep | 4m 22s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 31s | | branch-3.2 passed |
| +1 :green_heart: | compile | 15m 31s | | branch-3.2 passed |
| +1 :green_heart: | checkstyle | 2m 39s | | branch-3.2 passed |
| +1 :green_heart: | mvnsite | 2m 30s | | branch-3.2 passed |
| +1 :green_heart: | javadoc | 2m 6s | | branch-3.2 passed |
| +1 :green_heart: | spotbugs | 5m 3s | | branch-3.2 passed |
| +1 :green_heart: | shadedclient | 16m 34s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 55s | | the patch passed |
| +1 :green_heart: | compile | 14m 53s | | the patch passed |
| -1 :x: | javac | 14m 53s | [/results-compile-javac-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/5/artifact/out/results-compile-javac-root.txt) | root generated 1 new + 1576 unchanged - 1 fixed = 1577 total (was 1577) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 2m 42s | | root: The patch generated 0 new + 568 unchanged - 1 fixed = 568 total (was 569) |
| +1 :green_heart: | mvnsite | 2m 29s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 2s | | the patch passed |
| +1 :green_heart: | spotbugs | 5m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 42s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 15m 45s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 208m 36s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | asflicense | 1m 39s | | The patch does not generate ASF License warnings. |
| | | 357m 8s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestReconstructStripedFileWithValidator |
| | hadoop.hdfs.TestDecommissionWithStriped |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3200 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 573c79676bd4 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.2 / 9516ddf5488d512bd55f539363b0befe575b893a |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/5/testReport/ |
| Max. process+thread count | 1948 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/5/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-895926938
Thanks @amahussein for your works.
cc @sodonnel would you mind to give another checks? Consider branch-3.2.3 is prepared. I think it is good choice to involve this improvement there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-918521589
@Hexiaoqiao , There are two issues I want to bring with this merge. Can you please check once.?
1) Looks commit's are squashed and merged, this should n't good way as there are 5 differenet commits with 5 jira's..In furute,if we want to revert one commit(among these 5), then all commits we need to revert.
2) It's not merged to branch-3.2.3, But jira marked to hadoop-3.2.3 version..?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-917394169
Committed to branch-3.2. Thanks @amahussein for your works and thanks @brahmareddybattula for your reviews!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] amahussein commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
amahussein commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-916113850
Hi @Hexiaoqiao, Is this PR ready for merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] amahussein commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
amahussein commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-909234801
> Thanks @amahussein for your works. I trigger Yetus and it seems works well. Failed unit test `TestDirectoryScanner ` looks not related to changes. AND verify this feature with pseudo cluster, it also seems work well.
> Just notice that this PR include different cherry-pick commits together. I am not sure if it is traced gracefully when check in.
> @brahmareddybattula do you have any idea here? If not I would like give my +1.
Thanks @Hexiaoqiao for the review and the feedback. I appreciate it.
I intentionally cherry-picked the commits related to the feature so that the PR would be complete. Otherwise, if we did separate PRs for each commit, branch-3.2 would be broken.
I believe it should be okay to have multiple cherry-picks as long as the final merge would not squash them into one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-909239985
Thanks @amahussein. It is OK for me to squash and check in them together. Just find there are many checks failed reported by Yetus here. Do you mind give another check?
Considering branch-3.2 will release recently, I think it is time to push this PR into branch-3.2 before release happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-918845356
revert squashed commit and cherry-pick single commit for branch-3.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-909239985
Thanks @amahussein. It is OK for me to squash and check in them together. Just find there are many checks failed reported by Yetus here. Do you mind give another check?
Considering branch-3.2 will release recently, I think it is time to push this PR into branch-3.2 before release happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] amahussein edited a comment on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
amahussein edited a comment on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-909634740
After rebasing, it looks like there are javac checks added to the build that failed it.
Although this is a part of the original code, I decided to address it so that we can pass the build and move forward with the PR.
I also tested the TestDirectoryScanner and It passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] amahussein commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
amahussein commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-909634740
After rebasing, it looks like there are javac checks added to the build that failed it.
Although this is a part of the original code, I decided to address it so that we can pass the build and move forward with the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-894761816
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 18s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 9 new or modified test files. |
|||| _ branch-3.2 Compile Tests _ |
| +0 :ok: | mvndep | 3m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 29s | | branch-3.2 passed |
| +1 :green_heart: | compile | 15m 44s | | branch-3.2 passed |
| +1 :green_heart: | checkstyle | 2m 42s | | branch-3.2 passed |
| +1 :green_heart: | mvnsite | 2m 35s | | branch-3.2 passed |
| +1 :green_heart: | javadoc | 2m 14s | | branch-3.2 passed |
| +1 :green_heart: | spotbugs | 5m 10s | | branch-3.2 passed |
| +1 :green_heart: | shadedclient | 16m 45s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 58s | | the patch passed |
| +1 :green_heart: | compile | 15m 5s | | the patch passed |
| -1 :x: | javac | 15m 5s | [/results-compile-javac-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/2/artifact/out/results-compile-javac-root.txt) | root generated 1 new + 1587 unchanged - 1 fixed = 1588 total (was 1588) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 2m 39s | | root: The patch generated 0 new + 569 unchanged - 1 fixed = 569 total (was 570) |
| +1 :green_heart: | mvnsite | 2m 33s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 4s | | the patch passed |
| +1 :green_heart: | spotbugs | 5m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 49s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 15m 48s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 201m 29s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not generate ASF License warnings. |
| | | 338m 29s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3200 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 7941b9e37b6c 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.2 / 9516ddf5488d512bd55f539363b0befe575b893a |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/2/testReport/ |
| Max. process+thread count | 2331 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/2/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] amahussein commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
amahussein commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-909234801
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-906071805
@jojochuang Would you mind to have another check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-909105949
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 11m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 9 new or modified test files. |
|||| _ branch-3.2 Compile Tests _ |
| +0 :ok: | mvndep | 4m 22s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 31s | | branch-3.2 passed |
| +1 :green_heart: | compile | 15m 31s | | branch-3.2 passed |
| +1 :green_heart: | checkstyle | 2m 39s | | branch-3.2 passed |
| +1 :green_heart: | mvnsite | 2m 30s | | branch-3.2 passed |
| +1 :green_heart: | javadoc | 2m 6s | | branch-3.2 passed |
| +1 :green_heart: | spotbugs | 5m 3s | | branch-3.2 passed |
| +1 :green_heart: | shadedclient | 16m 34s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 55s | | the patch passed |
| +1 :green_heart: | compile | 14m 53s | | the patch passed |
| -1 :x: | javac | 14m 53s | [/results-compile-javac-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/5/artifact/out/results-compile-javac-root.txt) | root generated 1 new + 1576 unchanged - 1 fixed = 1577 total (was 1577) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 2m 42s | | root: The patch generated 0 new + 568 unchanged - 1 fixed = 568 total (was 569) |
| +1 :green_heart: | mvnsite | 2m 29s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 2s | | the patch passed |
| +1 :green_heart: | spotbugs | 5m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 42s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 15m 45s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 208m 36s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | asflicense | 1m 39s | | The patch does not generate ASF License warnings. |
| | | 357m 8s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestReconstructStripedFileWithValidator |
| | hadoop.hdfs.TestDecommissionWithStriped |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3200 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 573c79676bd4 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.2 / 9516ddf5488d512bd55f539363b0befe575b893a |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/5/testReport/ |
| Max. process+thread count | 1948 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/5/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-909806663
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 9 new or modified test files. |
|||| _ branch-3.2 Compile Tests _ |
| +0 :ok: | mvndep | 3m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 58s | | branch-3.2 passed |
| +1 :green_heart: | compile | 15m 2s | | branch-3.2 passed |
| +1 :green_heart: | checkstyle | 2m 36s | | branch-3.2 passed |
| +1 :green_heart: | mvnsite | 2m 42s | | branch-3.2 passed |
| +1 :green_heart: | javadoc | 2m 22s | | branch-3.2 passed |
| +1 :green_heart: | spotbugs | 5m 2s | | branch-3.2 passed |
| +1 :green_heart: | shadedclient | 15m 20s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 54s | | the patch passed |
| +1 :green_heart: | compile | 14m 21s | | the patch passed |
| -1 :x: | javac | 14m 21s | [/results-compile-javac-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/7/artifact/out/results-compile-javac-root.txt) | root generated 1 new + 1587 unchanged - 1 fixed = 1588 total (was 1588) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 2m 32s | | root: The patch generated 0 new + 569 unchanged - 1 fixed = 569 total (was 570) |
| +1 :green_heart: | mvnsite | 2m 41s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 17s | | the patch passed |
| +1 :green_heart: | spotbugs | 5m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 30s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 15m 57s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 175m 41s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | asflicense | 1m 4s | | The patch does not generate ASF License warnings. |
| | | 306m 53s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.TestRedudantBlocks |
| | hadoop.hdfs.server.namenode.TestFsck |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3200 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux aa126cf10a5a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.2 / 610e6e46ec87d615c19a51cf979bf43721706e07 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/7/testReport/ |
| Max. process+thread count | 3439 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/7/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-913357847
@amahussein thanks for working on this.. At first glance,it's better to have seperate PR for Each jira backport.It can be easy to review and maintain in future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-905303834
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 23s | | HDFS-15160 does not apply to branch-3.3. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| JIRA Issue | HDFS-15160 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13022920/HDFS-15160-branch-3.3-001.patch |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/3/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] amahussein commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
amahussein commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-909242831
> Thanks @amahussein. It is OK for me to squash and check in them together. Just find there are many checks failed reported by Yetus here. Do you mind give another check?
> Considering branch-3.2 will release recently, I think it is time to push this PR into branch-3.2 before release happen.
I will rebase the code changes and push a fresh version.
P.S: the final merge "_should not_" squash the commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] amahussein commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
amahussein commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-909234801
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-918793612
@brahmareddybattula Good point. Will revert and check in per-commit. Thanks. For version, I forgot we have create branch-3.2.3 already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-905580662
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 11m 10s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 9 new or modified test files. |
|||| _ branch-3.2 Compile Tests _ |
| +0 :ok: | mvndep | 3m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 29s | | branch-3.2 passed |
| +1 :green_heart: | compile | 15m 28s | | branch-3.2 passed |
| +1 :green_heart: | checkstyle | 2m 42s | | branch-3.2 passed |
| +1 :green_heart: | mvnsite | 2m 32s | | branch-3.2 passed |
| +1 :green_heart: | javadoc | 2m 8s | | branch-3.2 passed |
| +1 :green_heart: | spotbugs | 5m 8s | | branch-3.2 passed |
| +1 :green_heart: | shadedclient | 16m 42s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 56s | | the patch passed |
| +1 :green_heart: | compile | 14m 55s | | the patch passed |
| -1 :x: | javac | 14m 55s | [/results-compile-javac-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/4/artifact/out/results-compile-javac-root.txt) | root generated 1 new + 1587 unchanged - 1 fixed = 1588 total (was 1588) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 2m 37s | | root: The patch generated 0 new + 568 unchanged - 1 fixed = 568 total (was 569) |
| +1 :green_heart: | mvnsite | 2m 28s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 1s | | the patch passed |
| +1 :green_heart: | spotbugs | 5m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 40s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 15m 41s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 198m 18s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not generate ASF License warnings. |
| | | 343m 54s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3200 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 25db101e175e 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.2 / 9516ddf5488d512bd55f539363b0befe575b893a |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/4/testReport/ |
| Max. process+thread count | 1991 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/4/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-909239985
Thanks @amahussein. It is OK for me to squash and check in them together. Just find there are many checks failed reported by Yetus here. Do you mind give another check?
Considering branch-3.2 will release recently, I think it is time to push this PR into branch-3.2 before release happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-916596838
Will commit to branch-3.2 for a short while if no more comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-909105949
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org