You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "OneSizeFitsQuorum (via GitHub)" <gi...@apache.org> on 2026/10/23 08:33:24 UTC
[PR] HADOOP-17611. Preserve DistCp file times after chunk concatenation. [hadoop]
OneSizeFitsQuorum opened a new pull request, #8858:
URL: https://github.com/apache/hadoop/pull/8858
### Description of PR
[HADOOP-17611](https://issues.apache.org/jira/browse/HADOOP-17611)
With `-blocksperchunk` and `-pt`, CopyMapper preserves timestamps on the chunks, but the subsequent concat in CopyCommitter changes the final file's modification time. Restore the source modification and access times after concatenation, rename, and checksum validation, only when `FileAttribute.TIMES` is requested.
Restore only timestamps: file entries in the copy listing do not contain extended ACLs, so replaying all preserved attributes here can erase ACLs already preserved by the mapper. Directory preservation continues to use the existing commit phase and copy scope.
Add 14 parameterized cases to AbstractContractDistCpTest covering:
- TIMES enabled and disabled with ordinary, direct, and atomic copies.
- Actual multi-block files, nested directories, exact-block files, small files, and empty files, with complete content verification.
- Parents outside the copy scope, `-updateRoot`, overwritten files, and fully skipped updates.
- ACLs, XAttrs, permissions, ownership, replication, and block size alongside optional timestamp preservation.
The tests use two concurrent local map tasks and assert that large source files really span multiple blocks. Capability checks skip unsupported filesystems.
### How was this patch tested?
JDK 17; local Maven reactor run:
```sh
mvn -B -pl hadoop-tools/hadoop-distcp -am \
-Dtest=TestHDFSContractDistCp,TestLocalContractDistCp,TestDistCpSystem,TestCopyMapper,TestCopyCommitter,TestDistCpWithAcls,TestDistCpWithXAttrs \
-Dmaven.test.failure.ignore=false \
-Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false \
-Dcheckstyle.skip -Drat.skip=true test
```
99 tests: 83 passed, 16 skipped, zero failures or errors. All 14 new HDFS cases passed. The skips are the 14 unsupported local-filesystem cases and two existing disabled HDFS tests. The run used a temporary Maven repository and mirror settings.
Separately ran the repository's Checkstyle configuration on both changed files: no findings on added lines. `git diff --check` passed.
Also ran temporary negative controls against the same contract tests: the original implementation fails the three TIMES-enabled copy-mode cases; unconditional mtime/atime preservation fails the TIMES-disabled cases; preserving an out-of-scope parent fails scope checks; and replaying all attributes fails the ACL checks.
Scope note: ordinary chunked single-file copying to a nonexistent destination filename has a separate staging-path failure (`Target is dir, Source is file`), reproduced with both the original and patched implementations. Scope tests cover ordinary file-to-existing-directory copying and direct file-to-new-filename copying. This PR does not change staging paths or optimize mapper-side timestamp calls.
### For code changes:
- [x] The title starts with the corresponding JIRA issue ID.
- Object storage integration tests: not run; validation used HDFS and local filesystem contracts.
- New dependencies: none.
- LICENSE/NOTICE changes: not applicable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-17611. Preserve DistCp file times after chunk concatenation. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8858:
URL: https://github.com/apache/hadoop/pull/8858#issuecomment-6408396404
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 5s | | trunk passed |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 0m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| -1 :x: | spotbugs | 1m 21s | [/branch-spotbugs-hadoop-tools_hadoop-distcp-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/3/artifact/out/branch-spotbugs-hadoop-tools_hadoop-distcp-warnings.html) | hadoop-tools/hadoop-distcp in trunk has 21 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 29m 34s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 23s | [/patch-mvninstall-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/3/artifact/out/patch-mvninstall-hadoop-tools_hadoop-distcp.txt) | hadoop-distcp in the patch failed. |
| -1 :x: | compile | 0m 24s | [/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-21.0.12.1+1-1-24.04.4-Ubuntu.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/3/artifact/out/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-21.0.12.1+1-1-24.04.4-Ubuntu.txt) | hadoop-distcp in the patch failed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu. |
| -1 :x: | javac | 0m 24s | [/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-21.0.12.1+1-1-24.04.4-Ubuntu.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/3/artifact/out/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-21.0.12.1+1-1-24.04.4-Ubuntu.txt) | hadoop-distcp in the patch failed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu. |
| -1 :x: | compile | 0m 25s | [/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-17.0.20.1+1-1-24.04-Ubuntu.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/3/artifact/out/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-17.0.20.1+1-1-24.04-Ubuntu.txt) | hadoop-distcp in the patch failed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu. |
| -1 :x: | javac | 0m 25s | [/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-17.0.20.1+1-1-24.04-Ubuntu.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/3/artifact/out/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-17.0.20.1+1-1-24.04-Ubuntu.txt) | hadoop-distcp in the patch failed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | the patch passed |
| -1 :x: | mvnsite | 0m 26s | [/patch-mvnsite-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/3/artifact/out/patch-mvnsite-hadoop-tools_hadoop-distcp.txt) | hadoop-distcp in the patch failed. |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| -1 :x: | spotbugs | 0m 25s | [/patch-spotbugs-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/3/artifact/out/patch-spotbugs-hadoop-tools_hadoop-distcp.txt) | hadoop-distcp in the patch failed. |
| +1 :green_heart: | shadedclient | 31m 10s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 29s | [/patch-unit-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/3/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt) | hadoop-distcp in the patch failed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 116m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8858 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 463b39763d19 5.15.0-198-generic #208-Ubuntu SMP Fri Sep 4 10:22:38 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 49ba2dc5c0f10b707f9245dee3a794183b762df1 |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/3/testReport/ |
| Max. process+thread count | 681 (vs. ulimit of 10000) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/3/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-17611. Preserve DistCp file times after chunk concatenation. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8858:
URL: https://github.com/apache/hadoop/pull/8858#issuecomment-6412980002
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 43s | | trunk passed |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 0m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| -1 :x: | spotbugs | 1m 15s | [/branch-spotbugs-hadoop-tools_hadoop-distcp-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/4/artifact/out/branch-spotbugs-hadoop-tools_hadoop-distcp-warnings.html) | hadoop-tools/hadoop-distcp in trunk has 21 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 35m 27s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 36m 2s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 0m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 15s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 38m 39s | | hadoop-distcp in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 171m 37s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/4/artifact/out/Dockerfile |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e755e85433cc 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d760975ecbfd56174d13e240ef2beebe70288fc2 |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/4/testReport/ |
| Max. process+thread count | 581 (vs. ulimit of 10000) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/4/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-17611. Preserve DistCp file times after chunk concatenation. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8858:
URL: https://github.com/apache/hadoop/pull/8858#issuecomment-6363824104
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 9s | | trunk passed |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 0m 51s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 0m 48s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| -1 :x: | spotbugs | 1m 14s | [/branch-spotbugs-hadoop-tools_hadoop-distcp-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/1/artifact/out/branch-spotbugs-hadoop-tools_hadoop-distcp-warnings.html) | hadoop-tools/hadoop-distcp in trunk has 21 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 35m 19s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 0m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 16s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 40m 29s | | hadoop-distcp in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not generate ASF License warnings. |
| | | 172m 47s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8858 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 87ea78b1abb3 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0dd807237e5e1fdd67cdeb61f5fc98b1d68836bc |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/1/testReport/ |
| Max. process+thread count | 585 (vs. ulimit of 10000) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/1/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-17611. Preserve DistCp file times after chunk concatenation. [hadoop]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8858:
URL: https://github.com/apache/hadoop/pull/8858#discussion_r4358034303
##########
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java:
##########
@@ -654,6 +654,13 @@ private void concatFileChunks(Configuration conf, Path sourceFile,
DistCpUtils.compareFileLengthsAndChecksums(srcFileStatus.getLen(),
srcfs, sourceFile, null, dstfs,
targetFile, skipCrc, srcFileStatus.getLen());
+ // Concat changes the modification time preserved on the first chunk.
+ // Restore times after validation, which may also update the access time.
+ if (DistCpUtils.unpackAttributes(conf.get(CONF_LABEL_PRESERVE_STATUS))
+ .contains(FileAttribute.TIMES)) {
+ dstfs.setTimes(targetFile, srcFileStatus.getModificationTime(),
+ srcFileStatus.getAccessTime());
Review Comment:
`setTimes(Path, mtime, atime)` typically uses `-1` to mean 'do not change'. If `srcFileStatus.getAccessTime()` is `0` (common on filesystems that don’t expose atime), this will actively set the destination atime to epoch rather than preserving/ignoring it. Consider passing `-1` when the source atime is non-positive (e.g., compute `long atime = srcFileStatus.getAccessTime() > 0 ? srcFileStatus.getAccessTime() : -1`) so enabling `TIMES` doesn’t inadvertently clobber atime on FSes without atime support.
##########
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##########
@@ -252,6 +271,243 @@ protected Path getRemoteDir() {
return remoteDir;
}
+ @ParameterizedTest(name = "preserveTimes={0}, direct={1}, atomic={2}")
+ @CsvSource({
+ "true, false, false",
+ "false, false, false",
+ "true, true, false",
+ "false, true, false",
+ "true, false, true",
+ "false, false, true"
+ })
+ public void testChunkCopyPreserveTimes(boolean preserveTimes,
+ boolean directWrite, boolean atomicCommit) throws Exception {
+ requireChunkCopySupport();
+ Path source = new Path(remoteDir, "source");
+ Path target = new Path(remoteDir, "target");
+ Map<String, FileStatus> sourceStatuses = createChunkCopyTree(source);
+
+ DistCpOptions.Builder builder = chunkCopyOptions(source, target,
+ preserveTimes).withDirectWrite(directWrite || shouldUseDirectWrite())
+ .withAtomicCommit(atomicCommit);
+ runDistCp(builder.build(), true);
+
+ // The source directory itself is in the copy listing, so its times and
+ // those of nested directories must be restored after all concatenations.
+ verifyChunkCopyTree(target, sourceStatuses, preserveTimes, true);
+ }
+
+ @ParameterizedTest(name = "preserveTimes={0}, renameTarget={1}")
+ @CsvSource({"true, false", "false, false", "true, true", "false, true"})
+ public void testChunkCopyOutsideParentScope(boolean preserveTimes,
+ boolean renameTarget)
+ throws Exception {
+ requireChunkCopySupport();
+ Path sourceParent = new Path(remoteDir, "source-parent");
+ Path source = new Path(sourceParent, "file");
+ Path targetParent = new Path(remoteDir, "target-parent");
+ Path target = new Path(targetParent,
+ renameTarget ? "renamed-file" : source.getName());
+ createChunkCopyFile(source, TEST_FILE_LEN * 3 + 1);
+ FileStatus sourceStatus = setChunkCopyTimes(source, 0);
+ FileStatus sourceParentStatus = setChunkCopyTimes(sourceParent, 1);
+ mkdirs(remoteFS, targetParent);
+ FileStatus targetParentStatus = setChunkCopyTimes(targetParent, 2);
+
+ DistCpOptions.Builder builder = chunkCopyOptions(source,
+ renameTarget ? target : targetParent, preserveTimes);
+ if (renameTarget) {
+ builder.withDirectWrite(true);
+ }
+ runDistCp(builder.build(), true);
+
+ assertChunkCopyTimes(sourceStatus, remoteFS.getFileStatus(target),
+ preserveTimes);
+ // Creating/concatenating children may change the destination parent's
+ // mtime naturally. It must not inherit the source parent's timestamps.
+ FileStatus parentAfterCopy = remoteFS.getFileStatus(targetParent);
+ Assertions.assertThat(parentAfterCopy.getModificationTime())
+ .describedAs("mtime of parent outside copy scope")
+ .isNotEqualTo(sourceParentStatus.getModificationTime());
+ assertEquals(targetParentStatus.getAccessTime(),
+ parentAfterCopy.getAccessTime(), "atime of parent outside copy scope");
+ verifyFileContents(remoteFS, target,
+ dataset((int) sourceStatus.getLen(), 'a', 26));
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ public void testChunkCopyUpdateRootTimes(boolean updateRoot)
+ throws Exception {
+ requireChunkCopySupport();
+ Path source = new Path(remoteDir, "source");
+ Path target = new Path(remoteDir, "target");
+ Map<String, FileStatus> sourceStatuses = createChunkCopyTree(source);
+ // Force an existing destination file to be replaced by a chunked copy.
+ createFile(remoteFS, new Path(target, "large"), true, new byte[1]);
+ FileStatus targetRootStatus = setChunkCopyTimes(target, 20);
+ DistCpOptions options = chunkCopyOptions(source, target, true)
+ .withSyncFolder(true).withUpdateRoot(updateRoot).build();
+
+ runDistCp(options, true);
+ verifyChunkCopyTree(target, sourceStatuses, true, updateRoot);
+ if (!updateRoot) {
+ assertEquals(targetRootStatus.getAccessTime(),
+ remoteFS.getFileStatus(target).getAccessTime(),
+ "-update must not preserve root times without -updateRoot");
+ }
+
+ // A fully skipped update must still preserve file times without
+ // attempting to concatenate nonexistent chunk files.
+ setChunkCopyTimes(new Path(target, "large"), 30);
+ Job skipped = runDistCp(options, true);
+ assertCounterInRange(skipped, CopyMapper.Counter.COPY, 0, 0);
+ assertCounterInRange(skipped, CopyMapper.Counter.SKIP, 1, -1);
+ assertCounterInRange(skipped, CopyMapper.Counter.BYTESCOPIED, 0, 0);
+ verifyChunkCopyTree(target, sourceStatuses, true, updateRoot);
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ public void testChunkCopyPreserveOtherAttributes(boolean preserveTimes)
+ throws Exception {
+ requireChunkCopySupport();
+ skipIfUnsupported(SUPPORTS_UNIX_PERMISSIONS);
+ if (!remoteFS.hasPathCapability(remoteDir, FS_ACLS)
+ || !remoteFS.hasPathCapability(remoteDir, FS_XATTRS)) {
+ skip("ACL and XAttr support required");
+ }
+ Path source = new Path(remoteDir, "source");
+ Path target = new Path(remoteDir, "target");
+ Map<String, FileStatus> sourceStatuses = createChunkCopyTree(source);
+ Path sourceFile = new Path(source, "large");
+ remoteFS.setPermission(sourceFile, new FsPermission((short) 0640));
+ remoteFS.modifyAclEntries(sourceFile, Collections.singletonList(
+ new AclEntry.Builder().setScope(AclEntryScope.ACCESS)
+ .setType(AclEntryType.USER).setName("distcp-reader")
+ .setPermission(FsAction.READ).build()));
+ String xattrName = "user.distcp";
+ byte[] xattrValue = {1, 2, 3};
+ remoteFS.setXAttr(sourceFile, xattrName, xattrValue);
+ sourceStatuses.put("large", setChunkCopyTimes(sourceFile, 0));
+ sourceStatuses.put("", setChunkCopyTimes(source, 11));
+
+ runDistCp(chunkCopyOptions(source, target, preserveTimes)
+ .preserve("rbugpcax").build(), true);
+
+ // File listing entries omit extended ACLs: restoring all attributes from
+ // them after concat would erase ACLs already preserved by the mapper.
+ Path targetFile = new Path(target, "large");
+ assertEquals(remoteFS.getAclStatus(sourceFile).getEntries(),
+ remoteFS.getAclStatus(targetFile).getEntries(), "file ACL entries");
+ assertArrayEquals(xattrValue, remoteFS.getXAttr(targetFile, xattrName));
+ FileStatus expected = sourceStatuses.get("large");
+ FileStatus actual = remoteFS.getFileStatus(targetFile);
+ assertEquals(expected.getPermission(), actual.getPermission());
+ assertEquals(expected.getOwner(), actual.getOwner());
+ assertEquals(expected.getGroup(), actual.getGroup());
+ assertEquals(expected.getReplication(), actual.getReplication());
+ assertEquals(expected.getBlockSize(), actual.getBlockSize());
+ verifyChunkCopyTree(target, sourceStatuses, preserveTimes, true);
+ }
+
+ private void requireChunkCopySupport() throws IOException {
+ skipIfUnsupported(SUPPORTS_CONCAT);
+ skipIfUnsupported(SUPPORTS_SETTIMES);
+ skipIfUnsupported(SUPPORTS_BLOCK_LOCALITY);
+ conf.setInt("mapreduce.local.map.tasks.maximum", 2);
+ }
+
+ private DistCpOptions.Builder chunkCopyOptions(Path source, Path target,
+ boolean preserveTimes) {
+ DistCpOptions.Builder builder = new DistCpOptions.Builder(
+ Collections.singletonList(source), target)
+ .withBlocksPerChunk(1).withDirectWrite(shouldUseDirectWrite());
+ if (preserveTimes) {
+ builder.preserve(FileAttribute.TIMES);
+ }
+ return builder;
+ }
+
+ private Map<String, FileStatus> createChunkCopyTree(Path source)
+ throws IOException {
+ Map<String, FileStatus> statuses = new LinkedHashMap<>();
+ String[] names = {"large", "large2", "nested/large", "block", "small",
+ "empty"};
+ int[] lengths = {TEST_FILE_LEN * 3 + 1, TEST_FILE_LEN * 4,
+ TEST_FILE_LEN * 2, TEST_FILE_LEN, TEST_FILE_LEN / 2, 0};
+ for (int i = 0; i < names.length; i++) {
+ Path file = new Path(source, names[i]);
+ createChunkCopyFile(file, lengths[i]);
+ statuses.put(names[i], setChunkCopyTimes(file, i));
+ }
+ // Set directory times only after all children have been created.
+ statuses.put("nested", setChunkCopyTimes(new Path(source, "nested"), 10));
+ statuses.put("", setChunkCopyTimes(source, 11));
+ return statuses;
+ }
+
+ private void createChunkCopyFile(Path path, int length) throws IOException {
+ writeDataset(remoteFS, path, dataset(length, 'a', 26), length,
+ TEST_FILE_LEN, true);
+ if (length > TEST_FILE_LEN) {
+ FileStatus status = remoteFS.getFileStatus(path);
+ assertTrue(status.getLen() > status.getBlockSize(),
+ "File must exceed its block size to exercise chunked copying");
+ assertTrue(remoteFS.getFileBlockLocations(status, 0, length).length > 1,
+ "File must have multiple blocks to exercise concatenation");
+ }
+ }
+
+ private FileStatus setChunkCopyTimes(Path path, int offset)
+ throws IOException {
+ long mtime = CHUNK_COPY_TIME + offset * MODIFICATION_TIME_OFFSET;
+ remoteFS.setTimes(path, mtime,
+ CHUNK_COPY_ACCESS_TIME + offset * MODIFICATION_TIME_OFFSET);
+ FileStatus status = remoteFS.getFileStatus(path);
+ assertEquals(mtime, status.getModificationTime(), "source fixture mtime");
Review Comment:
The fixture validates only mtime, but later tests assert both mtime and atime preservation. If `setTimes()` silently ignores atime on a given FS (or normalizes it), failures later can be harder to diagnose. Consider also asserting the expected atime when available (e.g., if `status.getAccessTime() > 0`, assert it matches the requested value), or documenting that atime may not round-trip even when `SUPPORTS_SETTIMES` is true.
##########
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java:
##########
@@ -654,6 +654,13 @@ private void concatFileChunks(Configuration conf, Path sourceFile,
DistCpUtils.compareFileLengthsAndChecksums(srcFileStatus.getLen(),
srcfs, sourceFile, null, dstfs,
targetFile, skipCrc, srcFileStatus.getLen());
+ // Concat changes the modification time preserved on the first chunk.
+ // Restore times after validation, which may also update the access time.
+ if (DistCpUtils.unpackAttributes(conf.get(CONF_LABEL_PRESERVE_STATUS))
+ .contains(FileAttribute.TIMES)) {
Review Comment:
`DistCpUtils.unpackAttributes(...)` appears to parse/unpack the preserved-attributes set on every `concatFileChunks()` call. Since concat may run once per copied file, this can add avoidable overhead at scale. Consider unpacking once (e.g., in the committer initialization / caller) and passing a boolean `preserveTimes`, or caching the unpacked set for reuse.
##########
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##########
@@ -76,6 +89,12 @@ public abstract class AbstractContractDistCpTest
/** Using offset to change modification time in tests. */
private static final long MODIFICATION_TIME_OFFSET = 10000;
+ private static final long CHUNK_COPY_TIME = 1600000000000L;
+
+ // Keep reads and checksum verification from advancing the source atime.
+ private static final long CHUNK_COPY_ACCESS_TIME =
+ (System.currentTimeMillis() / 1000 + 86400) * 1000;
Review Comment:
This makes test behavior time-dependent by setting atime relative to the wall clock (and into the future). On some filesystems/implementations, read-atime updates may set atime to 'now' even if that is earlier than the current atime, which could create non-deterministic failures depending on FS semantics. A more deterministic approach is to use a fixed constant (like `CHUNK_COPY_TIME +/- offset`) and, if needed, re-stat the source after the copy to derive the expected atime (or conditionally assert atime only when it remains stable on the contract FS).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-17611. Preserve DistCp file times after chunk concatenation. [hadoop]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8858:
URL: https://github.com/apache/hadoop/pull/8858#discussion_r4373477854
##########
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##########
@@ -252,6 +271,246 @@ protected Path getRemoteDir() {
return remoteDir;
}
+ @ParameterizedTest(name = "preserveTimes={0}, direct={1}, atomic={2}")
+ @CsvSource({
+ "true, false, false",
+ "false, false, false",
+ "true, true, false",
+ "false, true, false",
+ "true, false, true",
+ "false, false, true"
+ })
+ public void testChunkCopyPreserveTimes(boolean preserveTimes,
+ boolean directWrite, boolean atomicCommit) throws Exception {
+ requireChunkCopySupport();
+ Path source = new Path(remoteDir, "source");
+ Path target = new Path(remoteDir, "target");
+ Map<String, FileStatus> sourceStatuses = createChunkCopyTree(source);
+
+ DistCpOptions.Builder builder = chunkCopyOptions(source, target,
+ preserveTimes).withDirectWrite(directWrite || shouldUseDirectWrite())
+ .withAtomicCommit(atomicCommit);
+ runDistCp(builder.build(), true);
+
+ // The source directory itself is in the copy listing, so its times and
+ // those of nested directories must be restored after all concatenations.
+ verifyChunkCopyTree(target, sourceStatuses, preserveTimes, true);
+ }
+
+ @ParameterizedTest(name = "preserveTimes={0}, renameTarget={1}")
+ @CsvSource({"true, false", "false, false", "true, true", "false, true"})
+ public void testChunkCopyOutsideParentScope(boolean preserveTimes,
+ boolean renameTarget)
+ throws Exception {
+ requireChunkCopySupport();
+ Path sourceParent = new Path(remoteDir, "source-parent");
+ Path source = new Path(sourceParent, "file");
+ Path targetParent = new Path(remoteDir, "target-parent");
+ Path target = new Path(targetParent,
+ renameTarget ? "renamed-file" : source.getName());
+ createChunkCopyFile(source, TEST_FILE_LEN * 3 + 1);
+ FileStatus sourceStatus = setChunkCopyTimes(source, 0);
+ FileStatus sourceParentStatus = setChunkCopyTimes(sourceParent, 1);
+ mkdirs(remoteFS, targetParent);
+ FileStatus targetParentStatus = setChunkCopyTimes(targetParent, 2);
+
+ DistCpOptions.Builder builder = chunkCopyOptions(source,
+ renameTarget ? target : targetParent, preserveTimes);
+ if (renameTarget) {
+ builder.withDirectWrite(true);
+ }
+ runDistCp(builder.build(), true);
+
+ assertChunkCopyTimes(sourceStatus, remoteFS.getFileStatus(target),
+ preserveTimes);
+ // Creating/concatenating children may change the destination parent's
+ // mtime naturally. It must not inherit the source parent's timestamps.
+ FileStatus parentAfterCopy = remoteFS.getFileStatus(targetParent);
+ Assertions.assertThat(parentAfterCopy.getModificationTime())
+ .describedAs("mtime of parent outside copy scope")
+ .isNotEqualTo(sourceParentStatus.getModificationTime());
+ assertEquals(targetParentStatus.getAccessTime(),
+ parentAfterCopy.getAccessTime(), "atime of parent outside copy scope");
+ verifyFileContents(remoteFS, target,
+ dataset((int) sourceStatus.getLen(), 'a', 26));
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ public void testChunkCopyUpdateRootTimes(boolean updateRoot)
+ throws Exception {
+ requireChunkCopySupport();
+ Path source = new Path(remoteDir, "source");
+ Path target = new Path(remoteDir, "target");
+ Map<String, FileStatus> sourceStatuses = createChunkCopyTree(source);
+ // Force an existing destination file to be replaced by a chunked copy.
+ createFile(remoteFS, new Path(target, "large"), true, new byte[1]);
+ FileStatus targetRootStatus = setChunkCopyTimes(target, 20);
+ DistCpOptions options = chunkCopyOptions(source, target, true)
+ .withSyncFolder(true).withUpdateRoot(updateRoot).build();
+
+ runDistCp(options, true);
+ verifyChunkCopyTree(target, sourceStatuses, true, updateRoot);
+ if (!updateRoot) {
+ assertEquals(targetRootStatus.getAccessTime(),
+ remoteFS.getFileStatus(target).getAccessTime(),
+ "-update must not preserve root times without -updateRoot");
+ }
+
+ // A fully skipped update must still preserve file times without
+ // attempting to concatenate nonexistent chunk files.
+ setChunkCopyTimes(new Path(target, "large"), 30);
+ Job skipped = runDistCp(options, true);
+ assertCounterInRange(skipped, CopyMapper.Counter.COPY, 0, 0);
+ assertCounterInRange(skipped, CopyMapper.Counter.SKIP, 1, -1);
+ assertCounterInRange(skipped, CopyMapper.Counter.BYTESCOPIED, 0, 0);
+ verifyChunkCopyTree(target, sourceStatuses, true, updateRoot);
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ public void testChunkCopyPreserveOtherAttributes(boolean preserveTimes)
+ throws Exception {
+ requireChunkCopySupport();
+ skipIfUnsupported(SUPPORTS_UNIX_PERMISSIONS);
+ if (!remoteFS.hasPathCapability(remoteDir, FS_ACLS)
+ || !remoteFS.hasPathCapability(remoteDir, FS_XATTRS)) {
+ skip("ACL and XAttr support required");
+ }
+ Path source = new Path(remoteDir, "source");
+ Path target = new Path(remoteDir, "target");
+ Map<String, FileStatus> sourceStatuses = createChunkCopyTree(source);
+ Path sourceFile = new Path(source, "large");
+ remoteFS.setPermission(sourceFile, new FsPermission((short) 0640));
+ remoteFS.modifyAclEntries(sourceFile, Collections.singletonList(
+ new AclEntry.Builder().setScope(AclEntryScope.ACCESS)
+ .setType(AclEntryType.USER).setName("distcp-reader")
+ .setPermission(FsAction.READ).build()));
+ String xattrName = "user.distcp";
+ byte[] xattrValue = {1, 2, 3};
+ remoteFS.setXAttr(sourceFile, xattrName, xattrValue);
+ sourceStatuses.put("large", setChunkCopyTimes(sourceFile, 0));
+ sourceStatuses.put("", setChunkCopyTimes(source, 11));
+
+ runDistCp(chunkCopyOptions(source, target, preserveTimes)
+ .preserve("rbugpcax").build(), true);
+
+ // File listing entries omit extended ACLs: restoring all attributes from
+ // them after concat would erase ACLs already preserved by the mapper.
+ Path targetFile = new Path(target, "large");
+ assertEquals(remoteFS.getAclStatus(sourceFile).getEntries(),
+ remoteFS.getAclStatus(targetFile).getEntries(), "file ACL entries");
+ assertArrayEquals(xattrValue, remoteFS.getXAttr(targetFile, xattrName));
+ FileStatus expected = sourceStatuses.get("large");
+ FileStatus actual = remoteFS.getFileStatus(targetFile);
+ assertEquals(expected.getPermission(), actual.getPermission());
+ assertEquals(expected.getOwner(), actual.getOwner());
+ assertEquals(expected.getGroup(), actual.getGroup());
+ assertEquals(expected.getReplication(), actual.getReplication());
+ assertEquals(expected.getBlockSize(), actual.getBlockSize());
+ verifyChunkCopyTree(target, sourceStatuses, preserveTimes, true);
+ }
+
+ private void requireChunkCopySupport() throws IOException {
+ skipIfUnsupported(SUPPORTS_CONCAT);
+ skipIfUnsupported(SUPPORTS_SETTIMES);
+ skipIfUnsupported(SUPPORTS_BLOCK_LOCALITY);
Review Comment:
These checks admit `TestAdlContractDistCpLive`: `adls.xml` enables all three capabilities, but ADL ignores the requested block size and reports a fixed 256 MiB block size. The new 3–4 KiB fixtures therefore fail the multi-block assertion before DistCp runs on an otherwise working connector. Add a fixture-support hook or contract capability to skip implementations that cannot create these small multi-block files, or move these cases into the HDFS-specific suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-17611. Preserve DistCp file times after chunk concatenation. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8858:
URL: https://github.com/apache/hadoop/pull/8858#issuecomment-6408295885
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 18m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 55s | | trunk passed |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 0m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| -1 :x: | spotbugs | 1m 16s | [/branch-spotbugs-hadoop-tools_hadoop-distcp-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/2/artifact/out/branch-spotbugs-hadoop-tools_hadoop-distcp-warnings.html) | hadoop-tools/hadoop-distcp in trunk has 21 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 34m 59s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 25s | [/patch-mvninstall-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/2/artifact/out/patch-mvninstall-hadoop-tools_hadoop-distcp.txt) | hadoop-distcp in the patch failed. |
| -1 :x: | compile | 0m 24s | [/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-21.0.12.1+1-1-24.04.4-Ubuntu.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/2/artifact/out/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-21.0.12.1+1-1-24.04.4-Ubuntu.txt) | hadoop-distcp in the patch failed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu. |
| -1 :x: | javac | 0m 24s | [/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-21.0.12.1+1-1-24.04.4-Ubuntu.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/2/artifact/out/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-21.0.12.1+1-1-24.04.4-Ubuntu.txt) | hadoop-distcp in the patch failed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu. |
| -1 :x: | compile | 0m 25s | [/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-17.0.20.1+1-1-24.04-Ubuntu.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/2/artifact/out/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-17.0.20.1+1-1-24.04-Ubuntu.txt) | hadoop-distcp in the patch failed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu. |
| -1 :x: | javac | 0m 25s | [/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-17.0.20.1+1-1-24.04-Ubuntu.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/2/artifact/out/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-17.0.20.1+1-1-24.04-Ubuntu.txt) | hadoop-distcp in the patch failed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| -1 :x: | mvnsite | 0m 26s | [/patch-mvnsite-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/2/artifact/out/patch-mvnsite-hadoop-tools_hadoop-distcp.txt) | hadoop-distcp in the patch failed. |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| -1 :x: | spotbugs | 0m 24s | [/patch-spotbugs-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/2/artifact/out/patch-spotbugs-hadoop-tools_hadoop-distcp.txt) | hadoop-distcp in the patch failed. |
| +1 :green_heart: | shadedclient | 36m 27s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 28s | [/patch-unit-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/2/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt) | hadoop-distcp in the patch failed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 150m 33s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8858 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5fc053db0853 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1d2390aeb59a2c46d0bb8bcc165e3bf78fb1949d |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/2/testReport/ |
| Max. process+thread count | 583 (vs. ulimit of 10000) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8858/2/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org