You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "qijiale76 (via GitHub)" <gi...@apache.org> on 2026/07/01 11:00:22 UTC

[PR] HDFS-17948. Improve FedBalance DistCp option handling. [hadoop]

qijiale76 opened a new pull request, #8579:
URL: https://github.com/apache/hadoop/pull/8579

   ### Description of PR
   
   This PR improves how FedBalance builds the DistCp commands it submits.
   
   Previously, FedBalance used a mostly fixed DistCp option set. This patch adds operator-facing controls for several DistCp settings:
   
   - Add `-skipAclPreserve` to disable ACL preservation.
   - Add `hdfs.fedbalance.preserve.acl.enabled` to control ACL preservation from configuration.
   - Preserve ACLs only when both source and destination filesystems support ACLs.
   - Add `-preserveTimes` to preserve file timestamps.
   - Add `-distcpStrategy` to pass a DistCp copy strategy.
   - Add `-numListstatusThreads` to pass DistCp listStatus thread count.
   - Register `-diffThreshold` in the shared FedBalance CLI options.
   - Serialize the new options in `FedBalanceContext` so recovered jobs use the same DistCp settings.
   - Update FedBalance documentation and tests.
   
   The default behavior remains conservative: timestamp preservation is disabled unless requested, DistCp strategy and listStatus thread count use DistCp defaults unless specified, and ACL preservation remains enabled only when ACLs are supported on both sides.
   
   ### How was this patch tested?
   
   Ran the related FedBalance unit tests.
   
   This change has also been validated on a production cluster.
   
   ### AI Tooling
   
   Contains content generated by Codex and Claude Code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HDFS-17948. Improve FedBalance DistCp option handling. [hadoop]

Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8579:
URL: https://github.com/apache/hadoop/pull/8579#issuecomment-4859311115

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 35s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +0 :ok: |  xmllint  |   0m  0s |  |  xmllint was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to include 2 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  | 309m 34s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  56m 58s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   8m 35s |  |  trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu  |
   | +1 :green_heart: |  compile  |   8m 42s |  |  trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu  |
   | +1 :green_heart: |  checkstyle  |   3m  5s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 31s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m 15s |  |  trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu  |
   | +1 :green_heart: |  javadoc  |   1m 13s |  |  trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu  |
   | -1 :x: |  spotbugs  |   0m 43s | [/branch-spotbugs-hadoop-tools_hadoop-federation-balance-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8579/1/artifact/out/branch-spotbugs-hadoop-tools_hadoop-federation-balance-warnings.html) |  hadoop-tools/hadoop-federation-balance in trunk has 12 extant spotbugs warnings.  |
   | +1 :green_heart: |  shadedclient  |  15m 42s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 16s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   0m 48s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   8m  6s |  |  the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu  |
   | +1 :green_heart: |  javac  |   8m  6s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   8m 37s |  |  the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu  |
   | +1 :green_heart: |  javac  |   8m 37s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  checkstyle  |   2m 59s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   1m 30s |  |  the patch passed  |
   | -1 :x: |  javadoc  |   0m 34s | [/results-javadoc-javadoc-hadoop-tools_hadoop-federation-balance-jdkUbuntu-21.0.11+10-1-24.04.2-Ubuntu.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8579/1/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-federation-balance-jdkUbuntu-21.0.11+10-1-24.04.2-Ubuntu.txt) |  hadoop-tools_hadoop-federation-balance-jdkUbuntu-21.0.11+10-1-24.04.2-Ubuntu with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu generated 6 new + 97 unchanged - 0 fixed = 103 total (was 97)  |
   | -1 :x: |  javadoc  |   0m 35s | [/results-javadoc-javadoc-hadoop-tools_hadoop-federation-balance-jdkUbuntu-17.0.19+10-1-24.04.2-Ubuntu.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8579/1/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-federation-balance-jdkUbuntu-17.0.19+10-1-24.04.2-Ubuntu.txt) |  hadoop-tools_hadoop-federation-balance-jdkUbuntu-17.0.19+10-1-24.04.2-Ubuntu with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu generated 6 new + 93 unchanged - 0 fixed = 99 total (was 93)  |
   | +1 :green_heart: |  spotbugs  |   2m 23s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  16m 40s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   1m 59s |  |  hadoop-federation-balance in the patch passed.  |
   | +1 :green_heart: |  unit  |  37m 58s |  |  hadoop-hdfs-rbf in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 44s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 500m 20s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8579/1/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/8579 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
   | uname | Linux bf9e176846ad 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 53b09e0cc4c0a52f8634da5c2a2e502055110f5b |
   | Default Java | Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
   | Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8579/1/testReport/ |
   | Max. process+thread count | 4150 (vs. ulimit of 10000) |
   | modules | C: hadoop-tools/hadoop-federation-balance hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8579/1/console |
   | versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
   | Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HDFS-17948. Improve FedBalance DistCp option handling. [hadoop]

Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8579:
URL: https://github.com/apache/hadoop/pull/8579#issuecomment-4862281979

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 20s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +0 :ok: |  xmllint  |   0m  0s |  |  xmllint was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to include 2 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |   1m 49s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  25m 55s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   8m 13s |  |  trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu  |
   | +1 :green_heart: |  compile  |   8m 42s |  |  trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu  |
   | +1 :green_heart: |  checkstyle  |   2m 54s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 27s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m 19s |  |  trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu  |
   | +1 :green_heart: |  javadoc  |   1m 13s |  |  trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu  |
   | -1 :x: |  spotbugs  |   0m 45s | [/branch-spotbugs-hadoop-tools_hadoop-federation-balance-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8579/2/artifact/out/branch-spotbugs-hadoop-tools_hadoop-federation-balance-warnings.html) |  hadoop-tools/hadoop-federation-balance in trunk has 12 extant spotbugs warnings.  |
   | +1 :green_heart: |  shadedclient  |  16m 16s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 16s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   0m 43s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   8m  6s |  |  the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu  |
   | +1 :green_heart: |  javac  |   8m  6s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   8m 45s |  |  the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu  |
   | +1 :green_heart: |  javac  |   8m 45s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  checkstyle  |   2m 56s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   1m 29s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 33s |  |  hadoop-tools_hadoop-federation-balance-jdkUbuntu-21.0.11+10-1-24.04.2-Ubuntu with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu generated 0 new + 88 unchanged - 9 fixed = 88 total (was 97)  |
   | +1 :green_heart: |  javadoc  |   0m 41s |  |  hadoop-hdfs-rbf in the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu.  |
   | +1 :green_heart: |  javadoc  |   0m 34s |  |  hadoop-tools_hadoop-federation-balance-jdkUbuntu-17.0.19+10-1-24.04.2-Ubuntu with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu generated 0 new + 84 unchanged - 9 fixed = 84 total (was 93)  |
   | +1 :green_heart: |  javadoc  |   0m 38s |  |  hadoop-hdfs-rbf in the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu.  |
   | +1 :green_heart: |  spotbugs  |   2m 19s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  16m 41s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |   2m 14s |  |  hadoop-federation-balance in the patch passed.  |
   | +1 :green_heart: |  unit  |  37m 41s |  |  hadoop-hdfs-rbf in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 48s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 160m 55s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8579/2/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/8579 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
   | uname | Linux 96a9e1fddad0 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / f3be380094ed07f602defebcfb0a30c79365380f |
   | Default Java | Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
   | Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8579/2/testReport/ |
   | Max. process+thread count | 3691 (vs. ulimit of 10000) |
   | modules | C: hadoop-tools/hadoop-federation-balance hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8579/2/console |
   | versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
   | Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org