[jira] [Commented] (HDFS-17948) Improve FedBalance DistCp option handling

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-17948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18092924#comment-18092924 ] 

ASF GitHub Bot commented on HDFS-17948:
---------------------------------------

qijiale76 opened a new pull request, #8579:
URL: https://github.com/apache/hadoop/pull/8579

   ### Description of PR
   
   This PR improves how FedBalance builds the DistCp commands it submits.
   
   Previously, FedBalance used a mostly fixed DistCp option set. This patch adds operator-facing controls for several DistCp settings:
   
   - Add `-skipAclPreserve` to disable ACL preservation.
   - Add `hdfs.fedbalance.preserve.acl.enabled` to control ACL preservation from configuration.
   - Preserve ACLs only when both source and destination filesystems support ACLs.
   - Add `-preserveTimes` to preserve file timestamps.
   - Add `-distcpStrategy` to pass a DistCp copy strategy.
   - Add `-numListstatusThreads` to pass DistCp listStatus thread count.
   - Register `-diffThreshold` in the shared FedBalance CLI options.
   - Serialize the new options in `FedBalanceContext` so recovered jobs use the same DistCp settings.
   - Update FedBalance documentation and tests.
   
   The default behavior remains conservative: timestamp preservation is disabled unless requested, DistCp strategy and listStatus thread count use DistCp defaults unless specified, and ACL preservation remains enabled only when ACLs are supported on both sides.
   
   ### How was this patch tested?
   
   Ran the related FedBalance unit tests.
   
   This change has also been validated on a production cluster.
   
   ### AI Tooling
   
   Contains content generated by Codex and Claude Code.




> Improve FedBalance DistCp option handling
> -----------------------------------------
>
>                 Key: HDFS-17948
>                 URL: https://issues.apache.org/jira/browse/HDFS-17948
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: QI, Jiale
>            Priority: Minor
>
> FedBalance currently submits DistCp jobs with a fixed set of DistCp preserve options and does not expose several useful DistCp knobs through the FedBalance CLI.
> This makes FedBalance less flexible for operators. For example:
> - ACL preservation is always requested by the generated DistCp command, which can fail or be undesirable when ACLs are not enabled on one side.
> - File timestamp preservation cannot be requested explicitly.
> - DistCp copy strategy cannot be selected through FedBalance.
> - DistCp listStatus parallelism cannot be tuned through FedBalance.
> - `-diffThreshold` is documented but not registered in the shared CLI options.
> This issue proposes adding FedBalance options for these DistCp settings while keeping the existing default behavior as much as possible:
> - Add `-skipAclPreserve` to disable ACL preservation.
> - Preserve ACLs only when both source and destination filesystems support ACLs.
> - Add `hdfs.fedbalance.preserve.acl.enabled` as a configuration-level ACL preserve switch.
> - Add `-preserveTimes` to include file timestamp preservation.
> - Add `-distcpStrategy` to pass a DistCp copy strategy.
> - Add `-numListstatusThreads` to tune DistCp listStatus threads.
> - Register `-diffThreshold` in the shared CLI option set.
> - Persist the new options in `FedBalanceContext` so recovered jobs keep the same behavior.
> Defaults should remain conservative: ACL preservation stays enabled when supported by both sides, timestamp preservation remains disabled unless requested, and DistCp strategy/listStatus thread settings use DistCp defaults unless explicitly configured.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org