You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2024/10/11 08:53:00 UTC
[jira] [Commented] (HADOOP-19307) Add option to add parent directory of source directories to target directories
[ https://issues.apache.org/jira/browse/HADOOP-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17888568#comment-17888568 ]
ASF GitHub Bot commented on HADOOP-19307:
-----------------------------------------
AuthurWang2009 opened a new pull request, #7110:
URL: https://github.com/apache/hadoop/pull/7110
### Description of PR
support for copying source parent directories with -update and -overwrite options
### How was this patch tested?
add new integration tests
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
> Add option to add parent directory of source directories to target directories
> ------------------------------------------------------------------------------
>
> Key: HADOOP-19307
> URL: https://issues.apache.org/jira/browse/HADOOP-19307
> Project: Hadoop Common
> Issue Type: New Feature
> Components: tools/distcp
> Affects Versions: 3.0.0
> Environment: hadoop 3.3.1
> Reporter: Authur Wang
> Priority: Major
>
> Currently, when we execute the Hadoop distcp with -update -delete src1/* src2/* dest command to keep the source and target directories exactly the same。 When either -update or -overwrite is specified, the *contents* of the source-directories are copied to target, and not the source directories themselves.
> Consider a copy from /source/first/ and /source/second/ to /target/, where the source paths have the following contents:
> hdfs://nn1:8020/source/first/1
> hdfs://nn1:8020/source/first/2
> hdfs://nn1:8020/source/second/10
> hdfs://nn1:8020/source/second/20
> distcp2 -update hdfs://nn1:8020/source/first hdfs://nn1:8020/source/second hdfs://nn2:8020/target
> would yield the following contents in /target:
> hdfs://nn2:8020/target/1
> hdfs://nn2:8020/target/2
> hdfs://nn2:8020/target/10
> hdfs://nn2:8020/target/20
>
> But, sometimes, we need to preserve parent directories like this:
> hdfs://nn1:8020/target/first/1
> hdfs://nn1:8020/target/first/2
> hdfs://nn1:8020/target/second/10
> hdfs://nn1:8020/target/second/20
>
> So, should we introduce an option -preserveParentDir to keep the parent directories to be copied with -update or -overwrite ?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org