Issue in Distcp from S3 to HDFS

[Stuti Awasthi <st...@hcl.com>]

Hi,

I want to copy data from S3 to Hadoop set up on EC2 instances. To attain this I configured AWS keys in Hadoop configuration file and able to copy file from S3 to HDFS. But getting error if I want to copy the complete folder hierarchy to HDFS.

Command I used :

bin/hadoop distcp s3n://MyBucket/MyFolder/ hdfs://Hostname:Port/test/OUTPUT/

org.apache.hadoop.tools.DistCp$DuplicationException: Invalid input, there are duplicated files in the sources: s3n://MyBucket/MyFolder/Hello, s3n://MyBucket/MyFolder/Hello
at org.apache.hadoop.tools.DistCp.checkDuplication(DistCp.java:1368)
at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1176)
at org.apache.hadoop.tools.DistCp.copy(DistCp.java:666)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)

Please Suggest

Stuti


::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.

----------------------------------------------------------------------------------------------------------------------------------------------------