You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Rohini U <ro...@gmail.com> on 2012/03/23 20:43:58 UTC

Problem using distcp with -file option to copy a folder heirarchy

I have a file structure based on dates Ex: 2011/01/01 and 2010/01/01 etc. I
have to copy data from dates 2010/01/01 to 2011/05/01 and I have specified
the dates in a file and used dist cp using the command give below:

hadoop distcp  -i -f file:///home/hadoop/list.txt    /copiedData

I now have two 01 folders one from 2011 and one from 2010.
Then, I get an error as shown below:

org.apache.hadoop.tools.DistCp$DuplicationException: Invalid input, there
are duplicated files in the sources

Is there a way I can copy an entire folder heirarchy by specifying the list
of directories to be copied in a file using the -f option ?
ie I want to replicate the directory structure from source but I only want
part of it.

Thanks,
Rohini