You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by unmesha sreeveni <un...@gmail.com> on 2015/09/07 10:14:14 UTC
Not able to copy one HDFS data to another HDFS location using distcp
I am trying to copy one HDFS data to another HDFS location
I am able to achieve the same using "distcp" command
hadoop distcp hdfs://mySrcip:8020/copyDev/* hdfs://myDestip:8020/copyTest
But I want to try the same using Java Api
After a long search I found one code and executed . But it didnt copied my
src file to destination.
*public class TouchFile {*
* /***
* * @param args*
* * @throws Exception *
* */*
* public static void main(String[] args) throws Exception {*
* // TODO Auto-generated method stub*
* //create configuration object*
* Configuration config = new Configuration();*
* config.set("fs.defaultFS", "hdfs://mySrcip:8020/");*
* config.set("hadoop.job.ugi", "hdfs");*
* /**
* * Distcp*
* */*
* String sourceNameNode = "hdfs://mySrcip:8020/copyDev";*
* String destNameNode = "hdfs://myDestip:8020/copyTest";*
* String fileList = "myfile.txt";*
* distFileCopy(config,sourceNameNode,destNameNode,fileList);*
* }*
* /***
* * Copies files from one cloud to another using Hadoop's distributed
copy features. Uses*
* * input to build DISTCP configuration settings. *
* **
* * param config Hadoop configuration*
* * param sourceNameNode full HDFS path to parent source directory*
* * param destNameNode full HDFS path to parent destination directory*
* * param fileList Comma separated string of file names in
sourceNameNode to be copied to destNameNode*
* * returns Elapsed time in milliseconds to copy files*
* */*
* public static long distFileCopy( Configuration config, String
sourceNameNode, String destNameNode, String fileList ) throws Exception {*
* System.out.println("In dist copy");*
* StringTokenizer tokenizer = new StringTokenizer(fileList,",");*
* ArrayList<String> list = new ArrayList<>();*
* while ( tokenizer.hasMoreTokens() ){*
* String file = sourceNameNode + "/" + tokenizer.nextToken();*
* list.add( file );*
* }*
* String[] args = new String[list.size() + 1];*
* int count = 0;*
* for ( String filename : list ){*
* args[count++] = filename;*
* }*
* args[count] = destNameNode;*
* System.out.println("args------>"+Arrays.toString(args));*
* long st = System.currentTimeMillis(); *
* DistCp distCp=new DistCp(config,null);*
* distCp.run(args); *
* return System.currentTimeMillis() - st;*
* }*
*}*
Am I doing anything wrong.
Please suggest
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
http://www.unmeshasreeveni.blogspot.in/