You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ajey Shah <aj...@gmail.com> on 2008/05/01 00:00:57 UTC

How do I copy files from my linux file system to HDFS using a java prog?

Hello all,

I need to copy files from my linux file system to HDFS in a java program and
not manually. This is the piece of code that I have.

try {

			FileSystem hdfs = FileSystem.get(new Configuration());
			
			LocalFileSystem ls = null;
			
			ls = hdfs.getLocal(hdfs.getConf());
			
			hdfs.copyFromLocalFile(false, new Path(fileName), new Path(outputFile));

		} catch (Exception e) {
			e.printStackTrace();
		}

The problem is that it searches for the input path on the HDFS and not my
linux file system.

Can someone point out where I may be wrong. I feel it's some configuration
issue but have not been able to figure it out. 

Thanks.
-- 
View this message in context: http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-HDFS-using-a-java-prog--tp16992491p16992491.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.


RE: How do I copy files from my linux file system to HDFS using a java prog?

Posted by Ajey Shah <aj...@gmail.com>.
Thanks Suresh. It works! :)


Babu, Suresh wrote:
> 
> This program can read from local file system as well as HDFS.
> Try <java program> <Localfile> <HDFSDestination>
> 
> Thanks
> Suresh
> 
> -----Original Message-----
> From: Ajey Shah [mailto:ajey.shah@gmail.com] 
> Sent: Wednesday, May 07, 2008 4:04 AM
> To: core-user@hadoop.apache.org
> Subject: RE: How do I copy files from my linux file system to HDFS using
> a java prog?
> 
> 
> Thanks Suresh. But even this program reads and writes from the HDFS.
> What i need to do is read from my normal local linux harddrive and write
> to the HDFS.
> 
> I'm sorry if I misunderstood your program.
> 
> Thanks for replying. :)
> 
> 
> 
> Babu, Suresh wrote:
>> 
>> 
>> Try this program. Modify the HDFS configuration, if it is different 
>> from the default.
>> 
>> import java.io.File;
>> import java.io.IOException;
>> 
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.FileStatus; import 
>> org.apache.hadoop.fs.FileSystem; import 
>> org.apache.hadoop.fs.FSDataInputStream;
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.IOUtils;
>> 
>> public class HadoopDFSFileReadWrite {
>> 
>>   static void usage () {
>>     System.out.println("Usage : HadoopDFSFileReadWrite <inputfile> 
>> <output file>");
>>     System.exit(1);
>>   }
>> 
>>   static void printAndExit(String str) {
>>     System.err.println(str);
>>     System.exit(1);
>>   }
>> 
>>   public static void main (String[] argv) throws IOException {
>>     Configuration conf = new Configuration();
>>     conf.set("fs.default.name", "localhost:9000");
>>     FileSystem fs = FileSystem.get(conf);
>> 
>>     FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory());
>>     for(FileStatus status : fileStatus) {
>>         System.out.println("File: " + status.getPath());
>>     }
>> 
>>     if (argv.length != 2)
>>       usage();
>> 
>>     // HadoopDFS deals with Path
>>     Path inFile = new Path(argv[0]);
>>     Path outFile = new Path(argv[1]);
>> 
>>     // Check if input/output are valid
>>     if (!fs.exists(inFile))
>>       printAndExit("Input file not found");
>>     if (!fs.isFile(inFile))
>>       printAndExit("Input should be a file");
>>     if (fs.exists(outFile))
>>       printAndExit("Output already exists");
>> 
>>     // Read from and write to new file
>>     FSDataInputStream in = fs.open(inFile);
>>     FSDataOutputStream out = fs.create(outFile);
>>     byte buffer[] = new byte[256];
>>     try {
>>       int bytesRead = 0;
>>       while ((bytesRead = in.read(buffer)) > 0) {
>>         out.write(buffer, 0, bytesRead);
>>       }
>> 
>>     } catch (IOException e) {
>>       System.out.println("Error while copying file");
>>     } finally {
>>       in.close();
>>       out.close();
>>     }
>>   }
>> }
>> 
>> Suresh
>> 
>> 
>> -----Original Message-----
>> From: Ajey Shah [mailto:ajey.shah@gmail.com]
>> Sent: Thursday, May 01, 2008 3:31 AM
>> To: core-user@hadoop.apache.org
>> Subject: How do I copy files from my linux file system to HDFS using a
> 
>> java prog?
>> 
>> 
>> Hello all,
>> 
>> I need to copy files from my linux file system to HDFS in a java 
>> program and not manually. This is the piece of code that I have.
>> 
>> try {
>> 
>> 			FileSystem hdfs = FileSystem.get(new
> Configuration());
>> 			
>> 			LocalFileSystem ls = null;
>> 			
>> 			ls = hdfs.getLocal(hdfs.getConf());
>> 			
>> 			hdfs.copyFromLocalFile(false, new
>> Path(fileName), new Path(outputFile));
>> 
>> 		} catch (Exception e) {
>> 			e.printStackTrace();
>> 		}
>> 
>> The problem is that it searches for the input path on the HDFS and not
> 
>> my linux file system.
>> 
>> Can someone point out where I may be wrong. I feel it's some 
>> configuration issue but have not been able to figure it out.
>> 
>> Thanks.
>> --
>> View this message in context:
>> http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to
>> -H DFS-using-a-java-prog--tp16992491p16992491.html
>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>> 
>> 
>> 
> 
> --
> View this message in context:
> http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H
> DFS-using-a-java-prog--tp16992491p17093646.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-HDFS-using-a-java-prog--tp16992491p17136728.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.


Re: How do I copy files from my linux file system to HDFS using a java prog?

Posted by Ajey Shah <aj...@gmail.com>.
Thanks Ted. It solves my problem. :)

Ted Dunning-3 wrote:
> 
> 
> I think that file names of the form "file://directory-path" should work to
> give you local file access using this program.
> 
> 
> On 5/6/08 3:34 PM, "Ajey Shah" <aj...@gmail.com> wrote:
> 
>> 
>> Thanks Suresh. But even this program reads and writes from the HDFS. What
>> i
>> need to do is read from my normal local linux harddrive and write to the
>> HDFS.
>> 
>> I'm sorry if I misunderstood your program.
>> 
>> Thanks for replying. :)
>> 
>> 
>> 
>> Babu, Suresh wrote:
>>> 
>>> 
>>> Try this program. Modify the HDFS configuration, if it is different from
>>> the default.
>>> 
>>> import java.io.File;
>>> import java.io.IOException;
>>> 
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.fs.FileStatus;
>>> import org.apache.hadoop.fs.FileSystem;
>>> import org.apache.hadoop.fs.FSDataInputStream;
>>> import org.apache.hadoop.fs.FSDataOutputStream;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.IOUtils;
>>> 
>>> public class HadoopDFSFileReadWrite {
>>> 
>>>   static void usage () {
>>>     System.out.println("Usage : HadoopDFSFileReadWrite <inputfile>
>>> <output file>");
>>>     System.exit(1);
>>>   }
>>> 
>>>   static void printAndExit(String str) {
>>>     System.err.println(str);
>>>     System.exit(1);
>>>   }
>>> 
>>>   public static void main (String[] argv) throws IOException {
>>>     Configuration conf = new Configuration();
>>>     conf.set("fs.default.name", "localhost:9000");
>>>     FileSystem fs = FileSystem.get(conf);
>>> 
>>>     FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory());
>>>     for(FileStatus status : fileStatus) {
>>>         System.out.println("File: " + status.getPath());
>>>     }
>>> 
>>>     if (argv.length != 2)
>>>       usage();
>>> 
>>>     // HadoopDFS deals with Path
>>>     Path inFile = new Path(argv[0]);
>>>     Path outFile = new Path(argv[1]);
>>> 
>>>     // Check if input/output are valid
>>>     if (!fs.exists(inFile))
>>>       printAndExit("Input file not found");
>>>     if (!fs.isFile(inFile))
>>>       printAndExit("Input should be a file");
>>>     if (fs.exists(outFile))
>>>       printAndExit("Output already exists");
>>> 
>>>     // Read from and write to new file
>>>     FSDataInputStream in = fs.open(inFile);
>>>     FSDataOutputStream out = fs.create(outFile);
>>>     byte buffer[] = new byte[256];
>>>     try {
>>>       int bytesRead = 0;
>>>       while ((bytesRead = in.read(buffer)) > 0) {
>>>         out.write(buffer, 0, bytesRead);
>>>       }
>>> 
>>>     } catch (IOException e) {
>>>       System.out.println("Error while copying file");
>>>     } finally {
>>>       in.close();
>>>       out.close();
>>>     }
>>>   }
>>> }
>>> 
>>> Suresh
>>> 
>>> 
>>> -----Original Message-----
>>> From: Ajey Shah [mailto:ajey.shah@gmail.com]
>>> Sent: Thursday, May 01, 2008 3:31 AM
>>> To: core-user@hadoop.apache.org
>>> Subject: How do I copy files from my linux file system to HDFS using a
>>> java prog?
>>> 
>>> 
>>> Hello all,
>>> 
>>> I need to copy files from my linux file system to HDFS in a java program
>>> and not manually. This is the piece of code that I have.
>>> 
>>> try {
>>> 
>>> FileSystem hdfs = FileSystem.get(new
>>> Configuration());
>>> 
>>> LocalFileSystem ls = null;
>>> 
>>> ls = hdfs.getLocal(hdfs.getConf());
>>> 
>>> hdfs.copyFromLocalFile(false, new
>>> Path(fileName), new Path(outputFile));
>>> 
>>> } catch (Exception e) {
>>> e.printStackTrace();
>>> }
>>> 
>>> The problem is that it searches for the input path on the HDFS and not
>>> my linux file system.
>>> 
>>> Can someone point out where I may be wrong. I feel it's some
>>> configuration issue but have not been able to figure it out.
>>> 
>>> Thanks.
>>> --
>>> View this message in context:
>>> http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H
>>> DFS-using-a-java-prog--tp16992491p16992491.html
>>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>> 
>>> 
>>> 
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-HDFS-using-a-java-prog--tp16992491p17136713.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.


Re: How do I copy files from my linux file system to HDFS using a java prog?

Posted by Ted Dunning <td...@veoh.com>.
I think that file names of the form "file://directory-path" should work to
give you local file access using this program.


On 5/6/08 3:34 PM, "Ajey Shah" <aj...@gmail.com> wrote:

> 
> Thanks Suresh. But even this program reads and writes from the HDFS. What i
> need to do is read from my normal local linux harddrive and write to the
> HDFS.
> 
> I'm sorry if I misunderstood your program.
> 
> Thanks for replying. :)
> 
> 
> 
> Babu, Suresh wrote:
>> 
>> 
>> Try this program. Modify the HDFS configuration, if it is different from
>> the default.
>> 
>> import java.io.File;
>> import java.io.IOException;
>> 
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.FileStatus;
>> import org.apache.hadoop.fs.FileSystem;
>> import org.apache.hadoop.fs.FSDataInputStream;
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.IOUtils;
>> 
>> public class HadoopDFSFileReadWrite {
>> 
>>   static void usage () {
>>     System.out.println("Usage : HadoopDFSFileReadWrite <inputfile>
>> <output file>");
>>     System.exit(1);
>>   }
>> 
>>   static void printAndExit(String str) {
>>     System.err.println(str);
>>     System.exit(1);
>>   }
>> 
>>   public static void main (String[] argv) throws IOException {
>>     Configuration conf = new Configuration();
>>     conf.set("fs.default.name", "localhost:9000");
>>     FileSystem fs = FileSystem.get(conf);
>> 
>>     FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory());
>>     for(FileStatus status : fileStatus) {
>>         System.out.println("File: " + status.getPath());
>>     }
>> 
>>     if (argv.length != 2)
>>       usage();
>> 
>>     // HadoopDFS deals with Path
>>     Path inFile = new Path(argv[0]);
>>     Path outFile = new Path(argv[1]);
>> 
>>     // Check if input/output are valid
>>     if (!fs.exists(inFile))
>>       printAndExit("Input file not found");
>>     if (!fs.isFile(inFile))
>>       printAndExit("Input should be a file");
>>     if (fs.exists(outFile))
>>       printAndExit("Output already exists");
>> 
>>     // Read from and write to new file
>>     FSDataInputStream in = fs.open(inFile);
>>     FSDataOutputStream out = fs.create(outFile);
>>     byte buffer[] = new byte[256];
>>     try {
>>       int bytesRead = 0;
>>       while ((bytesRead = in.read(buffer)) > 0) {
>>         out.write(buffer, 0, bytesRead);
>>       }
>> 
>>     } catch (IOException e) {
>>       System.out.println("Error while copying file");
>>     } finally {
>>       in.close();
>>       out.close();
>>     }
>>   }
>> }
>> 
>> Suresh
>> 
>> 
>> -----Original Message-----
>> From: Ajey Shah [mailto:ajey.shah@gmail.com]
>> Sent: Thursday, May 01, 2008 3:31 AM
>> To: core-user@hadoop.apache.org
>> Subject: How do I copy files from my linux file system to HDFS using a
>> java prog?
>> 
>> 
>> Hello all,
>> 
>> I need to copy files from my linux file system to HDFS in a java program
>> and not manually. This is the piece of code that I have.
>> 
>> try {
>> 
>> FileSystem hdfs = FileSystem.get(new
>> Configuration());
>> 
>> LocalFileSystem ls = null;
>> 
>> ls = hdfs.getLocal(hdfs.getConf());
>> 
>> hdfs.copyFromLocalFile(false, new
>> Path(fileName), new Path(outputFile));
>> 
>> } catch (Exception e) {
>> e.printStackTrace();
>> }
>> 
>> The problem is that it searches for the input path on the HDFS and not
>> my linux file system.
>> 
>> Can someone point out where I may be wrong. I feel it's some
>> configuration issue but have not been able to figure it out.
>> 
>> Thanks.
>> --
>> View this message in context:
>> http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H
>> DFS-using-a-java-prog--tp16992491p16992491.html
>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>> 
>> 
>> 


RE: How do I copy files from my linux file system to HDFS using a java prog?

Posted by "Babu, Suresh" <Su...@corp.aol.com>.
This program can read from local file system as well as HDFS.
Try <java program> <Localfile> <HDFSDestination>

Thanks
Suresh

-----Original Message-----
From: Ajey Shah [mailto:ajey.shah@gmail.com] 
Sent: Wednesday, May 07, 2008 4:04 AM
To: core-user@hadoop.apache.org
Subject: RE: How do I copy files from my linux file system to HDFS using
a java prog?


Thanks Suresh. But even this program reads and writes from the HDFS.
What i need to do is read from my normal local linux harddrive and write
to the HDFS.

I'm sorry if I misunderstood your program.

Thanks for replying. :)



Babu, Suresh wrote:
> 
> 
> Try this program. Modify the HDFS configuration, if it is different 
> from the default.
> 
> import java.io.File;
> import java.io.IOException;
> 
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FileStatus; import 
> org.apache.hadoop.fs.FileSystem; import 
> org.apache.hadoop.fs.FSDataInputStream;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IOUtils;
> 
> public class HadoopDFSFileReadWrite {
> 
>   static void usage () {
>     System.out.println("Usage : HadoopDFSFileReadWrite <inputfile> 
> <output file>");
>     System.exit(1);
>   }
> 
>   static void printAndExit(String str) {
>     System.err.println(str);
>     System.exit(1);
>   }
> 
>   public static void main (String[] argv) throws IOException {
>     Configuration conf = new Configuration();
>     conf.set("fs.default.name", "localhost:9000");
>     FileSystem fs = FileSystem.get(conf);
> 
>     FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory());
>     for(FileStatus status : fileStatus) {
>         System.out.println("File: " + status.getPath());
>     }
> 
>     if (argv.length != 2)
>       usage();
> 
>     // HadoopDFS deals with Path
>     Path inFile = new Path(argv[0]);
>     Path outFile = new Path(argv[1]);
> 
>     // Check if input/output are valid
>     if (!fs.exists(inFile))
>       printAndExit("Input file not found");
>     if (!fs.isFile(inFile))
>       printAndExit("Input should be a file");
>     if (fs.exists(outFile))
>       printAndExit("Output already exists");
> 
>     // Read from and write to new file
>     FSDataInputStream in = fs.open(inFile);
>     FSDataOutputStream out = fs.create(outFile);
>     byte buffer[] = new byte[256];
>     try {
>       int bytesRead = 0;
>       while ((bytesRead = in.read(buffer)) > 0) {
>         out.write(buffer, 0, bytesRead);
>       }
> 
>     } catch (IOException e) {
>       System.out.println("Error while copying file");
>     } finally {
>       in.close();
>       out.close();
>     }
>   }
> }
> 
> Suresh
> 
> 
> -----Original Message-----
> From: Ajey Shah [mailto:ajey.shah@gmail.com]
> Sent: Thursday, May 01, 2008 3:31 AM
> To: core-user@hadoop.apache.org
> Subject: How do I copy files from my linux file system to HDFS using a

> java prog?
> 
> 
> Hello all,
> 
> I need to copy files from my linux file system to HDFS in a java 
> program and not manually. This is the piece of code that I have.
> 
> try {
> 
> 			FileSystem hdfs = FileSystem.get(new
Configuration());
> 			
> 			LocalFileSystem ls = null;
> 			
> 			ls = hdfs.getLocal(hdfs.getConf());
> 			
> 			hdfs.copyFromLocalFile(false, new
> Path(fileName), new Path(outputFile));
> 
> 		} catch (Exception e) {
> 			e.printStackTrace();
> 		}
> 
> The problem is that it searches for the input path on the HDFS and not

> my linux file system.
> 
> Can someone point out where I may be wrong. I feel it's some 
> configuration issue but have not been able to figure it out.
> 
> Thanks.
> --
> View this message in context:
> http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to
> -H DFS-using-a-java-prog--tp16992491p16992491.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
> 
> 
> 

--
View this message in context:
http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H
DFS-using-a-java-prog--tp16992491p17093646.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.


RE: How do I copy files from my linux file system to HDFS using a java prog?

Posted by Ajey Shah <aj...@gmail.com>.
Thanks Suresh. But even this program reads and writes from the HDFS. What i
need to do is read from my normal local linux harddrive and write to the
HDFS.

I'm sorry if I misunderstood your program.

Thanks for replying. :)



Babu, Suresh wrote:
> 
> 
> Try this program. Modify the HDFS configuration, if it is different from
> the default.
> 
> import java.io.File;
> import java.io.IOException;
> 
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FileStatus;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.FSDataInputStream;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IOUtils;
> 
> public class HadoopDFSFileReadWrite {
> 
>   static void usage () {
>     System.out.println("Usage : HadoopDFSFileReadWrite <inputfile>
> <output file>");
>     System.exit(1);
>   }
> 
>   static void printAndExit(String str) {
>     System.err.println(str);
>     System.exit(1);
>   }
> 
>   public static void main (String[] argv) throws IOException {
>     Configuration conf = new Configuration();
>     conf.set("fs.default.name", "localhost:9000");
>     FileSystem fs = FileSystem.get(conf);
> 
>     FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory());
>     for(FileStatus status : fileStatus) {
>         System.out.println("File: " + status.getPath());
>     }
> 
>     if (argv.length != 2)
>       usage();
> 
>     // HadoopDFS deals with Path
>     Path inFile = new Path(argv[0]);
>     Path outFile = new Path(argv[1]);
> 
>     // Check if input/output are valid
>     if (!fs.exists(inFile))
>       printAndExit("Input file not found");
>     if (!fs.isFile(inFile))
>       printAndExit("Input should be a file");
>     if (fs.exists(outFile))
>       printAndExit("Output already exists");
> 
>     // Read from and write to new file
>     FSDataInputStream in = fs.open(inFile);
>     FSDataOutputStream out = fs.create(outFile);
>     byte buffer[] = new byte[256];
>     try {
>       int bytesRead = 0;
>       while ((bytesRead = in.read(buffer)) > 0) {
>         out.write(buffer, 0, bytesRead);
>       }
> 
>     } catch (IOException e) {
>       System.out.println("Error while copying file");
>     } finally {
>       in.close();
>       out.close();
>     }
>   }
> }
> 
> Suresh
> 
> 
> -----Original Message-----
> From: Ajey Shah [mailto:ajey.shah@gmail.com] 
> Sent: Thursday, May 01, 2008 3:31 AM
> To: core-user@hadoop.apache.org
> Subject: How do I copy files from my linux file system to HDFS using a
> java prog?
> 
> 
> Hello all,
> 
> I need to copy files from my linux file system to HDFS in a java program
> and not manually. This is the piece of code that I have.
> 
> try {
> 
> 			FileSystem hdfs = FileSystem.get(new
> Configuration());
> 			
> 			LocalFileSystem ls = null;
> 			
> 			ls = hdfs.getLocal(hdfs.getConf());
> 			
> 			hdfs.copyFromLocalFile(false, new
> Path(fileName), new Path(outputFile));
> 
> 		} catch (Exception e) {
> 			e.printStackTrace();
> 		}
> 
> The problem is that it searches for the input path on the HDFS and not
> my linux file system.
> 
> Can someone point out where I may be wrong. I feel it's some
> configuration issue but have not been able to figure it out. 
> 
> Thanks.
> --
> View this message in context:
> http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H
> DFS-using-a-java-prog--tp16992491p16992491.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-HDFS-using-a-java-prog--tp16992491p17093646.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.


RE: How do I copy files from my linux file system to HDFS using a java prog?

Posted by "Babu, Suresh" <Su...@corp.aol.com>.
Try this program. Modify the HDFS configuration, if it is different from
the default.

import java.io.File;
import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

public class HadoopDFSFileReadWrite {

  static void usage () {
    System.out.println("Usage : HadoopDFSFileReadWrite <inputfile>
<output file>");
    System.exit(1);
  }

  static void printAndExit(String str) {
    System.err.println(str);
    System.exit(1);
  }

  public static void main (String[] argv) throws IOException {
    Configuration conf = new Configuration();
    conf.set("fs.default.name", "localhost:9000");
    FileSystem fs = FileSystem.get(conf);

    FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory());
    for(FileStatus status : fileStatus) {
        System.out.println("File: " + status.getPath());
    }

    if (argv.length != 2)
      usage();

    // HadoopDFS deals with Path
    Path inFile = new Path(argv[0]);
    Path outFile = new Path(argv[1]);

    // Check if input/output are valid
    if (!fs.exists(inFile))
      printAndExit("Input file not found");
    if (!fs.isFile(inFile))
      printAndExit("Input should be a file");
    if (fs.exists(outFile))
      printAndExit("Output already exists");

    // Read from and write to new file
    FSDataInputStream in = fs.open(inFile);
    FSDataOutputStream out = fs.create(outFile);
    byte buffer[] = new byte[256];
    try {
      int bytesRead = 0;
      while ((bytesRead = in.read(buffer)) > 0) {
        out.write(buffer, 0, bytesRead);
      }

    } catch (IOException e) {
      System.out.println("Error while copying file");
    } finally {
      in.close();
      out.close();
    }
  }
}

Suresh


-----Original Message-----
From: Ajey Shah [mailto:ajey.shah@gmail.com] 
Sent: Thursday, May 01, 2008 3:31 AM
To: core-user@hadoop.apache.org
Subject: How do I copy files from my linux file system to HDFS using a
java prog?


Hello all,

I need to copy files from my linux file system to HDFS in a java program
and not manually. This is the piece of code that I have.

try {

			FileSystem hdfs = FileSystem.get(new
Configuration());
			
			LocalFileSystem ls = null;
			
			ls = hdfs.getLocal(hdfs.getConf());
			
			hdfs.copyFromLocalFile(false, new
Path(fileName), new Path(outputFile));

		} catch (Exception e) {
			e.printStackTrace();
		}

The problem is that it searches for the input path on the HDFS and not
my linux file system.

Can someone point out where I may be wrong. I feel it's some
configuration issue but have not been able to figure it out. 

Thanks.
--
View this message in context:
http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H
DFS-using-a-java-prog--tp16992491p16992491.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.