You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ajey Shah <aj...@gmail.com> on 2008/05/01 00:00:57 UTC
How do I copy files from my linux file system to HDFS using a java
prog?
Hello all,
I need to copy files from my linux file system to HDFS in a java program and
not manually. This is the piece of code that I have.
try {
FileSystem hdfs = FileSystem.get(new Configuration());
LocalFileSystem ls = null;
ls = hdfs.getLocal(hdfs.getConf());
hdfs.copyFromLocalFile(false, new Path(fileName), new Path(outputFile));
} catch (Exception e) {
e.printStackTrace();
}
The problem is that it searches for the input path on the HDFS and not my
linux file system.
Can someone point out where I may be wrong. I feel it's some configuration
issue but have not been able to figure it out.
Thanks.
--
View this message in context: http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-HDFS-using-a-java-prog--tp16992491p16992491.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
RE: How do I copy files from my linux file system to HDFS using a
java prog?
Posted by Ajey Shah <aj...@gmail.com>.
Thanks Suresh. It works! :)
Babu, Suresh wrote:
>
> This program can read from local file system as well as HDFS.
> Try <java program> <Localfile> <HDFSDestination>
>
> Thanks
> Suresh
>
> -----Original Message-----
> From: Ajey Shah [mailto:ajey.shah@gmail.com]
> Sent: Wednesday, May 07, 2008 4:04 AM
> To: core-user@hadoop.apache.org
> Subject: RE: How do I copy files from my linux file system to HDFS using
> a java prog?
>
>
> Thanks Suresh. But even this program reads and writes from the HDFS.
> What i need to do is read from my normal local linux harddrive and write
> to the HDFS.
>
> I'm sorry if I misunderstood your program.
>
> Thanks for replying. :)
>
>
>
> Babu, Suresh wrote:
>>
>>
>> Try this program. Modify the HDFS configuration, if it is different
>> from the default.
>>
>> import java.io.File;
>> import java.io.IOException;
>>
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.FileStatus; import
>> org.apache.hadoop.fs.FileSystem; import
>> org.apache.hadoop.fs.FSDataInputStream;
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.IOUtils;
>>
>> public class HadoopDFSFileReadWrite {
>>
>> static void usage () {
>> System.out.println("Usage : HadoopDFSFileReadWrite <inputfile>
>> <output file>");
>> System.exit(1);
>> }
>>
>> static void printAndExit(String str) {
>> System.err.println(str);
>> System.exit(1);
>> }
>>
>> public static void main (String[] argv) throws IOException {
>> Configuration conf = new Configuration();
>> conf.set("fs.default.name", "localhost:9000");
>> FileSystem fs = FileSystem.get(conf);
>>
>> FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory());
>> for(FileStatus status : fileStatus) {
>> System.out.println("File: " + status.getPath());
>> }
>>
>> if (argv.length != 2)
>> usage();
>>
>> // HadoopDFS deals with Path
>> Path inFile = new Path(argv[0]);
>> Path outFile = new Path(argv[1]);
>>
>> // Check if input/output are valid
>> if (!fs.exists(inFile))
>> printAndExit("Input file not found");
>> if (!fs.isFile(inFile))
>> printAndExit("Input should be a file");
>> if (fs.exists(outFile))
>> printAndExit("Output already exists");
>>
>> // Read from and write to new file
>> FSDataInputStream in = fs.open(inFile);
>> FSDataOutputStream out = fs.create(outFile);
>> byte buffer[] = new byte[256];
>> try {
>> int bytesRead = 0;
>> while ((bytesRead = in.read(buffer)) > 0) {
>> out.write(buffer, 0, bytesRead);
>> }
>>
>> } catch (IOException e) {
>> System.out.println("Error while copying file");
>> } finally {
>> in.close();
>> out.close();
>> }
>> }
>> }
>>
>> Suresh
>>
>>
>> -----Original Message-----
>> From: Ajey Shah [mailto:ajey.shah@gmail.com]
>> Sent: Thursday, May 01, 2008 3:31 AM
>> To: core-user@hadoop.apache.org
>> Subject: How do I copy files from my linux file system to HDFS using a
>
>> java prog?
>>
>>
>> Hello all,
>>
>> I need to copy files from my linux file system to HDFS in a java
>> program and not manually. This is the piece of code that I have.
>>
>> try {
>>
>> FileSystem hdfs = FileSystem.get(new
> Configuration());
>>
>> LocalFileSystem ls = null;
>>
>> ls = hdfs.getLocal(hdfs.getConf());
>>
>> hdfs.copyFromLocalFile(false, new
>> Path(fileName), new Path(outputFile));
>>
>> } catch (Exception e) {
>> e.printStackTrace();
>> }
>>
>> The problem is that it searches for the input path on the HDFS and not
>
>> my linux file system.
>>
>> Can someone point out where I may be wrong. I feel it's some
>> configuration issue but have not been able to figure it out.
>>
>> Thanks.
>> --
>> View this message in context:
>> http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to
>> -H DFS-using-a-java-prog--tp16992491p16992491.html
>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H
> DFS-using-a-java-prog--tp16992491p17093646.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-HDFS-using-a-java-prog--tp16992491p17136728.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: How do I copy files from my linux file system to HDFS using a
java prog?
Posted by Ajey Shah <aj...@gmail.com>.
Thanks Ted. It solves my problem. :)
Ted Dunning-3 wrote:
>
>
> I think that file names of the form "file://directory-path" should work to
> give you local file access using this program.
>
>
> On 5/6/08 3:34 PM, "Ajey Shah" <aj...@gmail.com> wrote:
>
>>
>> Thanks Suresh. But even this program reads and writes from the HDFS. What
>> i
>> need to do is read from my normal local linux harddrive and write to the
>> HDFS.
>>
>> I'm sorry if I misunderstood your program.
>>
>> Thanks for replying. :)
>>
>>
>>
>> Babu, Suresh wrote:
>>>
>>>
>>> Try this program. Modify the HDFS configuration, if it is different from
>>> the default.
>>>
>>> import java.io.File;
>>> import java.io.IOException;
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.fs.FileStatus;
>>> import org.apache.hadoop.fs.FileSystem;
>>> import org.apache.hadoop.fs.FSDataInputStream;
>>> import org.apache.hadoop.fs.FSDataOutputStream;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.IOUtils;
>>>
>>> public class HadoopDFSFileReadWrite {
>>>
>>> static void usage () {
>>> System.out.println("Usage : HadoopDFSFileReadWrite <inputfile>
>>> <output file>");
>>> System.exit(1);
>>> }
>>>
>>> static void printAndExit(String str) {
>>> System.err.println(str);
>>> System.exit(1);
>>> }
>>>
>>> public static void main (String[] argv) throws IOException {
>>> Configuration conf = new Configuration();
>>> conf.set("fs.default.name", "localhost:9000");
>>> FileSystem fs = FileSystem.get(conf);
>>>
>>> FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory());
>>> for(FileStatus status : fileStatus) {
>>> System.out.println("File: " + status.getPath());
>>> }
>>>
>>> if (argv.length != 2)
>>> usage();
>>>
>>> // HadoopDFS deals with Path
>>> Path inFile = new Path(argv[0]);
>>> Path outFile = new Path(argv[1]);
>>>
>>> // Check if input/output are valid
>>> if (!fs.exists(inFile))
>>> printAndExit("Input file not found");
>>> if (!fs.isFile(inFile))
>>> printAndExit("Input should be a file");
>>> if (fs.exists(outFile))
>>> printAndExit("Output already exists");
>>>
>>> // Read from and write to new file
>>> FSDataInputStream in = fs.open(inFile);
>>> FSDataOutputStream out = fs.create(outFile);
>>> byte buffer[] = new byte[256];
>>> try {
>>> int bytesRead = 0;
>>> while ((bytesRead = in.read(buffer)) > 0) {
>>> out.write(buffer, 0, bytesRead);
>>> }
>>>
>>> } catch (IOException e) {
>>> System.out.println("Error while copying file");
>>> } finally {
>>> in.close();
>>> out.close();
>>> }
>>> }
>>> }
>>>
>>> Suresh
>>>
>>>
>>> -----Original Message-----
>>> From: Ajey Shah [mailto:ajey.shah@gmail.com]
>>> Sent: Thursday, May 01, 2008 3:31 AM
>>> To: core-user@hadoop.apache.org
>>> Subject: How do I copy files from my linux file system to HDFS using a
>>> java prog?
>>>
>>>
>>> Hello all,
>>>
>>> I need to copy files from my linux file system to HDFS in a java program
>>> and not manually. This is the piece of code that I have.
>>>
>>> try {
>>>
>>> FileSystem hdfs = FileSystem.get(new
>>> Configuration());
>>>
>>> LocalFileSystem ls = null;
>>>
>>> ls = hdfs.getLocal(hdfs.getConf());
>>>
>>> hdfs.copyFromLocalFile(false, new
>>> Path(fileName), new Path(outputFile));
>>>
>>> } catch (Exception e) {
>>> e.printStackTrace();
>>> }
>>>
>>> The problem is that it searches for the input path on the HDFS and not
>>> my linux file system.
>>>
>>> Can someone point out where I may be wrong. I feel it's some
>>> configuration issue but have not been able to figure it out.
>>>
>>> Thanks.
>>> --
>>> View this message in context:
>>> http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H
>>> DFS-using-a-java-prog--tp16992491p16992491.html
>>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>>
>>>
>>>
>
>
>
--
View this message in context: http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-HDFS-using-a-java-prog--tp16992491p17136713.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: How do I copy files from my linux file system to HDFS using a
java prog?
Posted by Ted Dunning <td...@veoh.com>.
I think that file names of the form "file://directory-path" should work to
give you local file access using this program.
On 5/6/08 3:34 PM, "Ajey Shah" <aj...@gmail.com> wrote:
>
> Thanks Suresh. But even this program reads and writes from the HDFS. What i
> need to do is read from my normal local linux harddrive and write to the
> HDFS.
>
> I'm sorry if I misunderstood your program.
>
> Thanks for replying. :)
>
>
>
> Babu, Suresh wrote:
>>
>>
>> Try this program. Modify the HDFS configuration, if it is different from
>> the default.
>>
>> import java.io.File;
>> import java.io.IOException;
>>
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.FileStatus;
>> import org.apache.hadoop.fs.FileSystem;
>> import org.apache.hadoop.fs.FSDataInputStream;
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> import org.apache.hadoop.fs.Path;
>> import org.apache.hadoop.io.IOUtils;
>>
>> public class HadoopDFSFileReadWrite {
>>
>> static void usage () {
>> System.out.println("Usage : HadoopDFSFileReadWrite <inputfile>
>> <output file>");
>> System.exit(1);
>> }
>>
>> static void printAndExit(String str) {
>> System.err.println(str);
>> System.exit(1);
>> }
>>
>> public static void main (String[] argv) throws IOException {
>> Configuration conf = new Configuration();
>> conf.set("fs.default.name", "localhost:9000");
>> FileSystem fs = FileSystem.get(conf);
>>
>> FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory());
>> for(FileStatus status : fileStatus) {
>> System.out.println("File: " + status.getPath());
>> }
>>
>> if (argv.length != 2)
>> usage();
>>
>> // HadoopDFS deals with Path
>> Path inFile = new Path(argv[0]);
>> Path outFile = new Path(argv[1]);
>>
>> // Check if input/output are valid
>> if (!fs.exists(inFile))
>> printAndExit("Input file not found");
>> if (!fs.isFile(inFile))
>> printAndExit("Input should be a file");
>> if (fs.exists(outFile))
>> printAndExit("Output already exists");
>>
>> // Read from and write to new file
>> FSDataInputStream in = fs.open(inFile);
>> FSDataOutputStream out = fs.create(outFile);
>> byte buffer[] = new byte[256];
>> try {
>> int bytesRead = 0;
>> while ((bytesRead = in.read(buffer)) > 0) {
>> out.write(buffer, 0, bytesRead);
>> }
>>
>> } catch (IOException e) {
>> System.out.println("Error while copying file");
>> } finally {
>> in.close();
>> out.close();
>> }
>> }
>> }
>>
>> Suresh
>>
>>
>> -----Original Message-----
>> From: Ajey Shah [mailto:ajey.shah@gmail.com]
>> Sent: Thursday, May 01, 2008 3:31 AM
>> To: core-user@hadoop.apache.org
>> Subject: How do I copy files from my linux file system to HDFS using a
>> java prog?
>>
>>
>> Hello all,
>>
>> I need to copy files from my linux file system to HDFS in a java program
>> and not manually. This is the piece of code that I have.
>>
>> try {
>>
>> FileSystem hdfs = FileSystem.get(new
>> Configuration());
>>
>> LocalFileSystem ls = null;
>>
>> ls = hdfs.getLocal(hdfs.getConf());
>>
>> hdfs.copyFromLocalFile(false, new
>> Path(fileName), new Path(outputFile));
>>
>> } catch (Exception e) {
>> e.printStackTrace();
>> }
>>
>> The problem is that it searches for the input path on the HDFS and not
>> my linux file system.
>>
>> Can someone point out where I may be wrong. I feel it's some
>> configuration issue but have not been able to figure it out.
>>
>> Thanks.
>> --
>> View this message in context:
>> http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H
>> DFS-using-a-java-prog--tp16992491p16992491.html
>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>
>>
>>
RE: How do I copy files from my linux file system to HDFS using a java prog?
Posted by "Babu, Suresh" <Su...@corp.aol.com>.
This program can read from local file system as well as HDFS.
Try <java program> <Localfile> <HDFSDestination>
Thanks
Suresh
-----Original Message-----
From: Ajey Shah [mailto:ajey.shah@gmail.com]
Sent: Wednesday, May 07, 2008 4:04 AM
To: core-user@hadoop.apache.org
Subject: RE: How do I copy files from my linux file system to HDFS using
a java prog?
Thanks Suresh. But even this program reads and writes from the HDFS.
What i need to do is read from my normal local linux harddrive and write
to the HDFS.
I'm sorry if I misunderstood your program.
Thanks for replying. :)
Babu, Suresh wrote:
>
>
> Try this program. Modify the HDFS configuration, if it is different
> from the default.
>
> import java.io.File;
> import java.io.IOException;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FileStatus; import
> org.apache.hadoop.fs.FileSystem; import
> org.apache.hadoop.fs.FSDataInputStream;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IOUtils;
>
> public class HadoopDFSFileReadWrite {
>
> static void usage () {
> System.out.println("Usage : HadoopDFSFileReadWrite <inputfile>
> <output file>");
> System.exit(1);
> }
>
> static void printAndExit(String str) {
> System.err.println(str);
> System.exit(1);
> }
>
> public static void main (String[] argv) throws IOException {
> Configuration conf = new Configuration();
> conf.set("fs.default.name", "localhost:9000");
> FileSystem fs = FileSystem.get(conf);
>
> FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory());
> for(FileStatus status : fileStatus) {
> System.out.println("File: " + status.getPath());
> }
>
> if (argv.length != 2)
> usage();
>
> // HadoopDFS deals with Path
> Path inFile = new Path(argv[0]);
> Path outFile = new Path(argv[1]);
>
> // Check if input/output are valid
> if (!fs.exists(inFile))
> printAndExit("Input file not found");
> if (!fs.isFile(inFile))
> printAndExit("Input should be a file");
> if (fs.exists(outFile))
> printAndExit("Output already exists");
>
> // Read from and write to new file
> FSDataInputStream in = fs.open(inFile);
> FSDataOutputStream out = fs.create(outFile);
> byte buffer[] = new byte[256];
> try {
> int bytesRead = 0;
> while ((bytesRead = in.read(buffer)) > 0) {
> out.write(buffer, 0, bytesRead);
> }
>
> } catch (IOException e) {
> System.out.println("Error while copying file");
> } finally {
> in.close();
> out.close();
> }
> }
> }
>
> Suresh
>
>
> -----Original Message-----
> From: Ajey Shah [mailto:ajey.shah@gmail.com]
> Sent: Thursday, May 01, 2008 3:31 AM
> To: core-user@hadoop.apache.org
> Subject: How do I copy files from my linux file system to HDFS using a
> java prog?
>
>
> Hello all,
>
> I need to copy files from my linux file system to HDFS in a java
> program and not manually. This is the piece of code that I have.
>
> try {
>
> FileSystem hdfs = FileSystem.get(new
Configuration());
>
> LocalFileSystem ls = null;
>
> ls = hdfs.getLocal(hdfs.getConf());
>
> hdfs.copyFromLocalFile(false, new
> Path(fileName), new Path(outputFile));
>
> } catch (Exception e) {
> e.printStackTrace();
> }
>
> The problem is that it searches for the input path on the HDFS and not
> my linux file system.
>
> Can someone point out where I may be wrong. I feel it's some
> configuration issue but have not been able to figure it out.
>
> Thanks.
> --
> View this message in context:
> http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to
> -H DFS-using-a-java-prog--tp16992491p16992491.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
>
--
View this message in context:
http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H
DFS-using-a-java-prog--tp16992491p17093646.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
RE: How do I copy files from my linux file system to HDFS using a
java prog?
Posted by Ajey Shah <aj...@gmail.com>.
Thanks Suresh. But even this program reads and writes from the HDFS. What i
need to do is read from my normal local linux harddrive and write to the
HDFS.
I'm sorry if I misunderstood your program.
Thanks for replying. :)
Babu, Suresh wrote:
>
>
> Try this program. Modify the HDFS configuration, if it is different from
> the default.
>
> import java.io.File;
> import java.io.IOException;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FileStatus;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.FSDataInputStream;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.IOUtils;
>
> public class HadoopDFSFileReadWrite {
>
> static void usage () {
> System.out.println("Usage : HadoopDFSFileReadWrite <inputfile>
> <output file>");
> System.exit(1);
> }
>
> static void printAndExit(String str) {
> System.err.println(str);
> System.exit(1);
> }
>
> public static void main (String[] argv) throws IOException {
> Configuration conf = new Configuration();
> conf.set("fs.default.name", "localhost:9000");
> FileSystem fs = FileSystem.get(conf);
>
> FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory());
> for(FileStatus status : fileStatus) {
> System.out.println("File: " + status.getPath());
> }
>
> if (argv.length != 2)
> usage();
>
> // HadoopDFS deals with Path
> Path inFile = new Path(argv[0]);
> Path outFile = new Path(argv[1]);
>
> // Check if input/output are valid
> if (!fs.exists(inFile))
> printAndExit("Input file not found");
> if (!fs.isFile(inFile))
> printAndExit("Input should be a file");
> if (fs.exists(outFile))
> printAndExit("Output already exists");
>
> // Read from and write to new file
> FSDataInputStream in = fs.open(inFile);
> FSDataOutputStream out = fs.create(outFile);
> byte buffer[] = new byte[256];
> try {
> int bytesRead = 0;
> while ((bytesRead = in.read(buffer)) > 0) {
> out.write(buffer, 0, bytesRead);
> }
>
> } catch (IOException e) {
> System.out.println("Error while copying file");
> } finally {
> in.close();
> out.close();
> }
> }
> }
>
> Suresh
>
>
> -----Original Message-----
> From: Ajey Shah [mailto:ajey.shah@gmail.com]
> Sent: Thursday, May 01, 2008 3:31 AM
> To: core-user@hadoop.apache.org
> Subject: How do I copy files from my linux file system to HDFS using a
> java prog?
>
>
> Hello all,
>
> I need to copy files from my linux file system to HDFS in a java program
> and not manually. This is the piece of code that I have.
>
> try {
>
> FileSystem hdfs = FileSystem.get(new
> Configuration());
>
> LocalFileSystem ls = null;
>
> ls = hdfs.getLocal(hdfs.getConf());
>
> hdfs.copyFromLocalFile(false, new
> Path(fileName), new Path(outputFile));
>
> } catch (Exception e) {
> e.printStackTrace();
> }
>
> The problem is that it searches for the input path on the HDFS and not
> my linux file system.
>
> Can someone point out where I may be wrong. I feel it's some
> configuration issue but have not been able to figure it out.
>
> Thanks.
> --
> View this message in context:
> http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H
> DFS-using-a-java-prog--tp16992491p16992491.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-HDFS-using-a-java-prog--tp16992491p17093646.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
RE: How do I copy files from my linux file system to HDFS using a java prog?
Posted by "Babu, Suresh" <Su...@corp.aol.com>.
Try this program. Modify the HDFS configuration, if it is different from
the default.
import java.io.File;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
public class HadoopDFSFileReadWrite {
static void usage () {
System.out.println("Usage : HadoopDFSFileReadWrite <inputfile>
<output file>");
System.exit(1);
}
static void printAndExit(String str) {
System.err.println(str);
System.exit(1);
}
public static void main (String[] argv) throws IOException {
Configuration conf = new Configuration();
conf.set("fs.default.name", "localhost:9000");
FileSystem fs = FileSystem.get(conf);
FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory());
for(FileStatus status : fileStatus) {
System.out.println("File: " + status.getPath());
}
if (argv.length != 2)
usage();
// HadoopDFS deals with Path
Path inFile = new Path(argv[0]);
Path outFile = new Path(argv[1]);
// Check if input/output are valid
if (!fs.exists(inFile))
printAndExit("Input file not found");
if (!fs.isFile(inFile))
printAndExit("Input should be a file");
if (fs.exists(outFile))
printAndExit("Output already exists");
// Read from and write to new file
FSDataInputStream in = fs.open(inFile);
FSDataOutputStream out = fs.create(outFile);
byte buffer[] = new byte[256];
try {
int bytesRead = 0;
while ((bytesRead = in.read(buffer)) > 0) {
out.write(buffer, 0, bytesRead);
}
} catch (IOException e) {
System.out.println("Error while copying file");
} finally {
in.close();
out.close();
}
}
}
Suresh
-----Original Message-----
From: Ajey Shah [mailto:ajey.shah@gmail.com]
Sent: Thursday, May 01, 2008 3:31 AM
To: core-user@hadoop.apache.org
Subject: How do I copy files from my linux file system to HDFS using a
java prog?
Hello all,
I need to copy files from my linux file system to HDFS in a java program
and not manually. This is the piece of code that I have.
try {
FileSystem hdfs = FileSystem.get(new
Configuration());
LocalFileSystem ls = null;
ls = hdfs.getLocal(hdfs.getConf());
hdfs.copyFromLocalFile(false, new
Path(fileName), new Path(outputFile));
} catch (Exception e) {
e.printStackTrace();
}
The problem is that it searches for the input path on the HDFS and not
my linux file system.
Can someone point out where I may be wrong. I feel it's some
configuration issue but have not been able to figure it out.
Thanks.
--
View this message in context:
http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H
DFS-using-a-java-prog--tp16992491p16992491.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.