Append hdfs file in Java

[Jamil Shehzad <ja...@gmail.com>]

Hi I am working on hdfs in java and want to append data to an existing
file. I am using hadoop 2.2.0 and searched a lot about it but found nothing.

I know that if I want to append data then i can first copy data to a
temporary file and then write it back to the original file with appended
data. But my file data is of millions of lines. So each time performing
this operation will be very expensive for me.

Is there any built in functionality for appending data to hdfs file in
hadoop using java?

Currently I am using following piece of code, but its not efficient.

BufferedReader bfr=new BufferedReader(new
InputStreamReader(hdfs.open(newFilePath)));     //open file first
            String str = null;
            BufferedWriter br=new BufferedWriter(new
OutputStreamWriter(hdfs.create(newFilePath,true)));
            while ((str = bfr.readLine())!= null)
            {
                br.write(str); // write file content
                br.newLine();
                //System.out.println("   ->>>>>  "+str);
            }
            br.write(name);  // append into file
            br.newLine();
            br.close(); // close it

Any efficient idea about this?

Regards
Jamil Shehzad

RE: Append hdfs file in Java

["Webb, Ryan L." <Ry...@jhuapl.edu>]

3 Things;

1.       Wrong mailing list

2.       You might want to consider using hbase or accumulo instead of raw HDFS files

3.       It was not hard for me to google “hadoop 2.2.0 append” to find the API calls and configuration settings needed

From: Jamil Shehzad [mailto:jamil.shehzad9@gmail.com]
Sent: Thursday, October 30, 2014 2:14 PM
To: user@storm.apache.org
Subject: Append hdfs file in Java

Hi I am working on hdfs in java and want to append data to an existing file. I am using hadoop 2.2.0 and searched a lot about it but found nothing.
I know that if I want to append data then i can first copy data to a temporary file and then write it back to the original file with appended data. But my file data is of millions of lines. So each time performing this operation will be very expensive for me.
Is there any built in functionality for appending data to hdfs file in hadoop using java?
Currently I am using following piece of code, but its not efficient.

BufferedReader bfr=new BufferedReader(new InputStreamReader(hdfs.open(newFilePath)));     //open file first
            String str = null;
            BufferedWriter br=new BufferedWriter(new OutputStreamWriter(hdfs.create(newFilePath,true)));
            while ((str = bfr.readLine())!= null)
            {
                br.write(str); // write file content
                br.newLine();
                //System.out.println("   ->>>>>  "+str);
            }
            br.write(name);  // append into file
            br.newLine();
            br.close(); // close it
Any efficient idea about this?

Regards
Jamil Shehzad