You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Raghava Mutharaju <m....@gmail.com> on 2010/02/17 01:54:46 UTC
appending to files on HDFS
Hi all,
I am using hadoop 0.20.1 version. I am trying to append some content
from a file on HDFS to another file on HDFS. I am using FileSystem.append()
method to get the stream and then passed it to IOUtils.copyBytes() method. I
got the following exception
org.apache.hadoop.ipc.RemoteException: java.io.IOException: Append to hdfs
not supported. Please refer to dfs.support.append configuration parameter.
I set the dfs.support.append to true in hdfs-site.xml. I restarted hadoop
(stop-all & start-all) after this. Even after this, I am getting the above
exception.
Where am I going wrong?
Thank you for your suggestions.
Regards,
Raghava.
Re: appending to files on HDFS
Posted by Raghava Mutharaju <m....@gmail.com>.
Hi Eric,
Thank you for your suggestions. I would do that and check it out.
Regards,
Raghava.
On Tue, Feb 16, 2010 at 8:00 PM, E. Sammer <er...@lifeless.net> wrote:
> On 2/16/10 7:54 PM, Raghava Mutharaju wrote:
>
>> Hi all,
>>
>> I am using hadoop 0.20.1 version. I am trying to append some content
>> from a file on HDFS to another file on HDFS. I am using
>> FileSystem.append()
>> method to get the stream and then passed it to IOUtils.copyBytes() method.
>> I
>> got the following exception
>>
>>
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Append to hdfs
>> not supported. Please refer to dfs.support.append configuration parameter.
>>
>>
>> I set the dfs.support.append to true in hdfs-site.xml. I restarted hadoop
>> (stop-all& start-all) after this. Even after this, I am getting the above
>> exception.
>>
>> Where am I going wrong?
>>
>> Thank you for your suggestions.
>>
>> Regards,
>> Raghava.
>>
>>
> Raghava:
>
> In short, appends in HDFS are extremely experimental and dangerous. Most
> would advise you to leave this disabled. Your best option for "append" like
> behavior is to rewrite the file with new content being added at the end.
> Append support was briefly introduced and then removed as a number of issues
> came up. I believe the open (parent) JIRA issue tracking this is:
>
> http://issues.apache.org/jira/browse/HDFS-265
>
> Hope this helps.
> --
> Eric Sammer
> eric@lifeless.net
> http://esammer.blogspot.com
>
Re: appending to files on HDFS
Posted by "E. Sammer" <er...@lifeless.net>.
On 2/16/10 7:54 PM, Raghava Mutharaju wrote:
> Hi all,
>
> I am using hadoop 0.20.1 version. I am trying to append some content
> from a file on HDFS to another file on HDFS. I am using FileSystem.append()
> method to get the stream and then passed it to IOUtils.copyBytes() method. I
> got the following exception
>
>
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Append to hdfs
> not supported. Please refer to dfs.support.append configuration parameter.
>
>
> I set the dfs.support.append to true in hdfs-site.xml. I restarted hadoop
> (stop-all& start-all) after this. Even after this, I am getting the above
> exception.
>
> Where am I going wrong?
>
> Thank you for your suggestions.
>
> Regards,
> Raghava.
>
Raghava:
In short, appends in HDFS are extremely experimental and dangerous. Most
would advise you to leave this disabled. Your best option for "append"
like behavior is to rewrite the file with new content being added at the
end. Append support was briefly introduced and then removed as a number
of issues came up. I believe the open (parent) JIRA issue tracking this is:
http://issues.apache.org/jira/browse/HDFS-265
Hope this helps.
--
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com