You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Tony Burton <TB...@SportingIndex.com> on 2013/03/18 16:44:53 UTC
hadoop file append
Hi list,
I'm using Hadoop 1.0.3 for a MapReduce task and I thought it might be a simple job to append a Counter value and some text to the end of a file (which ultimately will be in AWS S3). How wrong I was :)
I've been reading about o.a.h.fs.FileSystem.append and whether it does or doesn't work in my version of hadoop, and I'm not sure if it does or not. Can anyone say for definite either way? I'm going with the conclusion that it can't be trusted to perform as expected.
What I'd like to do is this: Perform a daily MapReduce job and add up some some values along the way, and as well as the output of the MR job, add that day's date with the Counter value to the end of a file (in a different location), so the file grows by one line daily. Assuming o.a.h.fs.FileSystem.append is currently out of favour, do you have any suggestions for achieving this?
Thanks!
Tony
**********************************************************************
Please consider the environment before printing this email or attachments
This email and any attachments are confidential, protected by copyright and may be legally privileged. If you are not the intended recipient, then the dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Neither Sporting Index nor the sender accepts responsibility for any virus, or any other defect which might affect any computer or IT system into which the email is received and/or opened. It is the responsibility of the recipient to scan the email and no responsibility is accepted for any loss or damage arising in any way from receipt or use of this email. Sporting Index Ltd is a company registered in England and Wales with company number 2636842, whose registered office is at Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is authorised and regulated by the UK Financial Services Authority (reg. no. 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any financial promotion contained herein has been issued
and approved by Sporting Index Ltd.
Outbound email has been scanned for viruses and SPAM
Re: hadoop file append
Posted by Harsh J <ha...@cloudera.com>.
Appending on 1.x releases is available but not tested/supported and
can be toggled to be disabled completely. Appending works better on
2.x releases.
On Mon, Mar 18, 2013 at 9:14 PM, Tony Burton <TB...@sportingindex.com> wrote:
> Hi list,
>
>
>
> I’m using Hadoop 1.0.3 for a MapReduce task and I thought it might be a
> simple job to append a Counter value and some text to the end of a file
> (which ultimately will be in AWS S3). How wrong I was :)
>
>
>
> I’ve been reading about o.a.h.fs.FileSystem.append and whether it does or
> doesn’t work in my version of hadoop, and I'm not sure if it does or not.
> Can anyone say for definite either way? I'm going with the conclusion that
> it can't be trusted to perform as expected.
>
>
>
> What I'd like to do is this: Perform a daily MapReduce job and add up some
> some values along the way, and as well as the output of the MR job, add that
> day's date with the Counter value to the end of a file (in a different
> location), so the file grows by one line daily. Assuming
> o.a.h.fs.FileSystem.append is currently out of favour, do you have any
> suggestions for achieving this?
>
>
>
> Thanks!
>
>
>
> Tony
>
>
>
>
> *****************************************************************************
> P Please consider the environment before printing this email or attachments
>
>
> This email and any attachments are confidential, protected by copyright and
> may be legally privileged. If you are not the intended recipient, then the
> dissemination or copying of this email is prohibited. If you have received
> this in error, please notify the sender by replying by email and then delete
> the email completely from your system. Neither Sporting Index nor the sender
> accepts responsibility for any virus, or any other defect which might affect
> any computer or IT system into which the email is received and/or opened. It
> is the responsibility of the recipient to scan the email and no
> responsibility is accepted for any loss or damage arising in any way from
> receipt or use of this email. Sporting Index Ltd is a company registered in
> England and Wales with company number 2636842, whose registered office is at
> Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is
> authorised and regulated by the UK Financial Services Authority (reg. no.
> 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any
> financial promotion contained herein has been issued and approved by
> Sporting Index Ltd.
>
>
> Outbound email has been scanned for viruses and SPAM
--
Harsh J
Re: hadoop file append
Posted by Harsh J <ha...@cloudera.com>.
Appending on 1.x releases is available but not tested/supported and
can be toggled to be disabled completely. Appending works better on
2.x releases.
On Mon, Mar 18, 2013 at 9:14 PM, Tony Burton <TB...@sportingindex.com> wrote:
> Hi list,
>
>
>
> I’m using Hadoop 1.0.3 for a MapReduce task and I thought it might be a
> simple job to append a Counter value and some text to the end of a file
> (which ultimately will be in AWS S3). How wrong I was :)
>
>
>
> I’ve been reading about o.a.h.fs.FileSystem.append and whether it does or
> doesn’t work in my version of hadoop, and I'm not sure if it does or not.
> Can anyone say for definite either way? I'm going with the conclusion that
> it can't be trusted to perform as expected.
>
>
>
> What I'd like to do is this: Perform a daily MapReduce job and add up some
> some values along the way, and as well as the output of the MR job, add that
> day's date with the Counter value to the end of a file (in a different
> location), so the file grows by one line daily. Assuming
> o.a.h.fs.FileSystem.append is currently out of favour, do you have any
> suggestions for achieving this?
>
>
>
> Thanks!
>
>
>
> Tony
>
>
>
>
> *****************************************************************************
> P Please consider the environment before printing this email or attachments
>
>
> This email and any attachments are confidential, protected by copyright and
> may be legally privileged. If you are not the intended recipient, then the
> dissemination or copying of this email is prohibited. If you have received
> this in error, please notify the sender by replying by email and then delete
> the email completely from your system. Neither Sporting Index nor the sender
> accepts responsibility for any virus, or any other defect which might affect
> any computer or IT system into which the email is received and/or opened. It
> is the responsibility of the recipient to scan the email and no
> responsibility is accepted for any loss or damage arising in any way from
> receipt or use of this email. Sporting Index Ltd is a company registered in
> England and Wales with company number 2636842, whose registered office is at
> Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is
> authorised and regulated by the UK Financial Services Authority (reg. no.
> 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any
> financial promotion contained herein has been issued and approved by
> Sporting Index Ltd.
>
>
> Outbound email has been scanned for viruses and SPAM
--
Harsh J
Re: hadoop file append
Posted by Harsh J <ha...@cloudera.com>.
Appending on 1.x releases is available but not tested/supported and
can be toggled to be disabled completely. Appending works better on
2.x releases.
On Mon, Mar 18, 2013 at 9:14 PM, Tony Burton <TB...@sportingindex.com> wrote:
> Hi list,
>
>
>
> I’m using Hadoop 1.0.3 for a MapReduce task and I thought it might be a
> simple job to append a Counter value and some text to the end of a file
> (which ultimately will be in AWS S3). How wrong I was :)
>
>
>
> I’ve been reading about o.a.h.fs.FileSystem.append and whether it does or
> doesn’t work in my version of hadoop, and I'm not sure if it does or not.
> Can anyone say for definite either way? I'm going with the conclusion that
> it can't be trusted to perform as expected.
>
>
>
> What I'd like to do is this: Perform a daily MapReduce job and add up some
> some values along the way, and as well as the output of the MR job, add that
> day's date with the Counter value to the end of a file (in a different
> location), so the file grows by one line daily. Assuming
> o.a.h.fs.FileSystem.append is currently out of favour, do you have any
> suggestions for achieving this?
>
>
>
> Thanks!
>
>
>
> Tony
>
>
>
>
> *****************************************************************************
> P Please consider the environment before printing this email or attachments
>
>
> This email and any attachments are confidential, protected by copyright and
> may be legally privileged. If you are not the intended recipient, then the
> dissemination or copying of this email is prohibited. If you have received
> this in error, please notify the sender by replying by email and then delete
> the email completely from your system. Neither Sporting Index nor the sender
> accepts responsibility for any virus, or any other defect which might affect
> any computer or IT system into which the email is received and/or opened. It
> is the responsibility of the recipient to scan the email and no
> responsibility is accepted for any loss or damage arising in any way from
> receipt or use of this email. Sporting Index Ltd is a company registered in
> England and Wales with company number 2636842, whose registered office is at
> Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is
> authorised and regulated by the UK Financial Services Authority (reg. no.
> 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any
> financial promotion contained herein has been issued and approved by
> Sporting Index Ltd.
>
>
> Outbound email has been scanned for viruses and SPAM
--
Harsh J
Re: hadoop file append
Posted by Harsh J <ha...@cloudera.com>.
Appending on 1.x releases is available but not tested/supported and
can be toggled to be disabled completely. Appending works better on
2.x releases.
On Mon, Mar 18, 2013 at 9:14 PM, Tony Burton <TB...@sportingindex.com> wrote:
> Hi list,
>
>
>
> I’m using Hadoop 1.0.3 for a MapReduce task and I thought it might be a
> simple job to append a Counter value and some text to the end of a file
> (which ultimately will be in AWS S3). How wrong I was :)
>
>
>
> I’ve been reading about o.a.h.fs.FileSystem.append and whether it does or
> doesn’t work in my version of hadoop, and I'm not sure if it does or not.
> Can anyone say for definite either way? I'm going with the conclusion that
> it can't be trusted to perform as expected.
>
>
>
> What I'd like to do is this: Perform a daily MapReduce job and add up some
> some values along the way, and as well as the output of the MR job, add that
> day's date with the Counter value to the end of a file (in a different
> location), so the file grows by one line daily. Assuming
> o.a.h.fs.FileSystem.append is currently out of favour, do you have any
> suggestions for achieving this?
>
>
>
> Thanks!
>
>
>
> Tony
>
>
>
>
> *****************************************************************************
> P Please consider the environment before printing this email or attachments
>
>
> This email and any attachments are confidential, protected by copyright and
> may be legally privileged. If you are not the intended recipient, then the
> dissemination or copying of this email is prohibited. If you have received
> this in error, please notify the sender by replying by email and then delete
> the email completely from your system. Neither Sporting Index nor the sender
> accepts responsibility for any virus, or any other defect which might affect
> any computer or IT system into which the email is received and/or opened. It
> is the responsibility of the recipient to scan the email and no
> responsibility is accepted for any loss or damage arising in any way from
> receipt or use of this email. Sporting Index Ltd is a company registered in
> England and Wales with company number 2636842, whose registered office is at
> Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is
> authorised and regulated by the UK Financial Services Authority (reg. no.
> 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any
> financial promotion contained herein has been issued and approved by
> Sporting Index Ltd.
>
>
> Outbound email has been scanned for viruses and SPAM
--
Harsh J