You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Jan Stöcker <Ja...@q2web.de> on 2010/06/15 14:31:29 UTC
HDFS file append
Hello,
in the hdfs-default.xml, I found the warning that the property
"dfs.support.append" is false by default, because of the "append
code" containing bugs.
For our test case (which is not very complicated), I tried it out
nevertheless (in a C program using libhdfs), and I did not
encounter any problems yet. The only thing I noticed is that the
modification date of the file to which I append is not changed.
Could someone indicate which other bugs may occur? And in
which cases? Because the append could be quite useful for me,
and I would like to know the risk.
Thanks in advance
Jan
AW: HDFS file append
Posted by Jan Stöcker <Ja...@q2web.de>.
O.k., thanks for the information & the suggestions!
Jan
-----Ursprüngliche Nachricht-----
Von: Todd Lipcon [mailto:todd@cloudera.com]
Gesendet: Dienstag, 15. Juni 2010 18:55
An: general@hadoop.apache.org
Betreff: Re: HDFS file append
On Tue, Jun 15, 2010 at 9:53 AM, Dhruba Borthakur <dh...@gmail.com> wrote:
> You can start downloading and using the code form the hadoop-0.20-append.
>
>
Though it's still missing some patches, so I'd recommend SVN checkouts and
watching the branch :)
-Todd
> thanks,
> dhruba
>
> On Tue, Jun 15, 2010 at 8:24 AM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > On Tue, Jun 15, 2010 at 5:31 AM, Jan Stöcker <Ja...@q2web.de>
> > wrote:
> >
> > > Hello,
> > >
> > >
> > >
> > > in the hdfs-default.xml, I found the warning that the property
> > >
> > > "dfs.support.append" is false by default, because of the "append
> > >
> > > code" containing bugs.
> > >
> > > For our test case (which is not very complicated), I tried it out
> > >
> > > nevertheless (in a C program using libhdfs), and I did not
> > >
> > > encounter any problems yet. The only thing I noticed is that the
> > >
> > > modification date of the file to which I append is not changed.
> > >
> > >
> > >
> > > Could someone indicate which other bugs may occur? And in
> > >
> > > which cases? Because the append could be quite useful for me,
> > >
> > > and I would like to know the risk.
> > >
> >
> > Mostly in failure handling cases, but not entirely. The result is usually
> > truncated files, sometimes truncated as if your append didn't happen,
> > sometimes the entire last block going missing. Look for the 0.20-append
> fix
> > version on JIRA for a more significantly list.
> >
> > I'd recommend waiting a couple weeks and using the hadoop-0.20-append
> > branch
> > which will contain fixes for many of these bugs.
> >
> > -Todd
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: HDFS file append
Posted by Todd Lipcon <to...@cloudera.com>.
On Tue, Jun 15, 2010 at 9:53 AM, Dhruba Borthakur <dh...@gmail.com> wrote:
> You can start downloading and using the code form the hadoop-0.20-append.
>
>
Though it's still missing some patches, so I'd recommend SVN checkouts and
watching the branch :)
-Todd
> thanks,
> dhruba
>
> On Tue, Jun 15, 2010 at 8:24 AM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > On Tue, Jun 15, 2010 at 5:31 AM, Jan Stöcker <Ja...@q2web.de>
> > wrote:
> >
> > > Hello,
> > >
> > >
> > >
> > > in the hdfs-default.xml, I found the warning that the property
> > >
> > > "dfs.support.append" is false by default, because of the "append
> > >
> > > code" containing bugs.
> > >
> > > For our test case (which is not very complicated), I tried it out
> > >
> > > nevertheless (in a C program using libhdfs), and I did not
> > >
> > > encounter any problems yet. The only thing I noticed is that the
> > >
> > > modification date of the file to which I append is not changed.
> > >
> > >
> > >
> > > Could someone indicate which other bugs may occur? And in
> > >
> > > which cases? Because the append could be quite useful for me,
> > >
> > > and I would like to know the risk.
> > >
> >
> > Mostly in failure handling cases, but not entirely. The result is usually
> > truncated files, sometimes truncated as if your append didn't happen,
> > sometimes the entire last block going missing. Look for the 0.20-append
> fix
> > version on JIRA for a more significantly list.
> >
> > I'd recommend waiting a couple weeks and using the hadoop-0.20-append
> > branch
> > which will contain fixes for many of these bugs.
> >
> > -Todd
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: HDFS file append
Posted by Dhruba Borthakur <dh...@gmail.com>.
You can start downloading and using the code form the hadoop-0.20-append.
thanks,
dhruba
On Tue, Jun 15, 2010 at 8:24 AM, Todd Lipcon <to...@cloudera.com> wrote:
> On Tue, Jun 15, 2010 at 5:31 AM, Jan Stöcker <Ja...@q2web.de>
> wrote:
>
> > Hello,
> >
> >
> >
> > in the hdfs-default.xml, I found the warning that the property
> >
> > "dfs.support.append" is false by default, because of the "append
> >
> > code" containing bugs.
> >
> > For our test case (which is not very complicated), I tried it out
> >
> > nevertheless (in a C program using libhdfs), and I did not
> >
> > encounter any problems yet. The only thing I noticed is that the
> >
> > modification date of the file to which I append is not changed.
> >
> >
> >
> > Could someone indicate which other bugs may occur? And in
> >
> > which cases? Because the append could be quite useful for me,
> >
> > and I would like to know the risk.
> >
>
> Mostly in failure handling cases, but not entirely. The result is usually
> truncated files, sometimes truncated as if your append didn't happen,
> sometimes the entire last block going missing. Look for the 0.20-append fix
> version on JIRA for a more significantly list.
>
> I'd recommend waiting a couple weeks and using the hadoop-0.20-append
> branch
> which will contain fixes for many of these bugs.
>
> -Todd
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
Connect to me at http://www.facebook.com/dhruba
Re: HDFS file append
Posted by Todd Lipcon <to...@cloudera.com>.
On Tue, Jun 15, 2010 at 5:31 AM, Jan Stöcker <Ja...@q2web.de> wrote:
> Hello,
>
>
>
> in the hdfs-default.xml, I found the warning that the property
>
> "dfs.support.append" is false by default, because of the "append
>
> code" containing bugs.
>
> For our test case (which is not very complicated), I tried it out
>
> nevertheless (in a C program using libhdfs), and I did not
>
> encounter any problems yet. The only thing I noticed is that the
>
> modification date of the file to which I append is not changed.
>
>
>
> Could someone indicate which other bugs may occur? And in
>
> which cases? Because the append could be quite useful for me,
>
> and I would like to know the risk.
>
Mostly in failure handling cases, but not entirely. The result is usually
truncated files, sometimes truncated as if your append didn't happen,
sometimes the entire last block going missing. Look for the 0.20-append fix
version on JIRA for a more significantly list.
I'd recommend waiting a couple weeks and using the hadoop-0.20-append branch
which will contain fixes for many of these bugs.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera