You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Stas Oskin <st...@gmail.com> on 2010/06/10 14:05:50 UTC
Appending and seeking files while writing
Hi.
Was the append functionality finally added to 0.20.1 version?
Also, is the ability to seek file being written and write data in other
place also supported?
Thanks in advance!
Re: Appending and seeking files while writing
Posted by Stas Oskin <st...@gmail.com>.
Actually, I noticed FUSE finally being supported in 0.20.1, any idea if this
supports RDWR mode for both writing and reading file at same time?
Regards.
On Thu, Jun 10, 2010 at 3:05 PM, Stas Oskin <st...@gmail.com> wrote:
> Hi.
>
> Was the append functionality finally added to 0.20.1 version?
>
> Also, is the ability to seek file being written and write data in other
> place also supported?
>
> Thanks in advance!
>
Re: Appending and seeking files while writing
Posted by Vidur Goyal <vi...@students.iiit.ac.in>.
hadooprocks,
As a project requirement I have to do the same and write a seek()
operation for FSDataoutputStream. It will be very kind of you to give me
some insight on this. I have explored the web about recompiling Hadoop
once I change the contents , Can you give me some documents that help me
do that too.
Thanks,
Vidur
> Stas,
>
> I also believe that there should be a seek interface on the write path so
> that the FS API is complete. The FsDataInputStream already support seek()
> -
> so should FsDataOutputStream. For File systems, that do not support the
> seek
> on the write path, the seek can be a no operation.
>
> Could you open a JIRA to track this. I am willing to provide the patch if
> you do not have the time to do so.
>
> thanks
> hadooprocks
>
>
> On Thu, Jun 10, 2010 at 5:05 AM, Stas Oskin <st...@gmail.com> wrote:
>
>> Hi.
>>
>> Was the append functionality finally added to 0.20.1 version?
>>
>> Also, is the ability to seek file being written and write data in other
>> place also supported?
>>
>> Thanks in advance!
>>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
Re: Appending and seeking files while writing
Posted by Todd Lipcon <to...@cloudera.com>.
On Mon, Jun 14, 2010 at 4:28 AM, Stas Oskin <st...@gmail.com> wrote:
> By the way, what about an ability for node to read file which is being
> written by another node?
>
This is allowed, though there are some remaining bugs to be ironed out here.
See https://issues.apache.org/jira/browse/HDFS-1057 for example.
> Or the file must be written and closed completely, before it becomes
> available for other nodes?
>
> (AFAIK in 0.18.3 the file appeared as 0 size until it was closed).
>
> Regards.
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Appending and seeking files while writing
Posted by Stas Oskin <st...@gmail.com>.
By the way, what about an ability for node to read file which is being
written by another node?
Or the file must be written and closed completely, before it becomes
available for other nodes?
(AFAIK in 0.18.3 the file appeared as 0 size until it was closed).
Regards.
Re: Appending and seeking files while writing
Posted by Stas Oskin <st...@gmail.com>.
Hi.
Should be out soon - Tom White is working hard on the release. Note that the
> first release, 0.21.0, will be somewhat of a "development quality" release
> not recommended for production use. Of course, the way it will become
> production-worthy is by less risk-averse people trying it and finding the
> bugs :)
>
>
> > Will it work both with Fuse and HDFS API?
> >
> > I don't know that the Fuse code has been updated to call append. My guess
> is that a small patch would be required.
>
>
> >
> > > Also, append does *not* add random write. It simply adds the ability to
> > > re-open a file and add more data to the end.
> > >
> > >
> > Just to clarify, even with append it won't be possible to:
> > 1) Pause writing of new file, skip to any position, and update the data.
> > 2) Open existing file, skip to any position and update the data.
> >
> > Correct, neither of those are allowed.
>
>
Thanks for clarification.
Re: Appending and seeking files while writing
Posted by Todd Lipcon <to...@cloudera.com>.
On Mon, Jun 14, 2010 at 4:00 AM, Stas Oskin <st...@gmail.com> wrote:
> Hi.
>
> Thanks for clarification.
>
> Append will be supported fully in 0.21.
> >
> >
> Any ETA for this version?
>
Should be out soon - Tom White is working hard on the release. Note that the
first release, 0.21.0, will be somewhat of a "development quality" release
not recommended for production use. Of course, the way it will become
production-worthy is by less risk-averse people trying it and finding the
bugs :)
> Will it work both with Fuse and HDFS API?
>
> I don't know that the Fuse code has been updated to call append. My guess
is that a small patch would be required.
>
> > Also, append does *not* add random write. It simply adds the ability to
> > re-open a file and add more data to the end.
> >
> >
> Just to clarify, even with append it won't be possible to:
> 1) Pause writing of new file, skip to any position, and update the data.
> 2) Open existing file, skip to any position and update the data.
>
> Correct, neither of those are allowed.
> This will be even with FUSE.
>
> Is this correct?
>
> Regards.
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Appending and seeking files while writing
Posted by Stas Oskin <st...@gmail.com>.
Hi.
Thanks for clarification.
Append will be supported fully in 0.21.
>
>
Any ETA for this version?
Will it work both with Fuse and HDFS API?
> Also, append does *not* add random write. It simply adds the ability to
> re-open a file and add more data to the end.
>
>
Just to clarify, even with append it won't be possible to:
1) Pause writing of new file, skip to any position, and update the data.
2) Open existing file, skip to any position and update the data.
This will be even with FUSE.
Is this correct?
Regards.
Re: Appending and seeking files while writing
Posted by Todd Lipcon <to...@cloudera.com>.
On Sun, Jun 13, 2010 at 12:46 AM, Vidur Goyal <vi...@students.iiit.ac.in>wrote:
> Append is supported in hadoop 0.20 .
>
>
Append will be supported in the 0.20-append branch, which is still in
progress. It is NOT supported in vanilla 0.20. You can turn on the config
option but it is dangerous and highly discouraged for real use.
Append will be supported fully in 0.21.
Also, append does *not* add random write. It simply adds the ability to
re-open a file and add more data to the end.
-Todd
>
> > Hi.
> >
> > I think this really depends on the append functionality, any idea whether
> > it
> > supports such behaviour now?
> >
> > Regards.
> >
> > On Fri, Jun 11, 2010 at 10:41 AM, hadooprcoks <ha...@gmail.com>
> > wrote:
> >
> >> Stas,
> >>
> >> I also believe that there should be a seek interface on the write path
> >> so
> >> that the FS API is complete. The FsDataInputStream already support
> >> seek() -
> >> so should FsDataOutputStream. For File systems, that do not support the
> >> seek
> >> on the write path, the seek can be a no operation.
> >>
> >> Could you open a JIRA to track this. I am willing to provide the patch
> >> if
> >> you do not have the time to do so.
> >>
> >> thanks
> >> hadooprocks
> >>
> >>
> >> On Thu, Jun 10, 2010 at 5:05 AM, Stas Oskin <st...@gmail.com>
> >> wrote:
> >>
> >> > Hi.
> >> >
> >> > Was the append functionality finally added to 0.20.1 version?
> >> >
> >> > Also, is the ability to seek file being written and write data in
> >> other
> >> > place also supported?
> >> >
> >> > Thanks in advance!
> >> >
> >>
> >
> > --
> > This message has been scanned for viruses and
> > dangerous content by MailScanner, and is
> > believed to be clean.
> >
> >
>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Appending and seeking files while writing
Posted by Vidur Goyal <vi...@students.iiit.ac.in>.
Append is supported in hadoop 0.20 .
> Hi.
>
> I think this really depends on the append functionality, any idea whether
> it
> supports such behaviour now?
>
> Regards.
>
> On Fri, Jun 11, 2010 at 10:41 AM, hadooprcoks <ha...@gmail.com>
> wrote:
>
>> Stas,
>>
>> I also believe that there should be a seek interface on the write path
>> so
>> that the FS API is complete. The FsDataInputStream already support
>> seek() -
>> so should FsDataOutputStream. For File systems, that do not support the
>> seek
>> on the write path, the seek can be a no operation.
>>
>> Could you open a JIRA to track this. I am willing to provide the patch
>> if
>> you do not have the time to do so.
>>
>> thanks
>> hadooprocks
>>
>>
>> On Thu, Jun 10, 2010 at 5:05 AM, Stas Oskin <st...@gmail.com>
>> wrote:
>>
>> > Hi.
>> >
>> > Was the append functionality finally added to 0.20.1 version?
>> >
>> > Also, is the ability to seek file being written and write data in
>> other
>> > place also supported?
>> >
>> > Thanks in advance!
>> >
>>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
Re: Appending and seeking files while writing
Posted by Stas Oskin <st...@gmail.com>.
Hi.
I think this really depends on the append functionality, any idea whether it
supports such behaviour now?
Regards.
On Fri, Jun 11, 2010 at 10:41 AM, hadooprcoks <ha...@gmail.com> wrote:
> Stas,
>
> I also believe that there should be a seek interface on the write path so
> that the FS API is complete. The FsDataInputStream already support seek() -
> so should FsDataOutputStream. For File systems, that do not support the
> seek
> on the write path, the seek can be a no operation.
>
> Could you open a JIRA to track this. I am willing to provide the patch if
> you do not have the time to do so.
>
> thanks
> hadooprocks
>
>
> On Thu, Jun 10, 2010 at 5:05 AM, Stas Oskin <st...@gmail.com> wrote:
>
> > Hi.
> >
> > Was the append functionality finally added to 0.20.1 version?
> >
> > Also, is the ability to seek file being written and write data in other
> > place also supported?
> >
> > Thanks in advance!
> >
>
Re: Appending and seeking files while writing
Posted by hadooprcoks <ha...@gmail.com>.
Stas,
I also believe that there should be a seek interface on the write path so
that the FS API is complete. The FsDataInputStream already support seek() -
so should FsDataOutputStream. For File systems, that do not support the seek
on the write path, the seek can be a no operation.
Could you open a JIRA to track this. I am willing to provide the patch if
you do not have the time to do so.
thanks
hadooprocks
On Thu, Jun 10, 2010 at 5:05 AM, Stas Oskin <st...@gmail.com> wrote:
> Hi.
>
> Was the append functionality finally added to 0.20.1 version?
>
> Also, is the ability to seek file being written and write data in other
> place also supported?
>
> Thanks in advance!
>