You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Jean-Daniel Cryans <jd...@apache.org> on 2010/01/12 01:42:28 UTC
Reading while appending in 0.21
Hi,
I'm trying to use the new hflush function from 0.21 so that a
SequenceFile.Reader could read edits from a SequenceFile.Writer after
a signal on a Condition. If I:
create the Writer
append entries
hflush
create the Reader
next() through the entries
It works fine. But after that if I only next(), using the same reader,
after appending/hflush I won't see the new edits. But, if I create a
new Reader after calling hflush, it works fine.
So this does not work:
create the Writer
append entries
hflush
create the Reader
next() through the entries
append entries
hflush
next() through the entries
append entries
hflush
next() through the entries
This does:
create the Writer
append entries
hflush
create the Reader
next() through the entries
append entries
hflush
create a new Reader
seek
next() through the entries
Is that the intended behavior?
Thx!
J-D
Re: Reading while appending in 0.21
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Hi Hairong,
I now understand that this is outside of the scope of 265 (which works great).
What I'm thinking is that tailing seems achievable now that we have
hflush, tho I'm saying this without much knowledge of the internals.
J-D
On Mon, Jan 11, 2010 at 5:42 PM, Hairong Kuang <ku...@gmail.com> wrote:
> This semantics is stated in our append specification attached to HDFS-265.
> Hflushed data are visible to new readers who open the file after the hflush.
>
> Hairong
>
> On Mon, Jan 11, 2010 at 5:33 PM, Jean-Daniel Cryans <jd...@apache.org>
> wrote:
>>
>> Thanks for the answer Konstantin, good to know. I also see that the
>> commenter doesn't seem pleased with the fact that the file has to be
>> reopened ;)
>>
>> So basically what I wrongly expected was a tailing feature. Searches
>> in Jira don't give me a hit, would it make sense to open one?
>>
>> In the mean time, I'm ok with recreating the reader.
>>
>> Thx,
>>
>> J-D
>>
>> On Mon, Jan 11, 2010 at 5:15 PM, Konstantin Boudnik <co...@yahoo-inc.com>
>> wrote:
>> > Jean,
>> >
>> > I believe this is how it has been intended to work. If you take a look
>> > at
>> > the test src/test/hdfs/org/apache/hadoop/hdfs/TestHFlush.java in HDFS
>> > workspace you'll a comment on this particular aspect in the line 124.
>> >
>> > Hope it helps,
>> > Cos
>> >
>> > On 1/11/10 16:42 , Jean-Daniel Cryans wrote:
>> >>
>> >> Hi,
>> >>
>> >> I'm trying to use the new hflush function from 0.21 so that a
>> >> SequenceFile.Reader could read edits from a SequenceFile.Writer after
>> >> a signal on a Condition. If I:
>> >>
>> >> create the Writer
>> >> append entries
>> >> hflush
>> >> create the Reader
>> >> next() through the entries
>> >>
>> >> It works fine. But after that if I only next(), using the same reader,
>> >> after appending/hflush I won't see the new edits. But, if I create a
>> >> new Reader after calling hflush, it works fine.
>> >>
>> >> So this does not work:
>> >>
>> >> create the Writer
>> >> append entries
>> >> hflush
>> >> create the Reader
>> >> next() through the entries
>> >> append entries
>> >> hflush
>> >> next() through the entries
>> >> append entries
>> >> hflush
>> >> next() through the entries
>> >>
>> >> This does:
>> >>
>> >> create the Writer
>> >> append entries
>> >> hflush
>> >> create the Reader
>> >> next() through the entries
>> >> append entries
>> >> hflush
>> >> create a new Reader
>> >> seek
>> >> next() through the entries
>> >>
>> >> Is that the intended behavior?
>> >>
>> >> Thx!
>> >>
>> >> J-D
>> >
>
>
Re: Reading while appending in 0.21
Posted by Hairong Kuang <ku...@gmail.com>.
This semantics is stated in our append specification attached to HDFS-265.
Hflushed data are visible to new readers who open the file after the hflush.
Hairong
On Mon, Jan 11, 2010 at 5:33 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Thanks for the answer Konstantin, good to know. I also see that the
> commenter doesn't seem pleased with the fact that the file has to be
> reopened ;)
>
> So basically what I wrongly expected was a tailing feature. Searches
> in Jira don't give me a hit, would it make sense to open one?
>
> In the mean time, I'm ok with recreating the reader.
>
> Thx,
>
> J-D
>
> On Mon, Jan 11, 2010 at 5:15 PM, Konstantin Boudnik <co...@yahoo-inc.com>
> wrote:
> > Jean,
> >
> > I believe this is how it has been intended to work. If you take a look at
> > the test src/test/hdfs/org/apache/hadoop/hdfs/TestHFlush.java in HDFS
> > workspace you'll a comment on this particular aspect in the line 124.
> >
> > Hope it helps,
> > Cos
> >
> > On 1/11/10 16:42 , Jean-Daniel Cryans wrote:
> >>
> >> Hi,
> >>
> >> I'm trying to use the new hflush function from 0.21 so that a
> >> SequenceFile.Reader could read edits from a SequenceFile.Writer after
> >> a signal on a Condition. If I:
> >>
> >> create the Writer
> >> append entries
> >> hflush
> >> create the Reader
> >> next() through the entries
> >>
> >> It works fine. But after that if I only next(), using the same reader,
> >> after appending/hflush I won't see the new edits. But, if I create a
> >> new Reader after calling hflush, it works fine.
> >>
> >> So this does not work:
> >>
> >> create the Writer
> >> append entries
> >> hflush
> >> create the Reader
> >> next() through the entries
> >> append entries
> >> hflush
> >> next() through the entries
> >> append entries
> >> hflush
> >> next() through the entries
> >>
> >> This does:
> >>
> >> create the Writer
> >> append entries
> >> hflush
> >> create the Reader
> >> next() through the entries
> >> append entries
> >> hflush
> >> create a new Reader
> >> seek
> >> next() through the entries
> >>
> >> Is that the intended behavior?
> >>
> >> Thx!
> >>
> >> J-D
> >
>
Re: Reading while appending in 0.21
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Thanks for the answer Konstantin, good to know. I also see that the
commenter doesn't seem pleased with the fact that the file has to be
reopened ;)
So basically what I wrongly expected was a tailing feature. Searches
in Jira don't give me a hit, would it make sense to open one?
In the mean time, I'm ok with recreating the reader.
Thx,
J-D
On Mon, Jan 11, 2010 at 5:15 PM, Konstantin Boudnik <co...@yahoo-inc.com> wrote:
> Jean,
>
> I believe this is how it has been intended to work. If you take a look at
> the test src/test/hdfs/org/apache/hadoop/hdfs/TestHFlush.java in HDFS
> workspace you'll a comment on this particular aspect in the line 124.
>
> Hope it helps,
> Cos
>
> On 1/11/10 16:42 , Jean-Daniel Cryans wrote:
>>
>> Hi,
>>
>> I'm trying to use the new hflush function from 0.21 so that a
>> SequenceFile.Reader could read edits from a SequenceFile.Writer after
>> a signal on a Condition. If I:
>>
>> create the Writer
>> append entries
>> hflush
>> create the Reader
>> next() through the entries
>>
>> It works fine. But after that if I only next(), using the same reader,
>> after appending/hflush I won't see the new edits. But, if I create a
>> new Reader after calling hflush, it works fine.
>>
>> So this does not work:
>>
>> create the Writer
>> append entries
>> hflush
>> create the Reader
>> next() through the entries
>> append entries
>> hflush
>> next() through the entries
>> append entries
>> hflush
>> next() through the entries
>>
>> This does:
>>
>> create the Writer
>> append entries
>> hflush
>> create the Reader
>> next() through the entries
>> append entries
>> hflush
>> create a new Reader
>> seek
>> next() through the entries
>>
>> Is that the intended behavior?
>>
>> Thx!
>>
>> J-D
>
Re: Reading while appending in 0.21
Posted by Konstantin Boudnik <co...@yahoo-inc.com>.
Jean,
I believe this is how it has been intended to work. If you take a look at the
test src/test/hdfs/org/apache/hadoop/hdfs/TestHFlush.java in HDFS workspace
you'll a comment on this particular aspect in the line 124.
Hope it helps,
Cos
On 1/11/10 16:42 , Jean-Daniel Cryans wrote:
> Hi,
>
> I'm trying to use the new hflush function from 0.21 so that a
> SequenceFile.Reader could read edits from a SequenceFile.Writer after
> a signal on a Condition. If I:
>
> create the Writer
> append entries
> hflush
> create the Reader
> next() through the entries
>
> It works fine. But after that if I only next(), using the same reader,
> after appending/hflush I won't see the new edits. But, if I create a
> new Reader after calling hflush, it works fine.
>
> So this does not work:
>
> create the Writer
> append entries
> hflush
> create the Reader
> next() through the entries
> append entries
> hflush
> next() through the entries
> append entries
> hflush
> next() through the entries
>
> This does:
>
> create the Writer
> append entries
> hflush
> create the Reader
> next() through the entries
> append entries
> hflush
> create a new Reader
> seek
> next() through the entries
>
> Is that the intended behavior?
>
> Thx!
>
> J-D