a file can be used as a queue?

[ltomuno <lt...@163.com>]

I heard a HDFS file as a producer - consumer queue, a file can be used as a queue? I am very confused

Re:Re: a file can be used as a queue?

[ltomuno <lt...@163.com>]

thank you very much
I see




At 2011-06-13 19:23:59，"Joey Echeverria" <jo...@cloudera.com> wrote:

>This feature doesn't currently work. I don't remember the JIRA for it, but there's a ticket which will allow a reader to read from an HDFS file before it's closed. In that case, you implement a queue by having the producer write to the end of the file and the reader read from the beginning of the file.
>
>I'm not sure if there will be a way to tell that a file is still being written, so you may need your own end of stream marker.
>
>-Joey
>
>On Jun 13, 2011, at 2:55, ltomuno <lt...@163.com> wrote:
>
>> I heard a HDFS file as a producer - consumer queue, a file can be used as a queue? I am very confused

Re: a file can be used as a queue?

[John George <jo...@yahoo-inc.com>]

On 6/13/11 6:23 AM, "Joey Echeverria" <jo...@cloudera.com> wrote:

> This feature doesn't currently work. I don't remember the JIRA for it, but
> there's a ticket which will allow a reader to read from an HDFS file before
> it's closed. In that case, you implement a queue by having the producer write
> to the end of the file and the reader read from the beginning of the file.
> 
> I'm not sure if there will be a way to tell that a file is still being
> written, so you may need your own end of stream marker.

One way to know the end of stream would be to call getVisibleLength() on the
input stream. As long as the writer has flushed (or closed) its stream, the
reader should be able to see those bytes. TestWriteRead.java might provide
you some clues  
(hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestWriteRead.java).

> 
> -Joey
> 
> On Jun 13, 2011, at 2:55, ltomuno <lt...@163.com> wrote:
> 
>> I heard a HDFS file as a producer - consumer queue, a file can be used as a
>> queue? I am very confused

Regards,
John George

Re: a file can be used as a queue?

[Joey Echeverria <jo...@cloudera.com>]

This feature doesn't currently work. I don't remember the JIRA for it, but there's a ticket which will allow a reader to read from an HDFS file before it's closed. In that case, you implement a queue by having the producer write to the end of the file and the reader read from the beginning of the file.

I'm not sure if there will be a way to tell that a file is still being written, so you may need your own end of stream marker.

-Joey

On Jun 13, 2011, at 2:55, ltomuno <lt...@163.com> wrote:

> I heard a HDFS file as a producer - consumer queue, a file can be used as a queue? I am very confused