Read sequence files that are being written

[Guillaume Perrot <gp...@ubikod.com>]

Hi everyone,

I am using Hadoop 1.0.3.

I write logs to an Hadoop sequence file into HDFS, I call syncFS() after
each bunch of logs but I never close the file (except when I am performing
daily rolling).

What I want to guarantee is that the file is available to readers while the
file is still being written.

I can read the bytes of the sequence file via FSDataInputStream, but if I
try to use SequenceFile.Reader.next(key,val), it returns false at the first
call.

I know the data is in the file since I can read it with FSDataInputStream
or with the cat command and I am 100% sure that syncFS() is called.

I checked the namenode and datanode logs, no error or warning. fsck shows
no corruption.

Why SequenceFile.Reader is unable to read my currently being written file ?