Zero file size after hsync

[Viliam Holub <vi...@ucd.ie>]

Hi all,

size of a newly created file is reported to be zero even though I've written
some data and hsync-ed them. Is that correct and expected effect?
hadoop fs -cat will retrieve the data correctly.

As a consequence SequenceFile fails to seek in the file since it tests the
position against file size. And data are there...

Thanks!
Viliam

Re: Zero file size after hsync

[Viliam Holub <vi...@ucd.ie>]

Hi Nicholas,

it's 0.21.0.

In nutshell, one process collects measured data and saves them in a MapFile.
It's low bandwidth - about 1 event/sec - and may run for days. I need recent
data available for (independent) processing.

One option is to close the MapFile. But then I have to open a new one since
MapFile does not enable appending (unless I'm missing something). And then I
end up with a lot of small files.

I was hoping that calling hsync on the underlying data and index files say
every 15 secs would to the trick. Apparently data are copied, but
SequenceFile uses file size (reported as 0) for seek checks and therefore
fails.

getVisibleLength() is a great tip! SequenceFile tweaked to use it appears to
be working.

Thanks for help,
Viliam

On 18. Mar (Friday) v 09:55:15 -0700 2011, Tsz Wo (Nicholas), Sze wrote:
> Hi Viliam,
> 
> Which version of Hadoop are you using?
> 
> First of all, hsyn is the same as hflush in 0.21 and above.  hflush/hsync won't 
> update the file length on the NameNode.  So the answer to your question is yes.  
> We have to call DFSDataInputStream.getVisibleLength() to get the visible length 
> of the file.
> 
> When is the SequenceFile opened?  Before or after hflush/hsync?  Note that only 
> new reader can see the new data.  So if the file, normal file or SequenceFile, 
> is opened before hflush/hsync, we have to re-open the file in order to see the 
> new data.
> 
> Anyway, please feel free to file a JIRA if you feel it is a bug or you like to 
> have a feature request.
> 
> Hope it helps.
> Nicholas
> 
> 
> 
> 
> ________________________________
> From: Viliam Holub <vi...@ucd.ie>
> To: hdfs-user@hadoop.apache.org
> Sent: Fri, March 18, 2011 9:29:32 AM
> Subject: Zero file size after hsync
> 
> 
> Hi all,
> 
> size of a newly created file is reported to be zero even though I've written
> some data and hsync-ed them. Is that correct and expected effect?
> hadoop fs -cat will retrieve the data correctly.
> 
> As a consequence SequenceFile fails to seek in the file since it tests the
> position against file size. And data are there...
> 
> Thanks!
> Viliam

Re: Zero file size after hsync

["Tsz Wo (Nicholas), Sze" <s2...@yahoo.com>]

Hi Viliam,

Which version of Hadoop are you using?

First of all, hsyn is the same as hflush in 0.21 and above.  hflush/hsync won't 
update the file length on the NameNode.  So the answer to your question is yes.  
We have to call DFSDataInputStream.getVisibleLength() to get the visible length 
of the file.

When is the SequenceFile opened?  Before or after hflush/hsync?  Note that only 
new reader can see the new data.  So if the file, normal file or SequenceFile, 
is opened before hflush/hsync, we have to re-open the file in order to see the 
new data.

Anyway, please feel free to file a JIRA if you feel it is a bug or you like to 
have a feature request.

Hope it helps.
Nicholas




________________________________
From: Viliam Holub <vi...@ucd.ie>
To: hdfs-user@hadoop.apache.org
Sent: Fri, March 18, 2011 9:29:32 AM
Subject: Zero file size after hsync


Hi all,

size of a newly created file is reported to be zero even though I've written
some data and hsync-ed them. Is that correct and expected effect?
hadoop fs -cat will retrieve the data correctly.

As a consequence SequenceFile fails to seek in the file since it tests the
position against file size. And data are there...

Thanks!
Viliam