FsDataOutputStream: Can't see data in HDFS until after calling close()

[JGibbs <ja...@gmail.com>]

Hi everyone, I know this is essentially a repost, but that topic was 3-4
years ago, and they were referencing incoming changes that I'm thinking has
already happened. or the links that they were linking to are dead. Most of
what everyone was saying is that sync is not completed yet or is buggy(sync
is now deprecated).

I'm trying to write an avro file to HDFS, the problem is, I can only see it
on HUE after calling close() and not hflush() or hsync(). Is there a way I
can see the data in HDFS, or read from it before calling close()?

Here's an example of some of the code I'm using:


Path path = new Path("/testing/test.avro");
//TODO add check to see if it's there before overwriting it

FSDataOutputStream out = fileSystem.create(path);
this.dataFileWriter = new DataFileWriter<RH>(userDatumWriter);
dataFileWriter.create(schema, out);

this.dataFileWriter.flush();
this.out.flush();
this.out.hsync();
this.out.hflush(); // shows up as empty no matter which of these are called

this.out.close();//finally it shows up as having data

Thanks,
Jake



--
View this message in context: http://hadoop.6.n7.nabble.com/FsDataOutputStream-Can-t-see-data-in-HDFS-until-after-calling-close-tp69593.html
Sent from the common-user mailing list archive at Nabble.com.