How can I syncronize writing to an hdfs file

[Raymond Jennings III <ra...@yahoo.com>]

I want to write to a common hdfs file from within my map method.  Given that each task runs in a separate jvm (on separate machines) making a method syncronized will not work I assume.  Are there any file locking or other methods to guarantee mutual exclusion on hdfs?

(I want to append to this file and I have the append option turned on.)  Thanks.


      

RE: How can I syncronize writing to an hdfs file

[Bill Habermaas <bi...@habermaas.us>]

I had a similar requirement. Hdfs has no locking that I am aware of, at
least I have never run across it in reading the source. My solution was to
build a distributed locking mechanism using ZooKeeper.  You might want to
visit http://hadoop.apache.org/zookeeper/docs/current/recipes.html
For some ideas.  The code you find there is a start but buggy. 

Bill 

-----Original Message-----
From: Raymond Jennings III [mailto:raymondjiii@yahoo.com] 
Sent: Friday, May 07, 2010 10:32 AM
To: common-user@hadoop.apache.org
Subject: How can I syncronize writing to an hdfs file

I want to write to a common hdfs file from within my map method.  Given that
each task runs in a separate jvm (on separate machines) making a method
syncronized will not work I assume.  Are there any file locking or other
methods to guarantee mutual exclusion on hdfs?

(I want to append to this file and I have the append option turned on.)
Thanks.