You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Ahmad Humayun <ah...@gmail.com> on 2008/03/01 10:26:38 UTC
Re: hdfsLock
Well this is what I want to do. Lets say I have two nodes A & B each working
with the HDFS. I have a file named, say, temp. In the file temp I just store
a single integer (1 or 0). Now both these nodes access the file temp.
Everything in concept should work fine as long as both the nodes are just
reading the file, but what if a Node has to change the integer inside temp.
I think the normal procedure will be to read the temp file's integer, delete
the file and write a new file temp with the new required value. Lets say
Node B wanted to go through this procedure of changing the value inside
temp. While Node B is "changing/writing" to temp, I don't want Node A to be
able to either read or write to temp. Its something like 'flock()' through
the same fd (file descriptor)
In short I want exclusive access for the Node changing a file. Its the
Readers Writers problem :)
Thanks for the wonderful support on this mailing list :)
regards,
On Fri, Feb 29, 2008 at 12:47 AM, Raghu Angadi <ra...@yahoo-inc.com>
wrote:
>
> What do you want to use these locks for? From your question, you seem to
> be concerned about multiple threads accessing a stream. FileLocks have
> nothing to do with that. They were meant to be something like 'flock()'
> system call.
>
> Raghu.
>
> Ahmad Humayun wrote:
> > Do you any reason why? Is it because only one thread can write to a
> specific
> > file in the hdfs at a single time....and multiple simultaneous readers
> > doesn't cause any problem...hence locks make no sense in this
> situation???
> >
> > thanks for the help.
> >
> >
> > regards,
> >
> > On Thu, Feb 28, 2008 at 10:28 PM, Raghu Angadi <ra...@yahoo-inc.com>
> > wrote:
> >
> >> File locking is not supported in HDFS. Not sure if it ever was
> supported
> >> properly. This interface was deprecated last year.
> >>
> >> Raghu.
> >>
> >> Ahmad Humayun wrote:
> >>> Hello everyone,
> >>>
> >>> Does anybody have an idea why hdfsLock and hdfsReleaseLock been taken
> >> out of
> >>> libhdfs? How do I lock a file now using libhdfs? Can somebody point me
> >> to
> >>> the changelog or smth?
> >>>
> >>>
> >>> regards,
> >>
> >
> >
>
>
--
Ahmad Humayun
Research Assistant
Computer Science Dpt., LUMS
+92 321 4457315
Re: hdfsLock
Posted by Ahmad Humayun <ah...@gmail.com>.
Is there any way to append a record at the end of a file....something like
what GFS has?
regards,
On Mon, Mar 3, 2008 at 12:23 PM, dhruba Borthakur <dh...@yahoo-inc.com>
wrote:
> Hi Ahmad,
>
> In the current implementation of HDFS, files are "write-once". You
> cannot change/modify the contents of a file once it is written.
>
> Thanks,
> dhruba
>
> -----Original Message-----
> From: Ahmad Humayun [mailto:ahmad.humyn@gmail.com]
> Sent: Saturday, March 01, 2008 1:27 AM
> To: core-dev@hadoop.apache.org
> Subject: Re: hdfsLock
>
> Well this is what I want to do. Lets say I have two nodes A & B each
> working
> with the HDFS. I have a file named, say, temp. In the file temp I just
> store
> a single integer (1 or 0). Now both these nodes access the file temp.
> Everything in concept should work fine as long as both the nodes are
> just
> reading the file, but what if a Node has to change the integer inside
> temp.
> I think the normal procedure will be to read the temp file's integer,
> delete
> the file and write a new file temp with the new required value. Lets say
> Node B wanted to go through this procedure of changing the value inside
> temp. While Node B is "changing/writing" to temp, I don't want Node A to
> be
> able to either read or write to temp. Its something like 'flock()'
> through
> the same fd (file descriptor)
>
> In short I want exclusive access for the Node changing a file. Its the
> Readers Writers problem :)
>
>
> Thanks for the wonderful support on this mailing list :)
>
> regards,
>
>
> On Fri, Feb 29, 2008 at 12:47 AM, Raghu Angadi <ra...@yahoo-inc.com>
> wrote:
>
> >
> > What do you want to use these locks for? From your question, you seem
> to
> > be concerned about multiple threads accessing a stream. FileLocks have
> > nothing to do with that. They were meant to be something like
> 'flock()'
> > system call.
> >
> > Raghu.
> >
> > Ahmad Humayun wrote:
> > > Do you any reason why? Is it because only one thread can write to a
> > specific
> > > file in the hdfs at a single time....and multiple simultaneous
> readers
> > > doesn't cause any problem...hence locks make no sense in this
> > situation???
> > >
> > > thanks for the help.
> > >
> > >
> > > regards,
> > >
> > > On Thu, Feb 28, 2008 at 10:28 PM, Raghu Angadi
> <ra...@yahoo-inc.com>
> > > wrote:
> > >
> > >> File locking is not supported in HDFS. Not sure if it ever was
> > supported
> > >> properly. This interface was deprecated last year.
> > >>
> > >> Raghu.
> > >>
> > >> Ahmad Humayun wrote:
> > >>> Hello everyone,
> > >>>
> > >>> Does anybody have an idea why hdfsLock and hdfsReleaseLock been
> taken
> > >> out of
> > >>> libhdfs? How do I lock a file now using libhdfs? Can somebody
> point me
> > >> to
> > >>> the changelog or smth?
> > >>>
> > >>>
> > >>> regards,
> > >>
> > >
> > >
> >
> >
>
>
> --
> Ahmad Humayun
> Research Assistant
> Computer Science Dpt., LUMS
> +92 321 4457315
>
--
Ahmad Humayun
Research Assistant
Computer Science Dpt., LUMS
+92 321 4457315
RE: hdfsLock
Posted by dhruba Borthakur <dh...@yahoo-inc.com>.
Hi Ahmad,
In the current implementation of HDFS, files are "write-once". You
cannot change/modify the contents of a file once it is written.
Thanks,
dhruba
-----Original Message-----
From: Ahmad Humayun [mailto:ahmad.humyn@gmail.com]
Sent: Saturday, March 01, 2008 1:27 AM
To: core-dev@hadoop.apache.org
Subject: Re: hdfsLock
Well this is what I want to do. Lets say I have two nodes A & B each
working
with the HDFS. I have a file named, say, temp. In the file temp I just
store
a single integer (1 or 0). Now both these nodes access the file temp.
Everything in concept should work fine as long as both the nodes are
just
reading the file, but what if a Node has to change the integer inside
temp.
I think the normal procedure will be to read the temp file's integer,
delete
the file and write a new file temp with the new required value. Lets say
Node B wanted to go through this procedure of changing the value inside
temp. While Node B is "changing/writing" to temp, I don't want Node A to
be
able to either read or write to temp. Its something like 'flock()'
through
the same fd (file descriptor)
In short I want exclusive access for the Node changing a file. Its the
Readers Writers problem :)
Thanks for the wonderful support on this mailing list :)
regards,
On Fri, Feb 29, 2008 at 12:47 AM, Raghu Angadi <ra...@yahoo-inc.com>
wrote:
>
> What do you want to use these locks for? From your question, you seem
to
> be concerned about multiple threads accessing a stream. FileLocks have
> nothing to do with that. They were meant to be something like
'flock()'
> system call.
>
> Raghu.
>
> Ahmad Humayun wrote:
> > Do you any reason why? Is it because only one thread can write to a
> specific
> > file in the hdfs at a single time....and multiple simultaneous
readers
> > doesn't cause any problem...hence locks make no sense in this
> situation???
> >
> > thanks for the help.
> >
> >
> > regards,
> >
> > On Thu, Feb 28, 2008 at 10:28 PM, Raghu Angadi
<ra...@yahoo-inc.com>
> > wrote:
> >
> >> File locking is not supported in HDFS. Not sure if it ever was
> supported
> >> properly. This interface was deprecated last year.
> >>
> >> Raghu.
> >>
> >> Ahmad Humayun wrote:
> >>> Hello everyone,
> >>>
> >>> Does anybody have an idea why hdfsLock and hdfsReleaseLock been
taken
> >> out of
> >>> libhdfs? How do I lock a file now using libhdfs? Can somebody
point me
> >> to
> >>> the changelog or smth?
> >>>
> >>>
> >>> regards,
> >>
> >
> >
>
>
--
Ahmad Humayun
Research Assistant
Computer Science Dpt., LUMS
+92 321 4457315