Re: Locks in HDFS

[Harsh J <ha...@cloudera.com>]

HDFS does not have such a client-side feature, but your applications
can use Apache Zookeeper to coordinate and implement this on their own
- it can be used to achieve distributed locking. While at ZooKeeper,
also checkout https://github.com/Netflix/curator which makes using it
for common needs very easy.

On Fri, Feb 22, 2013 at 5:17 AM, abhishek <ab...@gmail.com> wrote:
>
>> Hello,
>
>> How can I impose read lock, for a file in HDFS
>>
>> So that only one user (or) one application , can access file in hdfs at any point of time.
>>
>> Regards
>> Abhi
>
> --
>
>
>



--
Harsh J

Re: Locks in HDFS

[abhishek <ab...@gmail.com>]

Thanks for reply harsh.I would look into Zookeeper.

Regards
Abhi

On Feb 22, 2013, at 1:03 AM, Harsh J <ha...@cloudera.com> wrote:

> HDFS does not have such a client-side feature, but your applications
> can use Apache Zookeeper to coordinate and implement this on their own
> - it can be used to achieve distributed locking. While at ZooKeeper,
> also checkout https://github.com/Netflix/curator which makes using it
> for common needs very easy.
> 
> On Fri, Feb 22, 2013 at 5:17 AM, abhishek <ab...@gmail.com> wrote:
>> 
>>> Hello,
>> 
>>> How can I impose read lock, for a file in HDFS
>>> 
>>> So that only one user (or) one application , can access file in hdfs at any point of time.
>>> 
>>> Regards
>>> Abhi
>> 
>> --
> 
> 
> 
> --
> Harsh J
> 
> -- 
> 
> 
> 

Re: Locks in HDFS

[abhishek <ab...@gmail.com>]

Harsh,

As a part of my use case, my work was to read a file from HDFS and update a value in it , the value gets incremented for every source load ( load id ) 

Say

Source load1 is started the value in the file could incremented by 1.

Source load 2 is started then value in the file could incremented by 1.

If two sources or multiple sources kicks off at same time, then I want other sources to wait till source1 updates the value in the file.So that other sources read the correct value and increment it by 1.

So I am looking for a best approach to lock the file till source1 (or) which ever source increments the value in the file.

So I thought it may be possible to load or put a file in HDFS, with single replication and use java synchronized method. So that only user or app can read access file at any given point of time.

Thanks
Abhi

On Feb 22, 2013, at 11:09 AM, Harsh J <ha...@cloudera.com> wrote:

> Hi Abhishek,
> 
> I fail to understand what you mean by that; but HDFS generally has no
> client-exposed file locking on reads. There's leases for preventing
> multiple writers to a single file, but nothing on the read side.
> 
> Replication of the blocks under a file is a different concept and is
> completely unrelated to this.
> 
> This needs to be built at your application's/stack's access/control
> levels, since HDFS does not provide this.
> 
> On Fri, Feb 22, 2013 at 9:33 PM, abhishek <ab...@gmail.com> wrote:
>> Harsh,
>> 
>> Can we load the file into HDFS with one replication and lock the file.
>> 
>> Regards
>> Abhishek
>> 
>> 
>> On Feb 22, 2013, at 1:03 AM, Harsh J <ha...@cloudera.com> wrote:
>> 
>>> HDFS does not have such a client-side feature, but your applications
>>> can use Apache Zookeeper to coordinate and implement this on their own
>>> - it can be used to achieve distributed locking. While at ZooKeeper,
>>> also checkout https://github.com/Netflix/curator which makes using it
>>> for common needs very easy.
>>> 
>>> On Fri, Feb 22, 2013 at 5:17 AM, abhishek <ab...@gmail.com> wrote:
>>>> 
>>>>> Hello,
>>>> 
>>>>> How can I impose read lock, for a file in HDFS
>>>>> 
>>>>> So that only one user (or) one application , can access file in hdfs at any point of time.
>>>>> 
>>>>> Regards
>>>>> Abhi
>>>> 
>>>> --
>>> 
>>> 
>>> 
>>> --
>>> Harsh J
>>> 
>>> --
> 
> 
> 
> --
> Harsh J

Re: Locks in HDFS

[Harsh J <ha...@cloudera.com>]

Hi Abhishek,

I fail to understand what you mean by that; but HDFS generally has no
client-exposed file locking on reads. There's leases for preventing
multiple writers to a single file, but nothing on the read side.

Replication of the blocks under a file is a different concept and is
completely unrelated to this.

This needs to be built at your application's/stack's access/control
levels, since HDFS does not provide this.

On Fri, Feb 22, 2013 at 9:33 PM, abhishek <ab...@gmail.com> wrote:
> Harsh,
>
> Can we load the file into HDFS with one replication and lock the file.
>
> Regards
> Abhishek
>
>
> On Feb 22, 2013, at 1:03 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> HDFS does not have such a client-side feature, but your applications
>> can use Apache Zookeeper to coordinate and implement this on their own
>> - it can be used to achieve distributed locking. While at ZooKeeper,
>> also checkout https://github.com/Netflix/curator which makes using it
>> for common needs very easy.
>>
>> On Fri, Feb 22, 2013 at 5:17 AM, abhishek <ab...@gmail.com> wrote:
>>>
>>>> Hello,
>>>
>>>> How can I impose read lock, for a file in HDFS
>>>>
>>>> So that only one user (or) one application , can access file in hdfs at any point of time.
>>>>
>>>> Regards
>>>> Abhi
>>>
>>> --
>>
>>
>>
>> --
>> Harsh J
>>
>> --
>>
>>
>>



--
Harsh J

Re: Locks in HDFS

[abhishek <ab...@gmail.com>]

Harsh,

Can we load the file into HDFS with one replication and lock the file.

Regards
Abhishek 


On Feb 22, 2013, at 1:03 AM, Harsh J <ha...@cloudera.com> wrote:

> HDFS does not have such a client-side feature, but your applications
> can use Apache Zookeeper to coordinate and implement this on their own
> - it can be used to achieve distributed locking. While at ZooKeeper,
> also checkout https://github.com/Netflix/curator which makes using it
> for common needs very easy.
> 
> On Fri, Feb 22, 2013 at 5:17 AM, abhishek <ab...@gmail.com> wrote:
>> 
>>> Hello,
>> 
>>> How can I impose read lock, for a file in HDFS
>>> 
>>> So that only one user (or) one application , can access file in hdfs at any point of time.
>>> 
>>> Regards
>>> Abhi
>> 
>> --
> 
> 
> 
> --
> Harsh J
> 
> -- 
> 
> 
>