You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Elliot West <te...@gmail.com> on 2015/10/22 15:40:36 UTC

Locking when using the Metastore/HCatalog APIs.

I notice from the Hive locking wiki page
<https://cwiki.apache.org/confluence/display/Hive/Locking> that locks may
be acquired for a range of HQL DDL operations. I wanted to know how the
locking scheme mapped mapped/employed by equivalent operations in the
Metastore and HCatalog APIs. Consider the operation:

alter table T1 drop partition P1


This apparently requires a shared lock on T1 and an exclusive lock on P1.
However, when following the invocation chain on the equivalent APIs I can
see no acquisition of such locks:

HCatClient.dropPartitions(...)
IMetaStoreClient.dropPartition(...)


I notice that IMetaStoreClient exposes methods to lock and unlock resources
and I have used these when working with the ACID APIs. However I can find
nothing similar on the HCatalog API. As a user of these APIs:

   1. Do I need to acquire/release locks as declared on the wiki page when
   using equivalent operations in these APIs?
   2. If not, why not?
   3. Ideally, should the HCatClient also expose lock/unlock methods like
   the Metastore API?
   4. If not, why not?

Thank you - Elliot.

Re: Locking when using the Metastore/HCatalog APIs.

Posted by Elliot West <te...@gmail.com>.
Captured in HIVE-12285.

Thanks - Elliot.

On 28 October 2015 at 08:54, Elliot West <te...@gmail.com> wrote:

> Perhaps, I'd expect one might wish to make multiple API calls when holding
> a lock, but a suitably implemented client may be able to manage this
> seamlessly. Then again a user may also wish to have explicit control of
> locking. No reason it couldn't offer both. I'll create a JIRA to capture
> this. As we are working with these APIs currently, I expect that we may be
> able to submit a patch.
>
> Thanks for your responses,
>
> Elliot.
>
>
> On Tuesday, 27 October 2015, Eugene Koifman <ek...@hortonworks.com>
> wrote:
>
>> Wouldn’t it make more sense for the api to acquire required locks
>> automatically?   That seems like a simpler user model.
>>
>> From: Alan Gates <al...@gmail.com>
>> Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
>> Date: Tuesday, October 27, 2015 at 11:34 AM
>> To: "user@hive.apache.org" <us...@hive.apache.org>
>> Subject: Re: Locking when using the Metastore/HCatalog APIs.
>>
>> Answers inlined.
>>
>> Elliot West
>> October 22, 2015 at 6:40
>> I notice from the Hive locking wiki page
>> <https://cwiki.apache.org/confluence/display/Hive/Locking> that locks
>> may be acquired for a range of HQL DDL operations. I wanted to know how the
>> locking scheme mapped mapped/employed by equivalent operations in the
>> Metastore and HCatalog APIs. Consider the operation:
>>
>> alter table T1 drop partition P1
>>
>>
>> This apparently requires a shared lock on T1 and an exclusive lock on P1.
>> However, when following the invocation chain on the equivalent APIs I can
>> see no acquisition of such locks:
>>
>> HCatClient.dropPartitions(...)
>> IMetaStoreClient.dropPartition(...)
>>
>>
>> I notice that IMetaStoreClient exposes methods to lock and unlock
>> resources and I have used these when working with the ACID APIs. However I
>> can find nothing similar on the HCatalog API. As a user of these APIs:
>>
>>    1. Do I need to acquire/release locks as declared on the wiki page
>>    when using equivalent operations in these APIs?
>>
>> Yes.
>>
>>
>>    1. If not, why not?
>>    2. Ideally, should the HCatClient also expose lock/unlock methods
>>    like the Metastore API?
>>
>> Yes.
>>
>>
>>    1. If not, why not?
>>
>> Thank you - Elliot.
>>
>> Alan.
>>
>

Re: Locking when using the Metastore/HCatalog APIs.

Posted by Elliot West <te...@gmail.com>.
Perhaps, I'd expect one might wish to make multiple API calls when holding
a lock, but a suitably implemented client may be able to manage this
seamlessly. Then again a user may also wish to have explicit control of
locking. No reason it couldn't offer both. I'll create a JIRA to capture
this. As we are working with these APIs currently, I expect that we may be
able to submit a patch.

Thanks for your responses,

Elliot.

On Tuesday, 27 October 2015, Eugene Koifman <ek...@hortonworks.com>
wrote:

> Wouldn’t it make more sense for the api to acquire required locks
> automatically?   That seems like a simpler user model.
>
> From: Alan Gates <alanfgates@gmail.com
> <javascript:_e(%7B%7D,'cvml','alanfgates@gmail.com');>>
> Reply-To: "user@hive.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hive.apache.org');>" <
> user@hive.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hive.apache.org');>>
> Date: Tuesday, October 27, 2015 at 11:34 AM
> To: "user@hive.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hive.apache.org');>" <
> user@hive.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hive.apache.org');>>
> Subject: Re: Locking when using the Metastore/HCatalog APIs.
>
> Answers inlined.
>
> Elliot West <javascript:_e(%7B%7D,'cvml','teabot@gmail.com');>
> October 22, 2015 at 6:40
> I notice from the Hive locking wiki page
> <https://cwiki.apache.org/confluence/display/Hive/Locking> that locks may
> be acquired for a range of HQL DDL operations. I wanted to know how the
> locking scheme mapped mapped/employed by equivalent operations in the
> Metastore and HCatalog APIs. Consider the operation:
>
> alter table T1 drop partition P1
>
>
> This apparently requires a shared lock on T1 and an exclusive lock on P1.
> However, when following the invocation chain on the equivalent APIs I can
> see no acquisition of such locks:
>
> HCatClient.dropPartitions(...)
> IMetaStoreClient.dropPartition(...)
>
>
> I notice that IMetaStoreClient exposes methods to lock and unlock
> resources and I have used these when working with the ACID APIs. However I
> can find nothing similar on the HCatalog API. As a user of these APIs:
>
>    1. Do I need to acquire/release locks as declared on the wiki page
>    when using equivalent operations in these APIs?
>
> Yes.
>
>
>    1. If not, why not?
>    2. Ideally, should the HCatClient also expose lock/unlock methods like
>    the Metastore API?
>
> Yes.
>
>
>    1. If not, why not?
>
> Thank you - Elliot.
>
> Alan.
>

Re: Locking when using the Metastore/HCatalog APIs.

Posted by Eugene Koifman <ek...@hortonworks.com>.
Wouldn't it make more sense for the api to acquire required locks automatically?   That seems like a simpler user model.

From: Alan Gates <al...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Tuesday, October 27, 2015 at 11:34 AM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: Locking when using the Metastore/HCatalog APIs.

Answers inlined.

[cid:part1.03000101.04030306@gmail.com]
Elliot West<ma...@gmail.com>
October 22, 2015 at 6:40
I notice from the Hive locking wiki page<https://cwiki.apache.org/confluence/display/Hive/Locking> that locks may be acquired for a range of HQL DDL operations. I wanted to know how the locking scheme mapped mapped/employed by equivalent operations in the Metastore and HCatalog APIs. Consider the operation:

alter table T1 drop partition P1

This apparently requires a shared lock on T1 and an exclusive lock on P1. However, when following the invocation chain on the equivalent APIs I can see no acquisition of such locks:

HCatClient.dropPartitions(...)
IMetaStoreClient.dropPartition(...)

I notice that IMetaStoreClient exposes methods to lock and unlock resources and I have used these when working with the ACID APIs. However I can find nothing similar on the HCatalog API. As a user of these APIs:

  1.  Do I need to acquire/release locks as declared on the wiki page when using equivalent operations in these APIs?

Yes.

  1.  If not, why not?
  2.  Ideally, should the HCatClient also expose lock/unlock methods like the Metastore API?

Yes.

  1.  If not, why not?

Thank you - Elliot.
Alan.

Re: Locking when using the Metastore/HCatalog APIs.

Posted by Alan Gates <al...@gmail.com>.
Answers inlined.

> Elliot West <ma...@gmail.com>
> October 22, 2015 at 6:40
> I notice from the Hive locking wiki page 
> <https://cwiki.apache.org/confluence/display/Hive/Locking> that locks 
> may be acquired for a range of HQL DDL operations. I wanted to know 
> how the locking scheme mapped mapped/employed by equivalent operations 
> in the Metastore and HCatalog APIs. Consider the operation:
>
>     alter table T1 drop partition P1
>
>
> This apparently requires a shared lock on T1 and an exclusive lock on 
> P1. However, when following the invocation chain on the equivalent 
> APIs I can see no acquisition of such locks:
>
>     HCatClient.dropPartitions(...)
>     IMetaStoreClient.dropPartition(...)
>
>
> I notice that IMetaStoreClient exposes methods to lock and unlock 
> resources and I have used these when working with the ACID APIs. 
> However I can find nothing similar on the HCatalog API. As a user of 
> these APIs:
>
>  1. Do I need to acquire/release locks as declared on the wiki page
>     when using equivalent operations in these APIs?
>
Yes.
>
>  1. If not, why not?
>  2. Ideally, should the HCatClient also expose lock/unlock methods
>     like the Metastore API?
>
Yes.
>
>  1. If not, why not?
>
> Thank you - Elliot.
Alan.