You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Jim Green <op...@gmail.com> on 2015/02/26 02:24:50 UTC
Reading tables will lock entire database in Hive 0.13
Hi Team,
Here is old thread about this issue:
http://qnalist.com/questions/5091386/doubt-about-locking-mechanism-in-hive
I met the same issue that when reading one table in session A, we can not
create a table in the same database in session B. Basically it causes
database level lock in Hive 0.13 when hive.support.concurrency=true.
Minimum reproduce:
--Session A:
create table passwords (col0 string, col1 string, col2 string, col3 string,
col4 string,
col5 string, col6 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ":";
load data local inpath "/etc/passwd" into table passwords;
select count(*) from passwords a,passwords b, passwords c, passwords d
where a.col0=b.col0
and a.col0=c.col0
and a.col0=d.col0;
--Session B:
hive> create table test(id int);
conflicting lock present for default mode EXCLUSIVE
*Is this fixed or any jira related?*
--
Thanks,
www.openkb.info
(Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
Re: Reading tables will lock entire database in Hive 0.13
Posted by Jim Green <op...@gmail.com>.
I found the answer…hive.txn.manager is added starting in hive 0.13.
Thanks all.
On Mon, Aug 31, 2015 at 3:54 PM, Jim Green <op...@gmail.com> wrote:
> Hi Team,
>
> This issue only happens on hive 0.13, not in Hive 0.12.
> Does anybody know what is the change between 0.12 and 0.13 which caused
> this issue?
>
> Thanks.
>
> On Sat, Feb 28, 2015 at 3:47 AM, Chaoyu Tang <ct...@cloudera.com> wrote:
>
>> It was fixed for DummyTxnManager. DbTxnManager should not have this
>> issue.
>>
>> Thanks
>>
>> On Thu, Feb 26, 2015 at 1:52 PM, Jim Green <op...@gmail.com> wrote:
>>
>>> Hi Chaoyu,
>>>
>>> Thanks for the information.
>>> Actually by default, if we set hive.support.concurrency=true in Hive
>>> 0.13:
>>>
>>> hive.lock.manager=org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
>>> hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
>>>
>>> Hive-9199 mentioned it is fixed in 1.1, do you know if it will be fixed
>>> under DummyTxnManager? Or does it mean it is fixed using new DbTxnManager?
>>>
>>> Thanks.
>>>
>>>
>>>
>>> On Thu, Feb 26, 2015 at 4:51 AM, Chaoyu Tang <ct...@cloudera.com> wrote:
>>>
>>>> If you are using DummyTxnManager, please see HIVE-9199 which should
>>>> resolve the issue.
>>>>
>>>> Thanks
>>>>
>>>> Chaoyu
>>>>
>>>> On Wed, Feb 25, 2015 at 11:57 PM, Alan Gates <al...@gmail.com>
>>>> wrote:
>>>>
>>>>> What lock manager are you using?
>>>>>
>>>>> Alan.
>>>>>
>>>>> Jim Green <op...@gmail.com>
>>>>> February 25, 2015 at 17:24
>>>>> Hi Team,
>>>>>
>>>>> Here is old thread about this issue:
>>>>>
>>>>> http://qnalist.com/questions/5091386/doubt-about-locking-mechanism-in-hive
>>>>>
>>>>> I met the same issue that when reading one table in session A, we can
>>>>> not create a table in the same database in session B. Basically it causes
>>>>> database level lock in Hive 0.13 when hive.support.concurrency=true.
>>>>>
>>>>> Minimum reproduce:
>>>>> --Session A:
>>>>>
>>>>> create table passwords (col0 string, col1 string, col2 string, col3
>>>>> string, col4 string,
>>>>> col5 string, col6 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY
>>>>> ":";
>>>>> load data local inpath "/etc/passwd" into table passwords;
>>>>>
>>>>> select count(*) from passwords a,passwords b, passwords c, passwords d
>>>>> where a.col0=b.col0
>>>>> and a.col0=c.col0
>>>>> and a.col0=d.col0;
>>>>>
>>>>> --Session B:
>>>>>
>>>>> hive> create table test(id int);
>>>>> conflicting lock present for default mode EXCLUSIVE
>>>>>
>>>>>
>>>>> *Is this fixed or any jira related?*
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> www.openkb.info
>>>>> (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> www.openkb.info
>>> (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
>>>
>>
>>
>
>
> --
> Thanks,
> www.openkb.info
> (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
>
--
Thanks,
www.openkb.info
(Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
Re: Reading tables will lock entire database in Hive 0.13
Posted by Jim Green <op...@gmail.com>.
Hi Team,
This issue only happens on hive 0.13, not in Hive 0.12.
Does anybody know what is the change between 0.12 and 0.13 which caused
this issue?
Thanks.
On Sat, Feb 28, 2015 at 3:47 AM, Chaoyu Tang <ct...@cloudera.com> wrote:
> It was fixed for DummyTxnManager. DbTxnManager should not have this issue.
>
> Thanks
>
> On Thu, Feb 26, 2015 at 1:52 PM, Jim Green <op...@gmail.com> wrote:
>
>> Hi Chaoyu,
>>
>> Thanks for the information.
>> Actually by default, if we set hive.support.concurrency=true in Hive
>> 0.13:
>>
>> hive.lock.manager=org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
>> hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
>>
>> Hive-9199 mentioned it is fixed in 1.1, do you know if it will be fixed
>> under DummyTxnManager? Or does it mean it is fixed using new DbTxnManager?
>>
>> Thanks.
>>
>>
>>
>> On Thu, Feb 26, 2015 at 4:51 AM, Chaoyu Tang <ct...@cloudera.com> wrote:
>>
>>> If you are using DummyTxnManager, please see HIVE-9199 which should
>>> resolve the issue.
>>>
>>> Thanks
>>>
>>> Chaoyu
>>>
>>> On Wed, Feb 25, 2015 at 11:57 PM, Alan Gates <al...@gmail.com>
>>> wrote:
>>>
>>>> What lock manager are you using?
>>>>
>>>> Alan.
>>>>
>>>> Jim Green <op...@gmail.com>
>>>> February 25, 2015 at 17:24
>>>> Hi Team,
>>>>
>>>> Here is old thread about this issue:
>>>>
>>>> http://qnalist.com/questions/5091386/doubt-about-locking-mechanism-in-hive
>>>>
>>>> I met the same issue that when reading one table in session A, we can
>>>> not create a table in the same database in session B. Basically it causes
>>>> database level lock in Hive 0.13 when hive.support.concurrency=true.
>>>>
>>>> Minimum reproduce:
>>>> --Session A:
>>>>
>>>> create table passwords (col0 string, col1 string, col2 string, col3
>>>> string, col4 string,
>>>> col5 string, col6 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ":";
>>>> load data local inpath "/etc/passwd" into table passwords;
>>>>
>>>> select count(*) from passwords a,passwords b, passwords c, passwords d
>>>> where a.col0=b.col0
>>>> and a.col0=c.col0
>>>> and a.col0=d.col0;
>>>>
>>>> --Session B:
>>>>
>>>> hive> create table test(id int);
>>>> conflicting lock present for default mode EXCLUSIVE
>>>>
>>>>
>>>> *Is this fixed or any jira related?*
>>>>
>>>> --
>>>> Thanks,
>>>> www.openkb.info
>>>> (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
>>>>
>>>>
>>>
>>
>>
>> --
>> Thanks,
>> www.openkb.info
>> (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
>>
>
>
--
Thanks,
www.openkb.info
(Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
Re: Reading tables will lock entire database in Hive 0.13
Posted by Chaoyu Tang <ct...@cloudera.com>.
It was fixed for DummyTxnManager. DbTxnManager should not have this issue.
Thanks
On Thu, Feb 26, 2015 at 1:52 PM, Jim Green <op...@gmail.com> wrote:
> Hi Chaoyu,
>
> Thanks for the information.
> Actually by default, if we set hive.support.concurrency=true in Hive 0.13:
>
> hive.lock.manager=org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
>
> Hive-9199 mentioned it is fixed in 1.1, do you know if it will be fixed
> under DummyTxnManager? Or does it mean it is fixed using new DbTxnManager?
>
> Thanks.
>
>
>
> On Thu, Feb 26, 2015 at 4:51 AM, Chaoyu Tang <ct...@cloudera.com> wrote:
>
>> If you are using DummyTxnManager, please see HIVE-9199 which should
>> resolve the issue.
>>
>> Thanks
>>
>> Chaoyu
>>
>> On Wed, Feb 25, 2015 at 11:57 PM, Alan Gates <al...@gmail.com>
>> wrote:
>>
>>> What lock manager are you using?
>>>
>>> Alan.
>>>
>>> Jim Green <op...@gmail.com>
>>> February 25, 2015 at 17:24
>>> Hi Team,
>>>
>>> Here is old thread about this issue:
>>>
>>> http://qnalist.com/questions/5091386/doubt-about-locking-mechanism-in-hive
>>>
>>> I met the same issue that when reading one table in session A, we can
>>> not create a table in the same database in session B. Basically it causes
>>> database level lock in Hive 0.13 when hive.support.concurrency=true.
>>>
>>> Minimum reproduce:
>>> --Session A:
>>>
>>> create table passwords (col0 string, col1 string, col2 string, col3
>>> string, col4 string,
>>> col5 string, col6 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ":";
>>> load data local inpath "/etc/passwd" into table passwords;
>>>
>>> select count(*) from passwords a,passwords b, passwords c, passwords d
>>> where a.col0=b.col0
>>> and a.col0=c.col0
>>> and a.col0=d.col0;
>>>
>>> --Session B:
>>>
>>> hive> create table test(id int);
>>> conflicting lock present for default mode EXCLUSIVE
>>>
>>>
>>> *Is this fixed or any jira related?*
>>>
>>> --
>>> Thanks,
>>> www.openkb.info
>>> (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
>>>
>>>
>>
>
>
> --
> Thanks,
> www.openkb.info
> (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
>
Re: Reading tables will lock entire database in Hive 0.13
Posted by Jim Green <op...@gmail.com>.
Hi Chaoyu,
Thanks for the information.
Actually by default, if we set hive.support.concurrency=true in Hive 0.13:
hive.lock.manager=org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
Hive-9199 mentioned it is fixed in 1.1, do you know if it will be fixed
under DummyTxnManager? Or does it mean it is fixed using new DbTxnManager?
Thanks.
On Thu, Feb 26, 2015 at 4:51 AM, Chaoyu Tang <ct...@cloudera.com> wrote:
> If you are using DummyTxnManager, please see HIVE-9199 which should
> resolve the issue.
>
> Thanks
>
> Chaoyu
>
> On Wed, Feb 25, 2015 at 11:57 PM, Alan Gates <al...@gmail.com> wrote:
>
>> What lock manager are you using?
>>
>> Alan.
>>
>> Jim Green <op...@gmail.com>
>> February 25, 2015 at 17:24
>> Hi Team,
>>
>> Here is old thread about this issue:
>> http://qnalist.com/questions/5091386/doubt-about-locking-mechanism-in-hive
>>
>> I met the same issue that when reading one table in session A, we can not
>> create a table in the same database in session B. Basically it causes
>> database level lock in Hive 0.13 when hive.support.concurrency=true.
>>
>> Minimum reproduce:
>> --Session A:
>>
>> create table passwords (col0 string, col1 string, col2 string, col3
>> string, col4 string,
>> col5 string, col6 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ":";
>> load data local inpath "/etc/passwd" into table passwords;
>>
>> select count(*) from passwords a,passwords b, passwords c, passwords d
>> where a.col0=b.col0
>> and a.col0=c.col0
>> and a.col0=d.col0;
>>
>> --Session B:
>>
>> hive> create table test(id int);
>> conflicting lock present for default mode EXCLUSIVE
>>
>>
>> *Is this fixed or any jira related?*
>>
>> --
>> Thanks,
>> www.openkb.info
>> (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
>>
>>
>
--
Thanks,
www.openkb.info
(Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
Re: Reading tables will lock entire database in Hive 0.13
Posted by Chaoyu Tang <ct...@cloudera.com>.
If you are using DummyTxnManager, please see HIVE-9199 which should resolve
the issue.
Thanks
Chaoyu
On Wed, Feb 25, 2015 at 11:57 PM, Alan Gates <al...@gmail.com> wrote:
> What lock manager are you using?
>
> Alan.
>
> Jim Green <op...@gmail.com>
> February 25, 2015 at 17:24
> Hi Team,
>
> Here is old thread about this issue:
> http://qnalist.com/questions/5091386/doubt-about-locking-mechanism-in-hive
>
> I met the same issue that when reading one table in session A, we can not
> create a table in the same database in session B. Basically it causes
> database level lock in Hive 0.13 when hive.support.concurrency=true.
>
> Minimum reproduce:
> --Session A:
>
> create table passwords (col0 string, col1 string, col2 string, col3
> string, col4 string,
> col5 string, col6 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ":";
> load data local inpath "/etc/passwd" into table passwords;
>
> select count(*) from passwords a,passwords b, passwords c, passwords d
> where a.col0=b.col0
> and a.col0=c.col0
> and a.col0=d.col0;
>
> --Session B:
>
> hive> create table test(id int);
> conflicting lock present for default mode EXCLUSIVE
>
>
> *Is this fixed or any jira related?*
>
> --
> Thanks,
> www.openkb.info
> (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
>
>
Re: Reading tables will lock entire database in Hive 0.13
Posted by Alan Gates <al...@gmail.com>.
What lock manager are you using?
Alan.
> Jim Green <ma...@gmail.com>
> February 25, 2015 at 17:24
> Hi Team,
>
> Here is old thread about this issue:
> http://qnalist.com/questions/5091386/doubt-about-locking-mechanism-in-hive
>
> I met the same issue that when reading one table in session A, we can
> not create a table in the same database in session B. Basically it
> causes database level lock in Hive 0.13
> when hive.support.concurrency=true.
>
> Minimum reproduce:
> --Session A:
>
> create table passwords (col0 string, col1 string, col2 string, col3
> string, col4 string,
> col5 string, col6 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ":";
> load data local inpath "/etc/passwd" into table passwords;
>
> select count(*) from passwords a,passwords b, passwords c, passwords d
> where a.col0=b.col0
> and a.col0=c.col0
> and a.col0=d.col0;
>
> --Session B:
>
> hive> create table test(id int);
> conflicting lock present for default mode EXCLUSIVE
>
>
> *_/Is this fixed or any jira related?/_*
>
> --
> Thanks,
> www.openkb.info <http://www.openkb.info>
> (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)