You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Shady Xu <sh...@gmail.com> on 2015/05/04 03:29:53 UTC
Re: Too many connections from hive to zookeeper
The lock manager is used to provide concurrent ability for hive, turning it
off will affect the performance. I will try to set the
hive.server2.enable.doAs property to see what will happen.
2015-04-30 22:02 GMT+08:00 Eduardo Ferreira <ea...@gmail.com>:
> It looks to me using Hive's Table Lock Manager with ZK with Cloudera's
> CDH5.3.3 distribution (Hive 0.13.1) is a bad thing.
> The ticket indicates the problem is fixed in version 1.1.0 only.
> I asked earlier in a different email (Subject: Table Lock Manager:
> ZooKeeper cluster) about using the same ZK quorum setup for HBase for
> handling Hive's lock thing and I had the impression from a response from
> Cloudera that it should be fine.
> My thought now is that it's not fine. If Hive causes trouble in ZK server
> because it does not handle connections properly, it's a bad thing trying to
> use the same ZK quorum. It also seems to me this Table Lock feature should
> be turned off.
>
> On Thu, Apr 30, 2015 at 9:44 AM, Abe Weinograd <ab...@flonet.com> wrote:
>
>> I have had the same problem and bouncing the Hive server temporarily
>> alleviates this.
>>
>> Abe
>>
>> On Thu, Apr 30, 2015 at 2:20 AM, Shady Xu <sh...@gmail.com> wrote:
>>
>>> I found the patch here https://issues.apache.org/jira/browse/HIVE-9119.
>>> But upgrading is the last thing we want to do.
>>>
>>> Also, I found another solution which changes the
>>> hive.server2.enable.doAs property to false to use the same user to do
>>> the hive operations. But I'm not sure whether a user here is the same with
>>> a `client` described in the issue.
>>>
>>> 2015-04-30 11:01 GMT+08:00 Shady Xu <sh...@gmail.com>:
>>>
>>>> Thanks. But is there any other solution besides upgrading?
>>>>
>>>> 2015-04-30 10:55 GMT+08:00 Xuefu Zhang <xz...@cloudera.com>:
>>>>
>>>>> This is a known issue and has been fixed in later releases.
>>>>>
>>>>> --Xuefu
>>>>>
>>>>> On Wed, Apr 29, 2015 at 7:44 PM, Shady Xu <sh...@gmail.com> wrote:
>>>>>
>>>>>> Recently I found in the zookeeper log that there were too many client
>>>>>> connections and it was hive that was establishing more and more connections.
>>>>>>
>>>>>> I modified the max client connection property in zookeeper and that
>>>>>> fixed the problem, temporarily. But the connections hive made to zookeeper
>>>>>> were still growing.
>>>>>>
>>>>>> It seems that hive does not use a connection pool to reuse the
>>>>>> connections. So how can I configure to reduce the connections?
>>>>>>
>>>>>> P.S.
>>>>>> I am using Hive-0.12.0
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Too many connections from hive to zookeeper
Posted by Shady Xu <sh...@gmail.com>.
Setting the hive.server2.enable.doAs property to false does fix the problem
but may causes file permission problems.
If upgrading Hive is not permitted, maybe compiling the related Hive
packages with this patch (https://issues.apache.org/jira/browse/HIVE-9119)
to replace the jars in the lib will work. Anyone tried that? I plan to give
it a shot.
2015-05-04 9:29 GMT+08:00 Shady Xu <sh...@gmail.com>:
> The lock manager is used to provide concurrent ability for hive, turning
> it off will affect the performance. I will try to set the
> hive.server2.enable.doAs property to see what will happen.
>
> 2015-04-30 22:02 GMT+08:00 Eduardo Ferreira <ea...@gmail.com>:
>
>> It looks to me using Hive's Table Lock Manager with ZK with Cloudera's
>> CDH5.3.3 distribution (Hive 0.13.1) is a bad thing.
>> The ticket indicates the problem is fixed in version 1.1.0 only.
>> I asked earlier in a different email (Subject: Table Lock Manager:
>> ZooKeeper cluster) about using the same ZK quorum setup for HBase for
>> handling Hive's lock thing and I had the impression from a response from
>> Cloudera that it should be fine.
>> My thought now is that it's not fine. If Hive causes trouble in ZK server
>> because it does not handle connections properly, it's a bad thing trying to
>> use the same ZK quorum. It also seems to me this Table Lock feature should
>> be turned off.
>>
>> On Thu, Apr 30, 2015 at 9:44 AM, Abe Weinograd <ab...@flonet.com> wrote:
>>
>>> I have had the same problem and bouncing the Hive server temporarily
>>> alleviates this.
>>>
>>> Abe
>>>
>>> On Thu, Apr 30, 2015 at 2:20 AM, Shady Xu <sh...@gmail.com> wrote:
>>>
>>>> I found the patch here https://issues.apache.org/jira/browse/HIVE-9119.
>>>> But upgrading is the last thing we want to do.
>>>>
>>>> Also, I found another solution which changes the
>>>> hive.server2.enable.doAs property to false to use the same user to do
>>>> the hive operations. But I'm not sure whether a user here is the same with
>>>> a `client` described in the issue.
>>>>
>>>> 2015-04-30 11:01 GMT+08:00 Shady Xu <sh...@gmail.com>:
>>>>
>>>>> Thanks. But is there any other solution besides upgrading?
>>>>>
>>>>> 2015-04-30 10:55 GMT+08:00 Xuefu Zhang <xz...@cloudera.com>:
>>>>>
>>>>>> This is a known issue and has been fixed in later releases.
>>>>>>
>>>>>> --Xuefu
>>>>>>
>>>>>> On Wed, Apr 29, 2015 at 7:44 PM, Shady Xu <sh...@gmail.com> wrote:
>>>>>>
>>>>>>> Recently I found in the zookeeper log that there were too many
>>>>>>> client connections and it was hive that was establishing more and more
>>>>>>> connections.
>>>>>>>
>>>>>>> I modified the max client connection property in zookeeper and that
>>>>>>> fixed the problem, temporarily. But the connections hive made to zookeeper
>>>>>>> were still growing.
>>>>>>>
>>>>>>> It seems that hive does not use a connection pool to reuse the
>>>>>>> connections. So how can I configure to reduce the connections?
>>>>>>>
>>>>>>> P.S.
>>>>>>> I am using Hive-0.12.0
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>