You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Antoine DUBOIS <an...@cc.in2p3.fr> on 2021/09/02 14:46:16 UTC

Question regarding lock manager

Hello, 
I'm trying to configure ACID hive in a kerberos environment with : 
Hadoop 3.1.4 deployed in HA considered working 
and now I'm trying to setup hive with remote metastore and ACID configuration. 
I may misunderstand what is written in documentation https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions , but to me it's cleary stated that DbTxnManager do not need zookeeper to run properly, 
However it seems it in fact needs a zookeeper running somewhere as I have several attempt to connect to a local zookeeper instance: 
2021-09 -02T 15:22:21,708 INFO [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : Client will use GSSAPI as SASL mechanism. 
2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : creating sasl client: client=h******* ;service=zookeeper;serviceHostname=localhost 
2021-09-02T15:22:21,709 INFO [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Opening socket connection to server localhost/ 127 . 0 . 0 . 1 : 2181 . Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient' 
2021-09-02T15:22:21,710 WARN [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Session 0x0 for server null , unexpected error, closing socket connection and attempting reconnect 
java.net.ConnectException : Connection refused 

Could you please help me to understand properly the documentation ? Do I need or not a zookeeper instance if using DbTxnManager and if so, why it isn't stated precisely and explicitely in the documentation ? 
I hope you have the best of day. 

Antoine DUBOIS 

Re: Question regarding lock manager

Posted by Antoine DUBOIS <an...@cc.in2p3.fr>.
Thank you for your insight, 
however on Hive 3.1.2 code is like this : 

public void startPrivilegeSynchonizer (HiveConf hiveConf) throws Exception { 

PolicyProviderContainer policyContainer = new PolicyProviderContainer() ; 
HiveAuthorizer authorizer = SessionState. get ().getAuthorizerV2() ; 
if (authorizer.getHivePolicyProvider() != null ) { 
policyContainer.addAuthorizer(authorizer) ; 
} 
if (hiveConf.get(MetastoreConf.ConfVars. PRE_EVENT_LISTENERS .getVarname()) != null && 
hiveConf.get(MetastoreConf.ConfVars. PRE_EVENT_LISTENERS .getVarname()).contains( 
"org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener" ) && 
hiveConf.get(MetastoreConf.ConfVars. HIVE_AUTHORIZATION_MANAGER .getVarname())!= null ) { 
List<HiveMetastoreAuthorizationProvider> providers = HiveUtils. getMetaStoreAuthorizeProviderManagers ( 
hiveConf , HiveConf.ConfVars. HIVE_METASTORE_AUTHORIZATION_MANAGER , SessionState. get ().getAuthenticator()) ; 
for (HiveMetastoreAuthorizationProvider provider : providers) { 
if (provider.getHivePolicyProvider() != null ) { 
policyContainer.addAuthorizationProvider(provider) ; 
} 
} 
} 
I see that github master branch has a check with hive.privilege.synchronizer however this change seems very recent and there's no reference to such key in documentation either. 

Thank you for the tips ;-) 



De: "Rajesh Balamohan" <rb...@apache.org> 
À: user@hive.apache.org 
Envoyé: Mardi 7 Septembre 2021 00:46:43 
Objet: Re: Question regarding lock manager 

For the specific code you mentioned, check if you have " hive.privilege.synchronizer" enabled or not. If so, disable it explicitly. PrivSync is needed for populating information_schema. 

~Rajesh.B 

On Mon, Sep 6, 2021 at 8:04 PM Antoine DUBOIS < [ mailto:antoine.dubois@cc.in2p3.fr | antoine.dubois@cc.in2p3.fr ] > wrote: 



Hello all 
After some digging and remote java debug, 
I found out that in file HiveServer2.java line 1001: 

if (policyContainer.size() > 0 ) { 
zKClientForPrivSync = startZookeeperClient(hiveConf) ; 
String rootNamespace = hiveConf.getVar(HiveConf.ConfVars. HIVE_SERVER2_ZOOKEEPER_NAMESPACE ) ; 
In this case policyContainer.size is > 1 has there's an object RangerHiveAuthorizer 

In this case I'm a bit puzzled, why hive would need zookeeper has Ranger is supposed to maintains the policy the same for all node ? 

Have a very nice day. 


De: "Antoine DUBOIS" < [ mailto:antoine.dubois@cc.in2p3.fr | antoine.dubois@cc.in2p3.fr ] > 
À: [ mailto:user@hive.apache.org | user@hive.apache.org ] 
Envoyé: Lundi 6 Septembre 2021 10:03:59 
Objet: Re: Question regarding lock manager 

Hello Alan, 
Thank you for your answer, 
I'm pretty sure I've respected the guide provided and did not set any discovery service. 
However I'm trying to setup ranger as well and though I didn't though it could cause the issue, after futher investigation I suspect ranger might be causing the issue. 
Hadoop 3.1.4 
Hive 3.1.2 
Ranger 2.1.0 
Tez 0.9.2 
It might come from imcompatibles version but I cannot find a proper compatibility matrix for all hadoop ecosystem. 
Thank you very much. 

Antoine 


De: "Alan Gates" < [ mailto:alanfgates@gmail.com | alanfgates@gmail.com ] > 
À: [ mailto:user@hive.apache.org | user@hive.apache.org ] 
Envoyé: Vendredi 3 Septembre 2021 17:51:41 
Objet: Re: Question regarding lock manager 

You do not need ZooKeeper to use ACID in Hive. The first thing I would check is that you have configured your system as described on this page: [ https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ] . Also, make sure you have not set hive.lock.manager to zookeeper. 

There are other features in Hive that can optionally use ZK, such as the discovery service for HiveServer2, you'll want to make sure this isn't coming from there rather than from the transaction system. 

Alan. 

On Thu, Sep 2, 2021 at 7:46 AM Antoine DUBOIS < [ mailto:antoine.dubois@cc.in2p3.fr | antoine.dubois@cc.in2p3.fr ] > wrote: 

BQ_BEGIN


Hello, 
I'm trying to configure ACID hive in a kerberos environment with : 
Hadoop 3.1.4 deployed in HA considered working 
and now I'm trying to setup hive with remote metastore and ACID configuration. 
I may misunderstand what is written in documentation [ https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ] , but to me it's cleary stated that DbTxnManager do not need zookeeper to run properly, 
However it seems it in fact needs a zookeeper running somewhere as I have several attempt to connect to a local zookeeper instance: 
2021-09 -02T 15:22:21,708 INFO [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : Client will use GSSAPI as SASL mechanism. 
2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : creating sasl client: client=h******* ;service=zookeeper;serviceHostname=localhost 
2021-09-02T15:22:21,709 INFO [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Opening socket connection to server localhost/ 127 . 0 . 0 . 1 : 2181 . Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient' 
2021-09-02T15:22:21,710 WARN [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Session 0x0 for server null , unexpected error, closing socket connection and attempting reconnect 
java.net.ConnectException : Connection refused 

Could you please help me to understand properly the documentation ? Do I need or not a zookeeper instance if using DbTxnManager and if so, why it isn't stated precisely and explicitely in the documentation ? 
I hope you have the best of day. 

Antoine DUBOIS 




BQ_END



Re: Question regarding lock manager

Posted by Rajesh Balamohan <rb...@apache.org>.
For the specific code you mentioned, check if you have
"hive.privilege.synchronizer"
enabled or not. If so, disable it explicitly. PrivSync is needed for
populating information_schema.

~Rajesh.B

On Mon, Sep 6, 2021 at 8:04 PM Antoine DUBOIS <an...@cc.in2p3.fr>
wrote:

> Hello all
> After some digging and remote java debug,
> I found out that in file HiveServer2.java line 1001:
>
> if (policyContainer.size() > 0) {
>   zKClientForPrivSync = startZookeeperClient(hiveConf);
>   String rootNamespace = hiveConf.getVar(HiveConf.ConfVars.HIVE_SERVER2_ZOOKEEPER_NAMESPACE);
>
> In this case policyContainer.size is > 1 has there's an object
> RangerHiveAuthorizer
>
> In this case I'm a bit puzzled, why hive would need zookeeper has Ranger
> is supposed to maintains the policy the same for all node ?
>
> Have a very nice day.
>
> ------------------------------
> *De: *"Antoine DUBOIS" <an...@cc.in2p3.fr>
> *À: *user@hive.apache.org
> *Envoyé: *Lundi 6 Septembre 2021 10:03:59
> *Objet: *Re: Question regarding lock manager
>
> Hello Alan,
> Thank you for your answer,
> I'm pretty sure I've respected the guide provided and did not set any
> discovery service.
> However I'm trying to setup ranger as well and though I didn't though it
> could cause the issue, after futher investigation I suspect ranger might be
> causing the issue.
> Hadoop 3.1.4
> Hive 3.1.2
> Ranger 2.1.0
> Tez 0.9.2
> It might come from imcompatibles version but I cannot find a proper
> compatibility matrix for all hadoop ecosystem.
> Thank you very much.
>
> Antoine
>
> ------------------------------
> *De: *"Alan Gates" <al...@gmail.com>
> *À: *user@hive.apache.org
> *Envoyé: *Vendredi 3 Septembre 2021 17:51:41
> *Objet: *Re: Question regarding lock manager
>
> You do not need ZooKeeper to use ACID in Hive.  The first thing I would
> check is that you have configured your system as described on this page:
> https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions.
> Also, make sure you have not set hive.lock.manager to zookeeper.
>
> There are other features in Hive that can optionally use ZK, such as the
> discovery service for HiveServer2, you'll want to make sure this isn't
> coming from there rather than from the transaction system.
>
> Alan.
>
> On Thu, Sep 2, 2021 at 7:46 AM Antoine DUBOIS <an...@cc.in2p3.fr>
> wrote:
>
>>
>> Hello,
>> I'm trying to configure ACID hive in a kerberos environment with :
>> Hadoop 3.1.4 deployed in HA considered working
>> and now I'm trying to setup hive with remote metastore and ACID
>> configuration.
>> I may misunderstand what is written in documentation
>> https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions , but
>> to me it's cleary stated that DbTxnManager do not need zookeeper to run
>> properly,
>> However it seems it in fact needs a zookeeper running somewhere as I have
>> several attempt to connect to a local zookeeper instance:
>> 2021-09-02T15:22:21,708 INFO [main-SendThread(localhost:2181)]
>> client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
>> 2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost:2181)]
>> client.ZooKeeperSaslClient: creating sasl client: client=h*******
>> ;service=zookeeper;serviceHostname=localhost
>> 2021-09-02T15:22:21,709 INFO [main-SendThread(localhost:2181)]
>> zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0
>> .0.1:2181. Will attempt to SASL-authenticate using Login Context section
>> 'HiveZooKeeperClient'
>> 2021-09-02T15:22:21,710 WARN [main-SendThread(localhost:2181)]
>> zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error,
>> closing socket connection and attempting reconnect
>> java.net.ConnectException: Connection refused
>>
>> Could you please help me to understand properly the documentation ? Do I
>> need or not a zookeeper instance if using DbTxnManager and if so, why it
>> isn't stated precisely and explicitely in the documentation ?
>> I hope you have the best of day.
>>
>> Antoine DUBOIS
>>
>
>

Re: Question regarding lock manager

Posted by Antoine DUBOIS <an...@cc.in2p3.fr>.
Hello all 
After some digging and remote java debug, 
I found out that in file HiveServer2.java line 1001: 

if (policyContainer.size() > 0 ) { 
zKClientForPrivSync = startZookeeperClient(hiveConf) ; 
String rootNamespace = hiveConf.getVar(HiveConf.ConfVars. HIVE_SERVER2_ZOOKEEPER_NAMESPACE ) ; 
In this case policyContainer.size is > 1 has there's an object RangerHiveAuthorizer 

In this case I'm a bit puzzled, why hive would need zookeeper has Ranger is supposed to maintains the policy the same for all node ? 

Have a very nice day. 


De: "Antoine DUBOIS" <an...@cc.in2p3.fr> 
À: user@hive.apache.org 
Envoyé: Lundi 6 Septembre 2021 10:03:59 
Objet: Re: Question regarding lock manager 

Hello Alan, 
Thank you for your answer, 
I'm pretty sure I've respected the guide provided and did not set any discovery service. 
However I'm trying to setup ranger as well and though I didn't though it could cause the issue, after futher investigation I suspect ranger might be causing the issue. 
Hadoop 3.1.4 
Hive 3.1.2 
Ranger 2.1.0 
Tez 0.9.2 
It might come from imcompatibles version but I cannot find a proper compatibility matrix for all hadoop ecosystem. 
Thank you very much. 

Antoine 


De: "Alan Gates" <al...@gmail.com> 
À: user@hive.apache.org 
Envoyé: Vendredi 3 Septembre 2021 17:51:41 
Objet: Re: Question regarding lock manager 

You do not need ZooKeeper to use ACID in Hive. The first thing I would check is that you have configured your system as described on this page: [ https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ] . Also, make sure you have not set hive.lock.manager to zookeeper. 

There are other features in Hive that can optionally use ZK, such as the discovery service for HiveServer2, you'll want to make sure this isn't coming from there rather than from the transaction system. 

Alan. 

On Thu, Sep 2, 2021 at 7:46 AM Antoine DUBOIS < [ mailto:antoine.dubois@cc.in2p3.fr | antoine.dubois@cc.in2p3.fr ] > wrote: 




Hello, 
I'm trying to configure ACID hive in a kerberos environment with : 
Hadoop 3.1.4 deployed in HA considered working 
and now I'm trying to setup hive with remote metastore and ACID configuration. 
I may misunderstand what is written in documentation [ https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ] , but to me it's cleary stated that DbTxnManager do not need zookeeper to run properly, 
However it seems it in fact needs a zookeeper running somewhere as I have several attempt to connect to a local zookeeper instance: 
2021-09 -02T 15:22:21,708 INFO [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : Client will use GSSAPI as SASL mechanism. 
2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : creating sasl client: client=h******* ;service=zookeeper;serviceHostname=localhost 
2021-09-02T15:22:21,709 INFO [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Opening socket connection to server localhost/ 127 . 0 . 0 . 1 : 2181 . Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient' 
2021-09-02T15:22:21,710 WARN [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Session 0x0 for server null , unexpected error, closing socket connection and attempting reconnect 
java.net.ConnectException : Connection refused 

Could you please help me to understand properly the documentation ? Do I need or not a zookeeper instance if using DbTxnManager and if so, why it isn't stated precisely and explicitely in the documentation ? 
I hope you have the best of day. 

Antoine DUBOIS 





Re: Question regarding lock manager

Posted by Antoine DUBOIS <an...@cc.in2p3.fr>.
No, 
and for now i don't plan to make use of zookeeper.
And as you can see in my previous email, I now understand why Hive is trying to connect to any zookeeper instance despite my configuration clearly state that it should be used.
In this case the presence of RangerPolicyManager is responsible for the use of zookeeper by hiveserver2. However why using ZooKeeper is useful or relevant is another story I can't tell.
Regards

----- Mail original -----
De: "Jan Fili" <ja...@gmail.com>
À: user@hive.apache.org
Envoyé: Lundi 6 Septembre 2021 16:33:28
Objet: Re: Question regarding lock manager

Hi just chipping in here real quick.

java.net.ConnectException: Connection refused

is a pretty good indicator that Zookeeper is not running or at least
not listening on the port you are expecting

ZK connect seems to be
localhost:2181

Do you have zookeeper running and is it listening on 2181?
Or any fancy firewall rules on the box preventing the connection?

Best Jan

Am Mo., 6. Sept. 2021 um 10:04 Uhr schrieb Antoine DUBOIS
<an...@cc.in2p3.fr>:
>
> Hello Alan,
> Thank you for your answer,
> I'm pretty sure I've respected the guide provided and did not set any discovery service.
> However I'm trying to setup ranger as well and though I didn't though it could cause the issue, after futher investigation I suspect ranger might be causing the issue.
> Hadoop 3.1.4
> Hive 3.1.2
> Ranger 2.1.0
> Tez 0.9.2
> It might come from imcompatibles version but I cannot find a proper compatibility matrix for all hadoop ecosystem.
> Thank you very much.
>
> Antoine
>
> ________________________________
> De: "Alan Gates" <al...@gmail.com>
> À: user@hive.apache.org
> Envoyé: Vendredi 3 Septembre 2021 17:51:41
> Objet: Re: Question regarding lock manager
>
> You do not need ZooKeeper to use ACID in Hive.  The first thing I would check is that you have configured your system as described on this page: https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions.  Also, make sure you have not set hive.lock.manager to zookeeper.
>
> There are other features in Hive that can optionally use ZK, such as the discovery service for HiveServer2, you'll want to make sure this isn't coming from there rather than from the transaction system.
>
> Alan.
>
> On Thu, Sep 2, 2021 at 7:46 AM Antoine DUBOIS <an...@cc.in2p3.fr> wrote:
>>
>>
>> Hello,
>> I'm trying to configure ACID hive in a kerberos environment with :
>> Hadoop 3.1.4 deployed in HA considered working
>> and now I'm trying to setup hive with remote metastore and ACID configuration.
>> I may misunderstand what is written in documentation https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions , but to me it's cleary stated that DbTxnManager do not need zookeeper to run properly,
>> However it seems it in fact needs a zookeeper running somewhere as I have several attempt to connect to a local zookeeper instance:
>> 2021-09-02T15:22:21,708 INFO [main-SendThread(localhost:2181)] client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
>> 2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost:2181)] client.ZooKeeperSaslClient: creating sasl client: client=h*******;service=zookeeper;serviceHostname=localhost
>> 2021-09-02T15:22:21,709 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient'
>> 2021-09-02T15:22:21,710 WARN [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
>> java.net.ConnectException: Connection refused
>>
>> Could you please help me to understand properly the documentation ? Do I need or not a zookeeper instance if using DbTxnManager and if so, why it isn't stated precisely and explicitely in the documentation ?
>> I hope you have the best of day.
>>
>> Antoine DUBOIS
>
>

Re: Question regarding lock manager

Posted by Jan Fili <ja...@gmail.com>.
Hi just chipping in here real quick.

java.net.ConnectException: Connection refused

is a pretty good indicator that Zookeeper is not running or at least
not listening on the port you are expecting

ZK connect seems to be
localhost:2181

Do you have zookeeper running and is it listening on 2181?
Or any fancy firewall rules on the box preventing the connection?

Best Jan

Am Mo., 6. Sept. 2021 um 10:04 Uhr schrieb Antoine DUBOIS
<an...@cc.in2p3.fr>:
>
> Hello Alan,
> Thank you for your answer,
> I'm pretty sure I've respected the guide provided and did not set any discovery service.
> However I'm trying to setup ranger as well and though I didn't though it could cause the issue, after futher investigation I suspect ranger might be causing the issue.
> Hadoop 3.1.4
> Hive 3.1.2
> Ranger 2.1.0
> Tez 0.9.2
> It might come from imcompatibles version but I cannot find a proper compatibility matrix for all hadoop ecosystem.
> Thank you very much.
>
> Antoine
>
> ________________________________
> De: "Alan Gates" <al...@gmail.com>
> À: user@hive.apache.org
> Envoyé: Vendredi 3 Septembre 2021 17:51:41
> Objet: Re: Question regarding lock manager
>
> You do not need ZooKeeper to use ACID in Hive.  The first thing I would check is that you have configured your system as described on this page: https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions.  Also, make sure you have not set hive.lock.manager to zookeeper.
>
> There are other features in Hive that can optionally use ZK, such as the discovery service for HiveServer2, you'll want to make sure this isn't coming from there rather than from the transaction system.
>
> Alan.
>
> On Thu, Sep 2, 2021 at 7:46 AM Antoine DUBOIS <an...@cc.in2p3.fr> wrote:
>>
>>
>> Hello,
>> I'm trying to configure ACID hive in a kerberos environment with :
>> Hadoop 3.1.4 deployed in HA considered working
>> and now I'm trying to setup hive with remote metastore and ACID configuration.
>> I may misunderstand what is written in documentation https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions , but to me it's cleary stated that DbTxnManager do not need zookeeper to run properly,
>> However it seems it in fact needs a zookeeper running somewhere as I have several attempt to connect to a local zookeeper instance:
>> 2021-09-02T15:22:21,708 INFO [main-SendThread(localhost:2181)] client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
>> 2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost:2181)] client.ZooKeeperSaslClient: creating sasl client: client=h*******;service=zookeeper;serviceHostname=localhost
>> 2021-09-02T15:22:21,709 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient'
>> 2021-09-02T15:22:21,710 WARN [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
>> java.net.ConnectException: Connection refused
>>
>> Could you please help me to understand properly the documentation ? Do I need or not a zookeeper instance if using DbTxnManager and if so, why it isn't stated precisely and explicitely in the documentation ?
>> I hope you have the best of day.
>>
>> Antoine DUBOIS
>
>

Re: Question regarding lock manager

Posted by Antoine DUBOIS <an...@cc.in2p3.fr>.
Hello Alan, 
Thank you for your answer, 
I'm pretty sure I've respected the guide provided and did not set any discovery service. 
However I'm trying to setup ranger as well and though I didn't though it could cause the issue, after futher investigation I suspect ranger might be causing the issue. 
Hadoop 3.1.4 
Hive 3.1.2 
Ranger 2.1.0 
Tez 0.9.2 
It might come from imcompatibles version but I cannot find a proper compatibility matrix for all hadoop ecosystem. 
Thank you very much. 

Antoine 


De: "Alan Gates" <al...@gmail.com> 
À: user@hive.apache.org 
Envoyé: Vendredi 3 Septembre 2021 17:51:41 
Objet: Re: Question regarding lock manager 

You do not need ZooKeeper to use ACID in Hive. The first thing I would check is that you have configured your system as described on this page: [ https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ] . Also, make sure you have not set hive.lock.manager to zookeeper. 

There are other features in Hive that can optionally use ZK, such as the discovery service for HiveServer2, you'll want to make sure this isn't coming from there rather than from the transaction system. 

Alan. 

On Thu, Sep 2, 2021 at 7:46 AM Antoine DUBOIS < [ mailto:antoine.dubois@cc.in2p3.fr | antoine.dubois@cc.in2p3.fr ] > wrote: 




Hello, 
I'm trying to configure ACID hive in a kerberos environment with : 
Hadoop 3.1.4 deployed in HA considered working 
and now I'm trying to setup hive with remote metastore and ACID configuration. 
I may misunderstand what is written in documentation [ https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ] , but to me it's cleary stated that DbTxnManager do not need zookeeper to run properly, 
However it seems it in fact needs a zookeeper running somewhere as I have several attempt to connect to a local zookeeper instance: 
2021-09 -02T 15:22:21,708 INFO [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : Client will use GSSAPI as SASL mechanism. 
2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : creating sasl client: client=h******* ;service=zookeeper;serviceHostname=localhost 
2021-09-02T15:22:21,709 INFO [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Opening socket connection to server localhost/ 127 . 0 . 0 . 1 : 2181 . Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient' 
2021-09-02T15:22:21,710 WARN [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Session 0x0 for server null , unexpected error, closing socket connection and attempting reconnect 
java.net.ConnectException : Connection refused 

Could you please help me to understand properly the documentation ? Do I need or not a zookeeper instance if using DbTxnManager and if so, why it isn't stated precisely and explicitely in the documentation ? 
I hope you have the best of day. 

Antoine DUBOIS 





Re: Question regarding lock manager

Posted by Alan Gates <al...@gmail.com>.
You do not need ZooKeeper to use ACID in Hive.  The first thing I would
check is that you have configured your system as described on this page:
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions.  Also,
make sure you have not set hive.lock.manager to zookeeper.

There are other features in Hive that can optionally use ZK, such as the
discovery service for HiveServer2, you'll want to make sure this isn't
coming from there rather than from the transaction system.

Alan.

On Thu, Sep 2, 2021 at 7:46 AM Antoine DUBOIS <an...@cc.in2p3.fr>
wrote:

>
> Hello,
> I'm trying to configure ACID hive in a kerberos environment with :
> Hadoop 3.1.4 deployed in HA considered working
> and now I'm trying to setup hive with remote metastore and ACID
> configuration.
> I may misunderstand what is written in documentation
> https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions , but
> to me it's cleary stated that DbTxnManager do not need zookeeper to run
> properly,
> However it seems it in fact needs a zookeeper running somewhere as I have
> several attempt to connect to a local zookeeper instance:
> 2021-09-02T15:22:21,708 INFO [main-SendThread(localhost:2181)]
> client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
> 2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost:2181)]
> client.ZooKeeperSaslClient: creating sasl client: client=h*******
> ;service=zookeeper;serviceHostname=localhost
> 2021-09-02T15:22:21,709 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.
> 0.1:2181. Will attempt to SASL-authenticate using Login Context section
> 'HiveZooKeeperClient'
> 2021-09-02T15:22:21,710 WARN [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error,
> closing socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
>
> Could you please help me to understand properly the documentation ? Do I
> need or not a zookeeper instance if using DbTxnManager and if so, why it
> isn't stated precisely and explicitely in the documentation ?
> I hope you have the best of day.
>
> Antoine DUBOIS
>