You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Antoine DUBOIS <an...@cc.in2p3.fr> on 2021/09/02 14:46:16 UTC
Question regarding lock manager
Hello,
I'm trying to configure ACID hive in a kerberos environment with :
Hadoop 3.1.4 deployed in HA considered working
and now I'm trying to setup hive with remote metastore and ACID configuration.
I may misunderstand what is written in documentation https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions , but to me it's cleary stated that DbTxnManager do not need zookeeper to run properly,
However it seems it in fact needs a zookeeper running somewhere as I have several attempt to connect to a local zookeeper instance:
2021-09 -02T 15:22:21,708 INFO [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : Client will use GSSAPI as SASL mechanism.
2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : creating sasl client: client=h******* ;service=zookeeper;serviceHostname=localhost
2021-09-02T15:22:21,709 INFO [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Opening socket connection to server localhost/ 127 . 0 . 0 . 1 : 2181 . Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient'
2021-09-02T15:22:21,710 WARN [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Session 0x0 for server null , unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException : Connection refused
Could you please help me to understand properly the documentation ? Do I need or not a zookeeper instance if using DbTxnManager and if so, why it isn't stated precisely and explicitely in the documentation ?
I hope you have the best of day.
Antoine DUBOIS
Re: Question regarding lock manager
Posted by Antoine DUBOIS <an...@cc.in2p3.fr>.
Thank you for your insight,
however on Hive 3.1.2 code is like this :
public void startPrivilegeSynchonizer (HiveConf hiveConf) throws Exception {
PolicyProviderContainer policyContainer = new PolicyProviderContainer() ;
HiveAuthorizer authorizer = SessionState. get ().getAuthorizerV2() ;
if (authorizer.getHivePolicyProvider() != null ) {
policyContainer.addAuthorizer(authorizer) ;
}
if (hiveConf.get(MetastoreConf.ConfVars. PRE_EVENT_LISTENERS .getVarname()) != null &&
hiveConf.get(MetastoreConf.ConfVars. PRE_EVENT_LISTENERS .getVarname()).contains(
"org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener" ) &&
hiveConf.get(MetastoreConf.ConfVars. HIVE_AUTHORIZATION_MANAGER .getVarname())!= null ) {
List<HiveMetastoreAuthorizationProvider> providers = HiveUtils. getMetaStoreAuthorizeProviderManagers (
hiveConf , HiveConf.ConfVars. HIVE_METASTORE_AUTHORIZATION_MANAGER , SessionState. get ().getAuthenticator()) ;
for (HiveMetastoreAuthorizationProvider provider : providers) {
if (provider.getHivePolicyProvider() != null ) {
policyContainer.addAuthorizationProvider(provider) ;
}
}
}
I see that github master branch has a check with hive.privilege.synchronizer however this change seems very recent and there's no reference to such key in documentation either.
Thank you for the tips ;-)
De: "Rajesh Balamohan" <rb...@apache.org>
À: user@hive.apache.org
Envoyé: Mardi 7 Septembre 2021 00:46:43
Objet: Re: Question regarding lock manager
For the specific code you mentioned, check if you have " hive.privilege.synchronizer" enabled or not. If so, disable it explicitly. PrivSync is needed for populating information_schema.
~Rajesh.B
On Mon, Sep 6, 2021 at 8:04 PM Antoine DUBOIS < [ mailto:antoine.dubois@cc.in2p3.fr | antoine.dubois@cc.in2p3.fr ] > wrote:
Hello all
After some digging and remote java debug,
I found out that in file HiveServer2.java line 1001:
if (policyContainer.size() > 0 ) {
zKClientForPrivSync = startZookeeperClient(hiveConf) ;
String rootNamespace = hiveConf.getVar(HiveConf.ConfVars. HIVE_SERVER2_ZOOKEEPER_NAMESPACE ) ;
In this case policyContainer.size is > 1 has there's an object RangerHiveAuthorizer
In this case I'm a bit puzzled, why hive would need zookeeper has Ranger is supposed to maintains the policy the same for all node ?
Have a very nice day.
De: "Antoine DUBOIS" < [ mailto:antoine.dubois@cc.in2p3.fr | antoine.dubois@cc.in2p3.fr ] >
À: [ mailto:user@hive.apache.org | user@hive.apache.org ]
Envoyé: Lundi 6 Septembre 2021 10:03:59
Objet: Re: Question regarding lock manager
Hello Alan,
Thank you for your answer,
I'm pretty sure I've respected the guide provided and did not set any discovery service.
However I'm trying to setup ranger as well and though I didn't though it could cause the issue, after futher investigation I suspect ranger might be causing the issue.
Hadoop 3.1.4
Hive 3.1.2
Ranger 2.1.0
Tez 0.9.2
It might come from imcompatibles version but I cannot find a proper compatibility matrix for all hadoop ecosystem.
Thank you very much.
Antoine
De: "Alan Gates" < [ mailto:alanfgates@gmail.com | alanfgates@gmail.com ] >
À: [ mailto:user@hive.apache.org | user@hive.apache.org ]
Envoyé: Vendredi 3 Septembre 2021 17:51:41
Objet: Re: Question regarding lock manager
You do not need ZooKeeper to use ACID in Hive. The first thing I would check is that you have configured your system as described on this page: [ https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ] . Also, make sure you have not set hive.lock.manager to zookeeper.
There are other features in Hive that can optionally use ZK, such as the discovery service for HiveServer2, you'll want to make sure this isn't coming from there rather than from the transaction system.
Alan.
On Thu, Sep 2, 2021 at 7:46 AM Antoine DUBOIS < [ mailto:antoine.dubois@cc.in2p3.fr | antoine.dubois@cc.in2p3.fr ] > wrote:
BQ_BEGIN
Hello,
I'm trying to configure ACID hive in a kerberos environment with :
Hadoop 3.1.4 deployed in HA considered working
and now I'm trying to setup hive with remote metastore and ACID configuration.
I may misunderstand what is written in documentation [ https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ] , but to me it's cleary stated that DbTxnManager do not need zookeeper to run properly,
However it seems it in fact needs a zookeeper running somewhere as I have several attempt to connect to a local zookeeper instance:
2021-09 -02T 15:22:21,708 INFO [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : Client will use GSSAPI as SASL mechanism.
2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : creating sasl client: client=h******* ;service=zookeeper;serviceHostname=localhost
2021-09-02T15:22:21,709 INFO [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Opening socket connection to server localhost/ 127 . 0 . 0 . 1 : 2181 . Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient'
2021-09-02T15:22:21,710 WARN [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Session 0x0 for server null , unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException : Connection refused
Could you please help me to understand properly the documentation ? Do I need or not a zookeeper instance if using DbTxnManager and if so, why it isn't stated precisely and explicitely in the documentation ?
I hope you have the best of day.
Antoine DUBOIS
BQ_END
Re: Question regarding lock manager
Posted by Rajesh Balamohan <rb...@apache.org>.
For the specific code you mentioned, check if you have
"hive.privilege.synchronizer"
enabled or not. If so, disable it explicitly. PrivSync is needed for
populating information_schema.
~Rajesh.B
On Mon, Sep 6, 2021 at 8:04 PM Antoine DUBOIS <an...@cc.in2p3.fr>
wrote:
> Hello all
> After some digging and remote java debug,
> I found out that in file HiveServer2.java line 1001:
>
> if (policyContainer.size() > 0) {
> zKClientForPrivSync = startZookeeperClient(hiveConf);
> String rootNamespace = hiveConf.getVar(HiveConf.ConfVars.HIVE_SERVER2_ZOOKEEPER_NAMESPACE);
>
> In this case policyContainer.size is > 1 has there's an object
> RangerHiveAuthorizer
>
> In this case I'm a bit puzzled, why hive would need zookeeper has Ranger
> is supposed to maintains the policy the same for all node ?
>
> Have a very nice day.
>
> ------------------------------
> *De: *"Antoine DUBOIS" <an...@cc.in2p3.fr>
> *À: *user@hive.apache.org
> *Envoyé: *Lundi 6 Septembre 2021 10:03:59
> *Objet: *Re: Question regarding lock manager
>
> Hello Alan,
> Thank you for your answer,
> I'm pretty sure I've respected the guide provided and did not set any
> discovery service.
> However I'm trying to setup ranger as well and though I didn't though it
> could cause the issue, after futher investigation I suspect ranger might be
> causing the issue.
> Hadoop 3.1.4
> Hive 3.1.2
> Ranger 2.1.0
> Tez 0.9.2
> It might come from imcompatibles version but I cannot find a proper
> compatibility matrix for all hadoop ecosystem.
> Thank you very much.
>
> Antoine
>
> ------------------------------
> *De: *"Alan Gates" <al...@gmail.com>
> *À: *user@hive.apache.org
> *Envoyé: *Vendredi 3 Septembre 2021 17:51:41
> *Objet: *Re: Question regarding lock manager
>
> You do not need ZooKeeper to use ACID in Hive. The first thing I would
> check is that you have configured your system as described on this page:
> https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions.
> Also, make sure you have not set hive.lock.manager to zookeeper.
>
> There are other features in Hive that can optionally use ZK, such as the
> discovery service for HiveServer2, you'll want to make sure this isn't
> coming from there rather than from the transaction system.
>
> Alan.
>
> On Thu, Sep 2, 2021 at 7:46 AM Antoine DUBOIS <an...@cc.in2p3.fr>
> wrote:
>
>>
>> Hello,
>> I'm trying to configure ACID hive in a kerberos environment with :
>> Hadoop 3.1.4 deployed in HA considered working
>> and now I'm trying to setup hive with remote metastore and ACID
>> configuration.
>> I may misunderstand what is written in documentation
>> https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions , but
>> to me it's cleary stated that DbTxnManager do not need zookeeper to run
>> properly,
>> However it seems it in fact needs a zookeeper running somewhere as I have
>> several attempt to connect to a local zookeeper instance:
>> 2021-09-02T15:22:21,708 INFO [main-SendThread(localhost:2181)]
>> client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
>> 2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost:2181)]
>> client.ZooKeeperSaslClient: creating sasl client: client=h*******
>> ;service=zookeeper;serviceHostname=localhost
>> 2021-09-02T15:22:21,709 INFO [main-SendThread(localhost:2181)]
>> zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0
>> .0.1:2181. Will attempt to SASL-authenticate using Login Context section
>> 'HiveZooKeeperClient'
>> 2021-09-02T15:22:21,710 WARN [main-SendThread(localhost:2181)]
>> zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error,
>> closing socket connection and attempting reconnect
>> java.net.ConnectException: Connection refused
>>
>> Could you please help me to understand properly the documentation ? Do I
>> need or not a zookeeper instance if using DbTxnManager and if so, why it
>> isn't stated precisely and explicitely in the documentation ?
>> I hope you have the best of day.
>>
>> Antoine DUBOIS
>>
>
>
Re: Question regarding lock manager
Posted by Antoine DUBOIS <an...@cc.in2p3.fr>.
Hello all
After some digging and remote java debug,
I found out that in file HiveServer2.java line 1001:
if (policyContainer.size() > 0 ) {
zKClientForPrivSync = startZookeeperClient(hiveConf) ;
String rootNamespace = hiveConf.getVar(HiveConf.ConfVars. HIVE_SERVER2_ZOOKEEPER_NAMESPACE ) ;
In this case policyContainer.size is > 1 has there's an object RangerHiveAuthorizer
In this case I'm a bit puzzled, why hive would need zookeeper has Ranger is supposed to maintains the policy the same for all node ?
Have a very nice day.
De: "Antoine DUBOIS" <an...@cc.in2p3.fr>
À: user@hive.apache.org
Envoyé: Lundi 6 Septembre 2021 10:03:59
Objet: Re: Question regarding lock manager
Hello Alan,
Thank you for your answer,
I'm pretty sure I've respected the guide provided and did not set any discovery service.
However I'm trying to setup ranger as well and though I didn't though it could cause the issue, after futher investigation I suspect ranger might be causing the issue.
Hadoop 3.1.4
Hive 3.1.2
Ranger 2.1.0
Tez 0.9.2
It might come from imcompatibles version but I cannot find a proper compatibility matrix for all hadoop ecosystem.
Thank you very much.
Antoine
De: "Alan Gates" <al...@gmail.com>
À: user@hive.apache.org
Envoyé: Vendredi 3 Septembre 2021 17:51:41
Objet: Re: Question regarding lock manager
You do not need ZooKeeper to use ACID in Hive. The first thing I would check is that you have configured your system as described on this page: [ https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ] . Also, make sure you have not set hive.lock.manager to zookeeper.
There are other features in Hive that can optionally use ZK, such as the discovery service for HiveServer2, you'll want to make sure this isn't coming from there rather than from the transaction system.
Alan.
On Thu, Sep 2, 2021 at 7:46 AM Antoine DUBOIS < [ mailto:antoine.dubois@cc.in2p3.fr | antoine.dubois@cc.in2p3.fr ] > wrote:
Hello,
I'm trying to configure ACID hive in a kerberos environment with :
Hadoop 3.1.4 deployed in HA considered working
and now I'm trying to setup hive with remote metastore and ACID configuration.
I may misunderstand what is written in documentation [ https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ] , but to me it's cleary stated that DbTxnManager do not need zookeeper to run properly,
However it seems it in fact needs a zookeeper running somewhere as I have several attempt to connect to a local zookeeper instance:
2021-09 -02T 15:22:21,708 INFO [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : Client will use GSSAPI as SASL mechanism.
2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : creating sasl client: client=h******* ;service=zookeeper;serviceHostname=localhost
2021-09-02T15:22:21,709 INFO [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Opening socket connection to server localhost/ 127 . 0 . 0 . 1 : 2181 . Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient'
2021-09-02T15:22:21,710 WARN [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Session 0x0 for server null , unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException : Connection refused
Could you please help me to understand properly the documentation ? Do I need or not a zookeeper instance if using DbTxnManager and if so, why it isn't stated precisely and explicitely in the documentation ?
I hope you have the best of day.
Antoine DUBOIS
Re: Question regarding lock manager
Posted by Antoine DUBOIS <an...@cc.in2p3.fr>.
No,
and for now i don't plan to make use of zookeeper.
And as you can see in my previous email, I now understand why Hive is trying to connect to any zookeeper instance despite my configuration clearly state that it should be used.
In this case the presence of RangerPolicyManager is responsible for the use of zookeeper by hiveserver2. However why using ZooKeeper is useful or relevant is another story I can't tell.
Regards
----- Mail original -----
De: "Jan Fili" <ja...@gmail.com>
À: user@hive.apache.org
Envoyé: Lundi 6 Septembre 2021 16:33:28
Objet: Re: Question regarding lock manager
Hi just chipping in here real quick.
java.net.ConnectException: Connection refused
is a pretty good indicator that Zookeeper is not running or at least
not listening on the port you are expecting
ZK connect seems to be
localhost:2181
Do you have zookeeper running and is it listening on 2181?
Or any fancy firewall rules on the box preventing the connection?
Best Jan
Am Mo., 6. Sept. 2021 um 10:04 Uhr schrieb Antoine DUBOIS
<an...@cc.in2p3.fr>:
>
> Hello Alan,
> Thank you for your answer,
> I'm pretty sure I've respected the guide provided and did not set any discovery service.
> However I'm trying to setup ranger as well and though I didn't though it could cause the issue, after futher investigation I suspect ranger might be causing the issue.
> Hadoop 3.1.4
> Hive 3.1.2
> Ranger 2.1.0
> Tez 0.9.2
> It might come from imcompatibles version but I cannot find a proper compatibility matrix for all hadoop ecosystem.
> Thank you very much.
>
> Antoine
>
> ________________________________
> De: "Alan Gates" <al...@gmail.com>
> À: user@hive.apache.org
> Envoyé: Vendredi 3 Septembre 2021 17:51:41
> Objet: Re: Question regarding lock manager
>
> You do not need ZooKeeper to use ACID in Hive. The first thing I would check is that you have configured your system as described on this page: https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions. Also, make sure you have not set hive.lock.manager to zookeeper.
>
> There are other features in Hive that can optionally use ZK, such as the discovery service for HiveServer2, you'll want to make sure this isn't coming from there rather than from the transaction system.
>
> Alan.
>
> On Thu, Sep 2, 2021 at 7:46 AM Antoine DUBOIS <an...@cc.in2p3.fr> wrote:
>>
>>
>> Hello,
>> I'm trying to configure ACID hive in a kerberos environment with :
>> Hadoop 3.1.4 deployed in HA considered working
>> and now I'm trying to setup hive with remote metastore and ACID configuration.
>> I may misunderstand what is written in documentation https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions , but to me it's cleary stated that DbTxnManager do not need zookeeper to run properly,
>> However it seems it in fact needs a zookeeper running somewhere as I have several attempt to connect to a local zookeeper instance:
>> 2021-09-02T15:22:21,708 INFO [main-SendThread(localhost:2181)] client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
>> 2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost:2181)] client.ZooKeeperSaslClient: creating sasl client: client=h*******;service=zookeeper;serviceHostname=localhost
>> 2021-09-02T15:22:21,709 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient'
>> 2021-09-02T15:22:21,710 WARN [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
>> java.net.ConnectException: Connection refused
>>
>> Could you please help me to understand properly the documentation ? Do I need or not a zookeeper instance if using DbTxnManager and if so, why it isn't stated precisely and explicitely in the documentation ?
>> I hope you have the best of day.
>>
>> Antoine DUBOIS
>
>
Re: Question regarding lock manager
Posted by Jan Fili <ja...@gmail.com>.
Hi just chipping in here real quick.
java.net.ConnectException: Connection refused
is a pretty good indicator that Zookeeper is not running or at least
not listening on the port you are expecting
ZK connect seems to be
localhost:2181
Do you have zookeeper running and is it listening on 2181?
Or any fancy firewall rules on the box preventing the connection?
Best Jan
Am Mo., 6. Sept. 2021 um 10:04 Uhr schrieb Antoine DUBOIS
<an...@cc.in2p3.fr>:
>
> Hello Alan,
> Thank you for your answer,
> I'm pretty sure I've respected the guide provided and did not set any discovery service.
> However I'm trying to setup ranger as well and though I didn't though it could cause the issue, after futher investigation I suspect ranger might be causing the issue.
> Hadoop 3.1.4
> Hive 3.1.2
> Ranger 2.1.0
> Tez 0.9.2
> It might come from imcompatibles version but I cannot find a proper compatibility matrix for all hadoop ecosystem.
> Thank you very much.
>
> Antoine
>
> ________________________________
> De: "Alan Gates" <al...@gmail.com>
> À: user@hive.apache.org
> Envoyé: Vendredi 3 Septembre 2021 17:51:41
> Objet: Re: Question regarding lock manager
>
> You do not need ZooKeeper to use ACID in Hive. The first thing I would check is that you have configured your system as described on this page: https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions. Also, make sure you have not set hive.lock.manager to zookeeper.
>
> There are other features in Hive that can optionally use ZK, such as the discovery service for HiveServer2, you'll want to make sure this isn't coming from there rather than from the transaction system.
>
> Alan.
>
> On Thu, Sep 2, 2021 at 7:46 AM Antoine DUBOIS <an...@cc.in2p3.fr> wrote:
>>
>>
>> Hello,
>> I'm trying to configure ACID hive in a kerberos environment with :
>> Hadoop 3.1.4 deployed in HA considered working
>> and now I'm trying to setup hive with remote metastore and ACID configuration.
>> I may misunderstand what is written in documentation https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions , but to me it's cleary stated that DbTxnManager do not need zookeeper to run properly,
>> However it seems it in fact needs a zookeeper running somewhere as I have several attempt to connect to a local zookeeper instance:
>> 2021-09-02T15:22:21,708 INFO [main-SendThread(localhost:2181)] client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
>> 2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost:2181)] client.ZooKeeperSaslClient: creating sasl client: client=h*******;service=zookeeper;serviceHostname=localhost
>> 2021-09-02T15:22:21,709 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient'
>> 2021-09-02T15:22:21,710 WARN [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
>> java.net.ConnectException: Connection refused
>>
>> Could you please help me to understand properly the documentation ? Do I need or not a zookeeper instance if using DbTxnManager and if so, why it isn't stated precisely and explicitely in the documentation ?
>> I hope you have the best of day.
>>
>> Antoine DUBOIS
>
>
Re: Question regarding lock manager
Posted by Antoine DUBOIS <an...@cc.in2p3.fr>.
Hello Alan,
Thank you for your answer,
I'm pretty sure I've respected the guide provided and did not set any discovery service.
However I'm trying to setup ranger as well and though I didn't though it could cause the issue, after futher investigation I suspect ranger might be causing the issue.
Hadoop 3.1.4
Hive 3.1.2
Ranger 2.1.0
Tez 0.9.2
It might come from imcompatibles version but I cannot find a proper compatibility matrix for all hadoop ecosystem.
Thank you very much.
Antoine
De: "Alan Gates" <al...@gmail.com>
À: user@hive.apache.org
Envoyé: Vendredi 3 Septembre 2021 17:51:41
Objet: Re: Question regarding lock manager
You do not need ZooKeeper to use ACID in Hive. The first thing I would check is that you have configured your system as described on this page: [ https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ] . Also, make sure you have not set hive.lock.manager to zookeeper.
There are other features in Hive that can optionally use ZK, such as the discovery service for HiveServer2, you'll want to make sure this isn't coming from there rather than from the transaction system.
Alan.
On Thu, Sep 2, 2021 at 7:46 AM Antoine DUBOIS < [ mailto:antoine.dubois@cc.in2p3.fr | antoine.dubois@cc.in2p3.fr ] > wrote:
Hello,
I'm trying to configure ACID hive in a kerberos environment with :
Hadoop 3.1.4 deployed in HA considered working
and now I'm trying to setup hive with remote metastore and ACID configuration.
I may misunderstand what is written in documentation [ https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ] , but to me it's cleary stated that DbTxnManager do not need zookeeper to run properly,
However it seems it in fact needs a zookeeper running somewhere as I have several attempt to connect to a local zookeeper instance:
2021-09 -02T 15:22:21,708 INFO [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : Client will use GSSAPI as SASL mechanism.
2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost: 2181 )] client.ZooKeeperSaslClient : creating sasl client: client=h******* ;service=zookeeper;serviceHostname=localhost
2021-09-02T15:22:21,709 INFO [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Opening socket connection to server localhost/ 127 . 0 . 0 . 1 : 2181 . Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient'
2021-09-02T15:22:21,710 WARN [main-SendThread(localhost: 2181 )] zookeeper.ClientCnxn : Session 0x0 for server null , unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException : Connection refused
Could you please help me to understand properly the documentation ? Do I need or not a zookeeper instance if using DbTxnManager and if so, why it isn't stated precisely and explicitely in the documentation ?
I hope you have the best of day.
Antoine DUBOIS
Re: Question regarding lock manager
Posted by Alan Gates <al...@gmail.com>.
You do not need ZooKeeper to use ACID in Hive. The first thing I would
check is that you have configured your system as described on this page:
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions. Also,
make sure you have not set hive.lock.manager to zookeeper.
There are other features in Hive that can optionally use ZK, such as the
discovery service for HiveServer2, you'll want to make sure this isn't
coming from there rather than from the transaction system.
Alan.
On Thu, Sep 2, 2021 at 7:46 AM Antoine DUBOIS <an...@cc.in2p3.fr>
wrote:
>
> Hello,
> I'm trying to configure ACID hive in a kerberos environment with :
> Hadoop 3.1.4 deployed in HA considered working
> and now I'm trying to setup hive with remote metastore and ACID
> configuration.
> I may misunderstand what is written in documentation
> https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions , but
> to me it's cleary stated that DbTxnManager do not need zookeeper to run
> properly,
> However it seems it in fact needs a zookeeper running somewhere as I have
> several attempt to connect to a local zookeeper instance:
> 2021-09-02T15:22:21,708 INFO [main-SendThread(localhost:2181)]
> client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
> 2021-09-02T15:22:21,708 DEBUG [main-SendThread(localhost:2181)]
> client.ZooKeeperSaslClient: creating sasl client: client=h*******
> ;service=zookeeper;serviceHostname=localhost
> 2021-09-02T15:22:21,709 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.
> 0.1:2181. Will attempt to SASL-authenticate using Login Context section
> 'HiveZooKeeperClient'
> 2021-09-02T15:22:21,710 WARN [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error,
> closing socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
>
> Could you please help me to understand properly the documentation ? Do I
> need or not a zookeeper instance if using DbTxnManager and if so, why it
> isn't stated precisely and explicitely in the documentation ?
> I hope you have the best of day.
>
> Antoine DUBOIS
>