You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Aparajita Singh <ap...@gmail.com> on 2020/06/10 12:08:15 UTC
Zookeeper client fails during SASL authentication
Hi,
I am trying to migrate an unauthenticated zookeeper cluster to a kerberos
authenticated one. For the time being SSL is disabled. I have configured
the server and client as described below but when SASL is enabled I am
unable to retreive data using zookeeper shell client from the zookeeper
server. Could I get some help in understanding why this is failing?
server.log snippet
2020-06-10 17:09:01,263 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection
from /127.0.0.1:44994
2020-06-10 17:09:01,264 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command from /
127.0.0.1:44994
2020-06-10 17:09:01,265 - INFO [Thread-5:NIOServerCnxn@1007] - Closed
socket connection for client /127.0.0.1:44994 (no session established for
client)
2020-06-10 17:09:26,647 - INFO [main:Environment@100] - Client
environment:zookeeper.version=3.4.6-169--1, built on 02/10/2016 05:49 GMT
2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client environment:
host.name=stage-kdc-zk-ivy
2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client
environment:java.version=1.8.0_172
2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.vendor=Oracle Corporation
2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.home=/usr/lib/jvm/oracle-java8-jdk-amd64/jre
2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.class.path=/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/classes:/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../src/java/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/conf::/usr/hdp/2.4.0.0-169/zookeeper/conf:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper.jar:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/share/zookeeper/*
2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.io.tmpdir=/tmp
2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.compiler=<NA>
2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client environment:
os.name=Linux
2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
environment:os.arch=amd64
2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
environment:os.version=4.9.0-9-amd64
2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client environment:
user.name=root
2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
environment:user.home=/root
2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
environment:user.dir=/home/aparajita.singh
2020-06-10 17:09:26,653 - INFO [main:ZooKeeper@438] - Initiating client
connection, connectString=stage-kdc-zk-ivy sessionTimeout=30000
watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@379619aa
2020-06-10 17:09:26,752 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):Login@293] - successfully logged in.
2020-06-10 17:09:26,753 - INFO [Thread-0:Login$1@127] - TGT refresh thread
started.
2020-06-10 17:09:26,757 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ZooKeeperSaslClient$1@285] - Client
will use GSSAPI as SASL mechanism.
2020-06-10 17:09:26,758 - INFO [Thread-0:Login@301] - TGT valid starting
at: Wed Jun 10 15:17:21 IST 2020
2020-06-10 17:09:26,758 - INFO [Thread-0:Login@302] - TGT expires:
Thu Jun 11 15:17:21 IST 2020
2020-06-10 17:09:26,758 - INFO [Thread-0:Login$1@181] - TGT refresh
sleeping until: Thu Jun 11 11:17:04 IST 2020
2020-06-10 17:09:26,799 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1019] -
Opening socket connection to server stage-kdc-zk-ivy/10.33.203.225:2181.
Will attempt to SASL-authenticate using Login Context section 'Client'
2020-06-10 17:09:26,854 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection
from /10.33.203.225:45018
2020-06-10 17:09:26,854 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@864] - Socket
connection established to stage-kdc-zk-ivy/10.33.203.225:2181, initiating
session
2020-06-10 17:09:26,856 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish
new session at /10.33.203.225:45018
2020-06-10 17:09:26,859 - INFO [CommitProcessor:88:ZooKeeperServer@617] -
Established session 0x58729e0540980002 with negotiated timeout 30000 for
client /10.33.203.225:45018
2020-06-10 17:09:26,861 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1279] -
Session establishment complete on server stage-kdc-zk-ivy/10.33.203.225:2181,
sessionid = 0x58729e0540980002, negotiated timeout = 30000
2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969] - Client failed to SASL
authenticate: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: Failure unspecified at GSS-API level (Mechanism
level: Invalid argument (400) - Cannot find key of appropriate type to
decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)]
2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@975] - Closing client connection due
to SASL authentication failure.
2020-06-10 17:09:27,007 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
client /10.33.203.225:45018 which had sessionid 0x58729e0540980002
2020-06-10 17:09:27,008 - ERROR [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@178] - Unexpected Exception:
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
at
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
at
org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
at
org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer.java:936)
at
org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:373)
at
org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:200)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:244)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:748)
2020-06-10 17:09:27,008 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1142] -
Unable to read additional data from server sessionid 0x58729e0540980002,
likely server has closed socket, closing socket connection and attempting
reconnect
2020-06-10 17:09:27,008 - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@346] - Exception causing close of
session 0x58729e0540980002 due to java.nio.channels.CancelledKeyException
2020-06-10 17:10:01,317 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection
from /127.0.0.1:45004
2020-06-10 17:10:01,318 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command from /
127.0.0.1:45004
zookeeper shell client output
aparajita.singh@stage-kdc-zk-ivy:~$ sudo
/usr/hdp/2.4.0.0-169/zookeeper/bin/zookeeper-client -server
stage-kdc-zk-ivy get /test2
log4j:WARN Large window sizes are not allowed.
log4j:WARN MaxIndex reduced to 13.
Connecting to stage-kdc-zk-ivy
Debug is true storeKey false useTicketCache true useKeyTab true
doNotPrompt true ticketCache is /tmp/krb5cc_0 isInitiator true KeyTab is
/etc/krb5.keytab refreshKrb5Config is false principal is
zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka tryFirstPass is false
useFirstPass is false storePass is false clearPass is false
Acquire TGT from Cache
Principal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka
null credentials from Ticket Cache
principal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka
Will use keytab
Commit Succeeded
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
WATCHER::
WatchedEvent state:Disconnected type:None path:null
Exception in thread "main"
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /test2
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:717)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:354)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)
zoo.cfg
#setACL=False
autopurge.snapRetainCount=30
tickTime=2000
dataDir=/grid/1/var/lib/zookeeper
zookeeper_jmx_port=9009
initLimit=100
syncLimit=5
autopurge.purgeInterval=24
clientPort=2181
globalOutstandingLimit=5000
maxClientCnxns=2000
server.99=stage-kdc-zk-harley:2888:3888
server.88=stage-kdc-zk-ivy:2888:3888
server.77=stage-kdc-zk-2face:2888:3888
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=true
quorum.auth.serverRequireSasl=true
quorum.auth.kerberos.servicePrincipal=host/stage-kdc-zk-ivy@stage.fdp.kafka
quorum.cnxn.threads.size=20
java.env
SERVER_JVMFLAGS="${SERVER_JVMFLAGS}
-Djava.security.auth.login.config=/home/aparajita.singh/jaas/jaas.conf
-Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
-Dsun.security.krb5.debug=true"
CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS}
-Djava.security.auth.login.config=/home/aparajita.singh/jaas/client.conf
-Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
-Dsun.security.krb5.debug=true"
/home/aparajita.singh/jaas/jaas.conf
// Zookeeper server authentication
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
useTicketCache=false
//ticketCache="/tmp/krb5cc_0"
renewTicket=true
doNotPrompt=true
debug=true
keyTab="/etc/krb5.keytab"
serviceName="host"
principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
};
// Zookeeper quorum server authentication
QuorumServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
useTicketCache=false
//ticketCache="/tmp/krb5cc_0"
renewTicket=true
doNotPrompt=true
debug=true
keyTab="/etc/krb5.keytab"
serviceName="host"
principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
};
// Zookeeper learner authentication
QuorumLearner {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
useTicketCache=false
//ticketCache="/tmp/krb5cc_0"
renewTicket=true
doNotPrompt=true
debug=true
keyTab="/etc/krb5.keytab"
serviceName="host"
principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
};
/home/aparajita.singh/jaas/client.conf
// Zookeeper client authentication
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
useTicketCache=true
ticketCache="/tmp/krb5cc_0"
renewTicket=true
doNotPrompt=true
debug=true
keyTab="/etc/krb5.keytab"
serviceName="zookeeper"
principal="zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka";
};
Using kinit command I am able to generate the TGT for both principals. As
per the zookeeper server log, the TGT can be generated as expected. The
keytab file is accessible to all system users for now.
aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab
aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
host/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab
--
Thanks,
Aparajita
Re: Zookeeper client fails during SASL authentication
Posted by Aparajita Singh <ap...@gmail.com>.
Mate,
I tried your suggestions as well,
- make sure you have "Java Cryptography Extension (JCE) Unlimited Strength
Jurisdiction Policy Files" installed (I think you need them for AES256?)
and your java security configs are OK
-- the server and client are able to obtain the TGT individually which
indicates that the issue is not with JCE. the issue is when the client
tries to obtain a service ticket using the TGT it has already obtained.
- run "klist -e -k /etc/krb5.keytab" to see if what encryptions you have
in the keytabs
-- keys are available for both server and client, for AES256 and AES128
- check if you have full export support in JCE by "java KeyLengthDetector"
-- do you maybe have a link to a guide for this? i was not able to find
one through google. running this command as-is gives an error: "Error:
Could not find or load main class KeyLengthDetector"
- Maybe you can try with different encryption types in kerberos
configs / during
keytab generation.
-- i changed encryption type to AES128. generating a TGT using kinit is
working as expected, i.e., it returns an AES128 encrypted ticket. but
zookeeper server and client are still requesting for AES256 encrypted
tickets. i must have missed a config somewhere, i'll try to figure that out.
- trying to use a different java version (latest JDK patches have some known
kerberos backward-incompatibilities)
-- i'll need to explore this further and try it
On Thu, 11 Jun 2020 at 16:58, Aparajita Singh <ap...@gmail.com>
wrote:
> Thanks Mate and Arpit, I'll check out your suggestions.
>
> Jorn,
>
> 1. Did you register the service principal correctly in your AD/KDC?
> 1. yes, the client and principals are registered in KDC. using
> kinit with keytab on the remote server was generating the TGT as expected.
> 2. If AD then did you make sure that the attribute for the user is
> activated to enable AES256 Kerberos auth?
> 1. i'm using KDC, AES256 was enabled by default. i changed the
> kdc.conf file to use only AES128 but zookeeper is still using AES256. i'm
> looking to see if there is a config i have missed out somewhere during
> zookeeper startup which is forcing it to use AES256 always.
> 3. Do you have unlimited crypto policies installed with your JDK?
> 1. yes, i verified this by checking if US_export_policy.jar and
> local_policy.jar files are present in $JAVA_HOME/jre/lib/security/unlimited
> 4. Is the keytab accessible to zk?
> 1. yes
> 5. Did you create keytab with AES256 encryption type?
> 1. yes, keytab had 2 entries for each principal corresponding to
> AES256 and AES128
>
>
> On Thu, 11 Jun 2020 at 16:30, Jörn Franke <jo...@gmail.com> wrote:
>
>> Kerberos can be quite a beast for any application. I managed to use
>> Kerberos authentication for Zookeeper a couple of times. Usually the error
>> messages in Java are meaningless.
>>
>> Did you register the service principal correctly in your AD/KDC?
>>
>> If AD then did you make sure that the attribute for the user is activated
>> to enable AES256 Kerberos auth?
>>
>> Do you have unlimited crypto policies installed with your JDK?
>>
>> Is the keytab accessible to zk?
>>
>> Did you create keytab with AES256 encryption type?
>>
>>
>> > Am 10.06.2020 um 14:35 schrieb Aparajita Singh <
>> aparajita.1194@gmail.com>:
>> >
>> >
>> >>
>> >>
>> >> Hi,
>> >>
>> >> I am trying to migrate an unauthenticated zookeeper cluster to a
>> kerberos
>> >> authenticated one. For the time being SSL is disabled. I have
>> configured
>> >> the server and client as described below but when SASL is enabled I am
>> >> unable to retreive data using zookeeper shell client from the zookeeper
>> >> server. Could I get some help in understanding why this is failing?
>> >>
>> >> server.log snippet
>> >>
>> >> 2020-06-10 17:09:01,263 - INFO [NIOServerCxn.Factory:
>> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
>> >> connection from /127.0.0.1:44994
>> >>
>> >> 2020-06-10 17:09:01,264 - INFO [NIOServerCxn.Factory:
>> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command
>> from /
>> >> 127.0.0.1:44994
>> >>
>> >> 2020-06-10 17:09:01,265 - INFO [Thread-5:NIOServerCnxn@1007] - Closed
>> >> socket connection for client /127.0.0.1:44994 (no session established
>> for
>> >> client)
>> >>
>> >> 2020-06-10 17:09:26,647 - INFO [main:Environment@100] - Client
>> >> environment:zookeeper.version=3.4.6-169--1, built on 02/10/2016 05:49
>> GMT
>> >>
>> >> 2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client
>> >> environment:host.name=stage-kdc-zk-ivy
>> >>
>> >> 2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client
>> >> environment:java.version=1.8.0_172
>> >>
>> >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
>> >> environment:java.vendor=Oracle Corporation
>> >>
>> >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
>> >> environment:java.home=/usr/lib/jvm/oracle-java8-jdk-amd64/jre
>> >>
>> >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
>> >>
>> environment:java.class.path=/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/classes:/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/hdp/
>> 2.4.0.
>> 0-169/zookeeper/bin/../lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../src/java/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/conf::/usr/hdp/2.4.0.0-169/zookeeper/conf:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper.jar:/usr/hdp/
>> 2.4.0.
>> 0-169/zookeeper/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/hdp/
>> 2.4.0.
>> 0-169/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-codec-1.6.jar:/usr/hdp/
>> 2.4.0.
>> 0-169/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/share/zookeeper/*
>> >>
>> >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
>> >>
>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>> >>
>> >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
>> >> environment:java.io.tmpdir=/tmp
>> >>
>> >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
>> >> environment:java.compiler=<NA>
>> >>
>> >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
>> >> environment:os.name=Linux
>> >>
>> >> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
>> >> environment:os.arch=amd64
>> >>
>> >> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
>> >> environment:os.version=4.9.0-9-amd64
>> >>
>> >> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
>> >> environment:user.name=root
>> >>
>> >> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
>> >> environment:user.home=/root
>> >>
>> >> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
>> >> environment:user.dir=/home/aparajita.singh
>> >>
>> >> 2020-06-10 17:09:26,653 - INFO [main:ZooKeeper@438] - Initiating
>> client
>> >> connection, connectString=stage-kdc-zk-ivy sessionTimeout=30000
>> >> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@379619aa
>> >>
>> >> 2020-06-10 17:09:26,752 - INFO
>> >> [main-SendThread(stage-kdc-zk-ivy:2181):Login@293] - successfully
>> logged
>> >> in.
>> >>
>> >> 2020-06-10 17:09:26,753 - INFO [Thread-0:Login$1@127] - TGT refresh
>> >> thread started.
>> >>
>> >> 2020-06-10 17:09:26,757 - INFO
>> >> [main-SendThread(stage-kdc-zk-ivy:2181):ZooKeeperSaslClient$1@285] -
>> >> Client will use GSSAPI as SASL mechanism.
>> >>
>> >> 2020-06-10 17:09:26,758 - INFO [Thread-0:Login@301] - TGT valid
>> starting
>> >> at: Wed Jun 10 15:17:21 IST 2020
>> >>
>> >> 2020-06-10 17:09:26,758 - INFO [Thread-0:Login@302] - TGT expires:
>> >> Thu Jun 11 15:17:21 IST 2020
>> >>
>> >> 2020-06-10 17:09:26,758 - INFO [Thread-0:Login$1@181] - TGT refresh
>> >> sleeping until: Thu Jun 11 11:17:04 IST 2020
>> >>
>> >> 2020-06-10 17:09:26,799 - INFO
>> >> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1019] -
>> >> Opening socket connection to server stage-kdc-zk-ivy/
>> 10.33.203.225:2181.
>> >> Will attempt to SASL-authenticate using Login Context section 'Client'
>> >>
>> >> 2020-06-10 17:09:26,854 - INFO [NIOServerCxn.Factory:
>> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
>> >> connection from /10.33.203.225:45018
>> >>
>> >> 2020-06-10 17:09:26,854 - INFO
>> >> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@864] -
>> >> Socket connection established to stage-kdc-zk-ivy/10.33.203.225:2181,
>> >> initiating session
>> >>
>> >> 2020-06-10 17:09:26,856 - INFO [NIOServerCxn.Factory:
>> >> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
>> >> establish new session at /10.33.203.225:45018
>> >>
>> >> 2020-06-10 17:09:26,859 - INFO [CommitProcessor:88:ZooKeeperServer@617
>> ]
>> >> - Established session 0x58729e0540980002 with negotiated timeout 30000
>> for
>> >> client /10.33.203.225:45018
>> >>
>> >> 2020-06-10 17:09:26,861 - INFO
>> >> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1279] -
>> >> Session establishment complete on server stage-kdc-zk-ivy/
>> >> 10.33.203.225:2181, sessionid = 0x58729e0540980002, negotiated
>> timeout =
>> >> 30000
>> >>
>> >> 2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:
>> >> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969] - Client failed to SASL
>> >> authenticate: javax.security.sasl.SaslException: GSS initiate failed
>> >> [Caused by GSSException: Failure unspecified at GSS-API level
>> (Mechanism
>> >> level: Invalid argument (400) - Cannot find key of appropriate type to
>> >> decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)]
>> >>
>> >> 2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:
>> >> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@975] - Closing client connection
>> due
>> >> to SASL authentication failure.
>> >>
>> >> 2020-06-10 17:09:27,007 - INFO [NIOServerCxn.Factory:
>> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection
>> for
>> >> client /10.33.203.225:45018 which had sessionid 0x58729e0540980002
>> >>
>> >> 2020-06-10 17:09:27,008 - ERROR [NIOServerCxn.Factory:
>> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@178] - Unexpected Exception:
>> >>
>> >> java.nio.channels.CancelledKeyException
>> >>
>> >> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
>> >>
>> >> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
>> >>
>> >> at
>> >>
>> org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
>> >>
>> >> at
>> >>
>> org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
>> >>
>> >> at
>> >>
>> org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer.java:936)
>> >>
>> >> at
>> >>
>> org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:373)
>> >>
>> >> at
>> >>
>> org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:200)
>> >>
>> >> at
>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:244)
>> >>
>> >> at
>> >>
>> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>> >>
>> >> at java.lang.Thread.run(Thread.java:748)
>> >>
>> >> 2020-06-10 17:09:27,008 - INFO
>> >> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1142] -
>> >> Unable to read additional data from server sessionid
>> 0x58729e0540980002,
>> >> likely server has closed socket, closing socket connection and
>> attempting
>> >> reconnect
>> >>
>> >> 2020-06-10 17:09:27,008 - WARN [NIOServerCxn.Factory:
>> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@346] - Exception causing close of
>> >> session 0x58729e0540980002 due to
>> java.nio.channels.CancelledKeyException
>> >>
>> >> 2020-06-10 17:10:01,317 - INFO [NIOServerCxn.Factory:
>> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
>> >> connection from /127.0.0.1:45004
>> >>
>> >> 2020-06-10 17:10:01,318 - INFO [NIOServerCxn.Factory:
>> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command
>> from /
>> >> 127.0.0.1:45004
>> >>
>> >>
>> >>
>> >> zookeeper shell client output
>> >>
>> >> aparajita.singh@stage-kdc-zk-ivy:~$ sudo
>> >> /usr/hdp/2.4.0.0-169/zookeeper/bin/zookeeper-client -server
>> >> stage-kdc-zk-ivy get /test2
>> >>
>> >> log4j:WARN Large window sizes are not allowed.
>> >>
>> >> log4j:WARN MaxIndex reduced to 13.
>> >>
>> >> Connecting to stage-kdc-zk-ivy
>> >>
>> >> Debug is true storeKey false useTicketCache true useKeyTab true
>> >> doNotPrompt true ticketCache is /tmp/krb5cc_0 isInitiator true KeyTab
>> is
>> >> /etc/krb5.keytab refreshKrb5Config is false principal is
>> >> zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka tryFirstPass is false
>> >> useFirstPass is false storePass is false clearPass is false
>> >>
>> >> Acquire TGT from Cache
>> >>
>> >> Principal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka
>> >>
>> >> null credentials from Ticket Cache
>> >>
>> >> principal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka
>> >>
>> >> Will use keytab
>> >>
>> >> Commit Succeeded
>> >>
>> >>
>> >>
>> >> WATCHER::
>> >>
>> >>
>> >> WatchedEvent state:SyncConnected type:None path:null
>> >>
>> >>
>> >> WATCHER::
>> >>
>> >>
>> >> WatchedEvent state:Disconnected type:None path:null
>> >>
>> >> Exception in thread "main"
>> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> >> KeeperErrorCode = ConnectionLoss for /test2
>> >>
>> >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>> >>
>> >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> >>
>> >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>> >>
>> >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)
>> >>
>> >> at
>> org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:717)
>> >>
>> >> at
>> org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)
>> >>
>> >> at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:354)
>> >>
>> >> at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)
>> >>
>> >> zoo.cfg
>> >>
>> >> #setACL=False
>> >>
>> >> autopurge.snapRetainCount=30
>> >>
>> >> tickTime=2000
>> >>
>> >> dataDir=/grid/1/var/lib/zookeeper
>> >>
>> >> zookeeper_jmx_port=9009
>> >>
>> >> initLimit=100
>> >>
>> >> syncLimit=5
>> >>
>> >> autopurge.purgeInterval=24
>> >>
>> >> clientPort=2181
>> >>
>> >> globalOutstandingLimit=5000
>> >>
>> >> maxClientCnxns=2000
>> >>
>> >> server.99=stage-kdc-zk-harley:2888:3888
>> >>
>> >> server.88=stage-kdc-zk-ivy:2888:3888
>> >>
>> >> server.77=stage-kdc-zk-2face:2888:3888
>> >>
>> >>
>> >>
>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>> >>
>> >> requireClientAuthScheme=sasl
>> >>
>> >>
>> >> quorum.auth.enableSasl=true
>> >>
>> >> quorum.auth.learnerRequireSasl=true
>> >>
>> >> quorum.auth.serverRequireSasl=true
>> >>
>> >>
>> quorum.auth.kerberos.servicePrincipal=host/stage-kdc-zk-ivy@stage.fdp.kafka
>> >>
>> >> quorum.cnxn.threads.size=20
>> >>
>> >>
>> >>
>> >> java.env
>> >>
>> >> SERVER_JVMFLAGS="${SERVER_JVMFLAGS}
>> >> -Djava.security.auth.login.config=/home/aparajita.singh/jaas/jaas.conf
>> >>
>> -Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>> >> -Dsun.security.krb5.debug=true"
>> >>
>> >> CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS}
>> >>
>> -Djava.security.auth.login.config=/home/aparajita.singh/jaas/client.conf
>> >>
>> -Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>> >> -Dsun.security.krb5.debug=true"
>> >>
>> >>
>> >> /home/aparajita.singh/jaas/jaas.conf
>> >>
>> >> // Zookeeper server authentication
>> >>
>> >> Server {
>> >>
>> >> com.sun.security.auth.module.Krb5LoginModule required
>> >>
>> >> useKeyTab=true
>> >>
>> >> useTicketCache=false
>> >>
>> >> //ticketCache="/tmp/krb5cc_0"
>> >>
>> >> renewTicket=true
>> >>
>> >> doNotPrompt=true
>> >>
>> >> debug=true
>> >>
>> >> keyTab="/etc/krb5.keytab"
>> >>
>> >> serviceName="host"
>> >>
>> >> principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
>> >>
>> >> };
>> >>
>> >>
>> >> // Zookeeper quorum server authentication
>> >>
>> >> QuorumServer {
>> >>
>> >> com.sun.security.auth.module.Krb5LoginModule required
>> >>
>> >> useKeyTab=true
>> >>
>> >> useTicketCache=false
>> >>
>> >> //ticketCache="/tmp/krb5cc_0"
>> >>
>> >> renewTicket=true
>> >>
>> >> doNotPrompt=true
>> >>
>> >> debug=true
>> >>
>> >> keyTab="/etc/krb5.keytab"
>> >>
>> >> serviceName="host"
>> >>
>> >> principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
>> >>
>> >> };
>> >>
>> >>
>> >> // Zookeeper learner authentication
>> >>
>> >> QuorumLearner {
>> >>
>> >> com.sun.security.auth.module.Krb5LoginModule required
>> >>
>> >> useKeyTab=true
>> >>
>> >> useTicketCache=false
>> >>
>> >> //ticketCache="/tmp/krb5cc_0"
>> >>
>> >> renewTicket=true
>> >>
>> >> doNotPrompt=true
>> >>
>> >> debug=true
>> >>
>> >> keyTab="/etc/krb5.keytab"
>> >>
>> >> serviceName="host"
>> >>
>> >> principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
>> >>
>> >> };
>> >>
>> >>
>> >>
>> >> /home/aparajita.singh/jaas/client.conf
>> >>
>> >> // Zookeeper client authentication
>> >>
>> >> Client {
>> >>
>> >> com.sun.security.auth.module.Krb5LoginModule required
>> >>
>> >> useKeyTab=true
>> >>
>> >> useTicketCache=true
>> >>
>> >> ticketCache="/tmp/krb5cc_0"
>> >>
>> >> renewTicket=true
>> >>
>> >> doNotPrompt=true
>> >>
>> >> debug=true
>> >>
>> >> keyTab="/etc/krb5.keytab"
>> >>
>> >> serviceName="zookeeper"
>> >>
>> >> principal="zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka";
>> >>
>> >> };
>> >>
>> >>
>> >> Using kinit command I am able to generate the TGT for both principals.
>> As
>> >> per the zookeeper server log, the TGT can be generated as expected. The
>> >> keytab file is accessible to all system users for now.
>> >>
>> >> aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
>> >> zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab
>> >>
>> >> aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
>> >> host/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab
>> >>
>> >>
>> >> --
>> >> Thanks,
>> >> Aparajita
>> >>
>>
>
>
> --
> Thanks,
> Aparajita
>
--
Thanks,
Aparajita
Re: Zookeeper client fails during SASL authentication
Posted by Aparajita Singh <ap...@gmail.com>.
Thanks Mate and Arpit, I'll check out your suggestions.
Jorn,
1. Did you register the service principal correctly in your AD/KDC?
1. yes, the client and principals are registered in KDC. using kinit
with keytab on the remote server was generating the TGT as expected.
2. If AD then did you make sure that the attribute for the user is
activated to enable AES256 Kerberos auth?
1. i'm using KDC, AES256 was enabled by default. i changed the
kdc.conf file to use only AES128 but zookeeper is still using AES256. i'm
looking to see if there is a config i have missed out somewhere during
zookeeper startup which is forcing it to use AES256 always.
3. Do you have unlimited crypto policies installed with your JDK?
1. yes, i verified this by checking if US_export_policy.jar and
local_policy.jar files are present in
$JAVA_HOME/jre/lib/security/unlimited
4. Is the keytab accessible to zk?
1. yes
5. Did you create keytab with AES256 encryption type?
1. yes, keytab had 2 entries for each principal corresponding to
AES256 and AES128
On Thu, 11 Jun 2020 at 16:30, Jörn Franke <jo...@gmail.com> wrote:
> Kerberos can be quite a beast for any application. I managed to use
> Kerberos authentication for Zookeeper a couple of times. Usually the error
> messages in Java are meaningless.
>
> Did you register the service principal correctly in your AD/KDC?
>
> If AD then did you make sure that the attribute for the user is activated
> to enable AES256 Kerberos auth?
>
> Do you have unlimited crypto policies installed with your JDK?
>
> Is the keytab accessible to zk?
>
> Did you create keytab with AES256 encryption type?
>
>
> > Am 10.06.2020 um 14:35 schrieb Aparajita Singh <aparajita.1194@gmail.com
> >:
> >
> >
> >>
> >>
> >> Hi,
> >>
> >> I am trying to migrate an unauthenticated zookeeper cluster to a
> kerberos
> >> authenticated one. For the time being SSL is disabled. I have configured
> >> the server and client as described below but when SASL is enabled I am
> >> unable to retreive data using zookeeper shell client from the zookeeper
> >> server. Could I get some help in understanding why this is failing?
> >>
> >> server.log snippet
> >>
> >> 2020-06-10 17:09:01,263 - INFO [NIOServerCxn.Factory:
> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> >> connection from /127.0.0.1:44994
> >>
> >> 2020-06-10 17:09:01,264 - INFO [NIOServerCxn.Factory:
> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command from
> /
> >> 127.0.0.1:44994
> >>
> >> 2020-06-10 17:09:01,265 - INFO [Thread-5:NIOServerCnxn@1007] - Closed
> >> socket connection for client /127.0.0.1:44994 (no session established
> for
> >> client)
> >>
> >> 2020-06-10 17:09:26,647 - INFO [main:Environment@100] - Client
> >> environment:zookeeper.version=3.4.6-169--1, built on 02/10/2016 05:49
> GMT
> >>
> >> 2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client
> >> environment:host.name=stage-kdc-zk-ivy
> >>
> >> 2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client
> >> environment:java.version=1.8.0_172
> >>
> >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
> >> environment:java.vendor=Oracle Corporation
> >>
> >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
> >> environment:java.home=/usr/lib/jvm/oracle-java8-jdk-amd64/jre
> >>
> >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
> >>
> environment:java.class.path=/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/classes:/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/hdp/
> 2.4.0.
> 0-169/zookeeper/bin/../lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../src/java/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/conf::/usr/hdp/2.4.0.0-169/zookeeper/conf:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper.jar:/usr/hdp/
> 2.4.0.
> 0-169/zookeeper/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/hdp/
> 2.4.0.
> 0-169/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-codec-1.6.jar:/usr/hdp/
> 2.4.0.
> 0-169/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/share/zookeeper/*
> >>
> >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
> >>
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> >>
> >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
> >> environment:java.io.tmpdir=/tmp
> >>
> >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
> >> environment:java.compiler=<NA>
> >>
> >> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
> >> environment:os.name=Linux
> >>
> >> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
> >> environment:os.arch=amd64
> >>
> >> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
> >> environment:os.version=4.9.0-9-amd64
> >>
> >> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
> >> environment:user.name=root
> >>
> >> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
> >> environment:user.home=/root
> >>
> >> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
> >> environment:user.dir=/home/aparajita.singh
> >>
> >> 2020-06-10 17:09:26,653 - INFO [main:ZooKeeper@438] - Initiating
> client
> >> connection, connectString=stage-kdc-zk-ivy sessionTimeout=30000
> >> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@379619aa
> >>
> >> 2020-06-10 17:09:26,752 - INFO
> >> [main-SendThread(stage-kdc-zk-ivy:2181):Login@293] - successfully
> logged
> >> in.
> >>
> >> 2020-06-10 17:09:26,753 - INFO [Thread-0:Login$1@127] - TGT refresh
> >> thread started.
> >>
> >> 2020-06-10 17:09:26,757 - INFO
> >> [main-SendThread(stage-kdc-zk-ivy:2181):ZooKeeperSaslClient$1@285] -
> >> Client will use GSSAPI as SASL mechanism.
> >>
> >> 2020-06-10 17:09:26,758 - INFO [Thread-0:Login@301] - TGT valid
> starting
> >> at: Wed Jun 10 15:17:21 IST 2020
> >>
> >> 2020-06-10 17:09:26,758 - INFO [Thread-0:Login@302] - TGT expires:
> >> Thu Jun 11 15:17:21 IST 2020
> >>
> >> 2020-06-10 17:09:26,758 - INFO [Thread-0:Login$1@181] - TGT refresh
> >> sleeping until: Thu Jun 11 11:17:04 IST 2020
> >>
> >> 2020-06-10 17:09:26,799 - INFO
> >> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1019] -
> >> Opening socket connection to server stage-kdc-zk-ivy/10.33.203.225:2181
> .
> >> Will attempt to SASL-authenticate using Login Context section 'Client'
> >>
> >> 2020-06-10 17:09:26,854 - INFO [NIOServerCxn.Factory:
> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> >> connection from /10.33.203.225:45018
> >>
> >> 2020-06-10 17:09:26,854 - INFO
> >> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@864] -
> >> Socket connection established to stage-kdc-zk-ivy/10.33.203.225:2181,
> >> initiating session
> >>
> >> 2020-06-10 17:09:26,856 - INFO [NIOServerCxn.Factory:
> >> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> >> establish new session at /10.33.203.225:45018
> >>
> >> 2020-06-10 17:09:26,859 - INFO [CommitProcessor:88:ZooKeeperServer@617
> ]
> >> - Established session 0x58729e0540980002 with negotiated timeout 30000
> for
> >> client /10.33.203.225:45018
> >>
> >> 2020-06-10 17:09:26,861 - INFO
> >> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1279] -
> >> Session establishment complete on server stage-kdc-zk-ivy/
> >> 10.33.203.225:2181, sessionid = 0x58729e0540980002, negotiated timeout
> =
> >> 30000
> >>
> >> 2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:
> >> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969] - Client failed to SASL
> >> authenticate: javax.security.sasl.SaslException: GSS initiate failed
> >> [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism
> >> level: Invalid argument (400) - Cannot find key of appropriate type to
> >> decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)]
> >>
> >> 2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:
> >> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@975] - Closing client connection
> due
> >> to SASL authentication failure.
> >>
> >> 2020-06-10 17:09:27,007 - INFO [NIOServerCxn.Factory:
> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> >> client /10.33.203.225:45018 which had sessionid 0x58729e0540980002
> >>
> >> 2020-06-10 17:09:27,008 - ERROR [NIOServerCxn.Factory:
> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@178] - Unexpected Exception:
> >>
> >> java.nio.channels.CancelledKeyException
> >>
> >> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
> >>
> >> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
> >>
> >> at
> >>
> org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
> >>
> >> at
> >>
> org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
> >>
> >> at
> >>
> org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer.java:936)
> >>
> >> at
> >>
> org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:373)
> >>
> >> at
> >>
> org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:200)
> >>
> >> at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:244)
> >>
> >> at
> >>
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> >>
> >> at java.lang.Thread.run(Thread.java:748)
> >>
> >> 2020-06-10 17:09:27,008 - INFO
> >> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1142] -
> >> Unable to read additional data from server sessionid 0x58729e0540980002,
> >> likely server has closed socket, closing socket connection and
> attempting
> >> reconnect
> >>
> >> 2020-06-10 17:09:27,008 - WARN [NIOServerCxn.Factory:
> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@346] - Exception causing close of
> >> session 0x58729e0540980002 due to
> java.nio.channels.CancelledKeyException
> >>
> >> 2020-06-10 17:10:01,317 - INFO [NIOServerCxn.Factory:
> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> >> connection from /127.0.0.1:45004
> >>
> >> 2020-06-10 17:10:01,318 - INFO [NIOServerCxn.Factory:
> >> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command from
> /
> >> 127.0.0.1:45004
> >>
> >>
> >>
> >> zookeeper shell client output
> >>
> >> aparajita.singh@stage-kdc-zk-ivy:~$ sudo
> >> /usr/hdp/2.4.0.0-169/zookeeper/bin/zookeeper-client -server
> >> stage-kdc-zk-ivy get /test2
> >>
> >> log4j:WARN Large window sizes are not allowed.
> >>
> >> log4j:WARN MaxIndex reduced to 13.
> >>
> >> Connecting to stage-kdc-zk-ivy
> >>
> >> Debug is true storeKey false useTicketCache true useKeyTab true
> >> doNotPrompt true ticketCache is /tmp/krb5cc_0 isInitiator true KeyTab is
> >> /etc/krb5.keytab refreshKrb5Config is false principal is
> >> zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka tryFirstPass is false
> >> useFirstPass is false storePass is false clearPass is false
> >>
> >> Acquire TGT from Cache
> >>
> >> Principal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka
> >>
> >> null credentials from Ticket Cache
> >>
> >> principal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka
> >>
> >> Will use keytab
> >>
> >> Commit Succeeded
> >>
> >>
> >>
> >> WATCHER::
> >>
> >>
> >> WatchedEvent state:SyncConnected type:None path:null
> >>
> >>
> >> WATCHER::
> >>
> >>
> >> WatchedEvent state:Disconnected type:None path:null
> >>
> >> Exception in thread "main"
> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >> KeeperErrorCode = ConnectionLoss for /test2
> >>
> >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>
> >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>
> >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> >>
> >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)
> >>
> >> at
> org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:717)
> >>
> >> at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)
> >>
> >> at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:354)
> >>
> >> at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)
> >>
> >> zoo.cfg
> >>
> >> #setACL=False
> >>
> >> autopurge.snapRetainCount=30
> >>
> >> tickTime=2000
> >>
> >> dataDir=/grid/1/var/lib/zookeeper
> >>
> >> zookeeper_jmx_port=9009
> >>
> >> initLimit=100
> >>
> >> syncLimit=5
> >>
> >> autopurge.purgeInterval=24
> >>
> >> clientPort=2181
> >>
> >> globalOutstandingLimit=5000
> >>
> >> maxClientCnxns=2000
> >>
> >> server.99=stage-kdc-zk-harley:2888:3888
> >>
> >> server.88=stage-kdc-zk-ivy:2888:3888
> >>
> >> server.77=stage-kdc-zk-2face:2888:3888
> >>
> >>
> >>
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> >>
> >> requireClientAuthScheme=sasl
> >>
> >>
> >> quorum.auth.enableSasl=true
> >>
> >> quorum.auth.learnerRequireSasl=true
> >>
> >> quorum.auth.serverRequireSasl=true
> >>
> >>
> quorum.auth.kerberos.servicePrincipal=host/stage-kdc-zk-ivy@stage.fdp.kafka
> >>
> >> quorum.cnxn.threads.size=20
> >>
> >>
> >>
> >> java.env
> >>
> >> SERVER_JVMFLAGS="${SERVER_JVMFLAGS}
> >> -Djava.security.auth.login.config=/home/aparajita.singh/jaas/jaas.conf
> >>
> -Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> >> -Dsun.security.krb5.debug=true"
> >>
> >> CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS}
> >> -Djava.security.auth.login.config=/home/aparajita.singh/jaas/client.conf
> >>
> -Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> >> -Dsun.security.krb5.debug=true"
> >>
> >>
> >> /home/aparajita.singh/jaas/jaas.conf
> >>
> >> // Zookeeper server authentication
> >>
> >> Server {
> >>
> >> com.sun.security.auth.module.Krb5LoginModule required
> >>
> >> useKeyTab=true
> >>
> >> useTicketCache=false
> >>
> >> //ticketCache="/tmp/krb5cc_0"
> >>
> >> renewTicket=true
> >>
> >> doNotPrompt=true
> >>
> >> debug=true
> >>
> >> keyTab="/etc/krb5.keytab"
> >>
> >> serviceName="host"
> >>
> >> principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
> >>
> >> };
> >>
> >>
> >> // Zookeeper quorum server authentication
> >>
> >> QuorumServer {
> >>
> >> com.sun.security.auth.module.Krb5LoginModule required
> >>
> >> useKeyTab=true
> >>
> >> useTicketCache=false
> >>
> >> //ticketCache="/tmp/krb5cc_0"
> >>
> >> renewTicket=true
> >>
> >> doNotPrompt=true
> >>
> >> debug=true
> >>
> >> keyTab="/etc/krb5.keytab"
> >>
> >> serviceName="host"
> >>
> >> principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
> >>
> >> };
> >>
> >>
> >> // Zookeeper learner authentication
> >>
> >> QuorumLearner {
> >>
> >> com.sun.security.auth.module.Krb5LoginModule required
> >>
> >> useKeyTab=true
> >>
> >> useTicketCache=false
> >>
> >> //ticketCache="/tmp/krb5cc_0"
> >>
> >> renewTicket=true
> >>
> >> doNotPrompt=true
> >>
> >> debug=true
> >>
> >> keyTab="/etc/krb5.keytab"
> >>
> >> serviceName="host"
> >>
> >> principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
> >>
> >> };
> >>
> >>
> >>
> >> /home/aparajita.singh/jaas/client.conf
> >>
> >> // Zookeeper client authentication
> >>
> >> Client {
> >>
> >> com.sun.security.auth.module.Krb5LoginModule required
> >>
> >> useKeyTab=true
> >>
> >> useTicketCache=true
> >>
> >> ticketCache="/tmp/krb5cc_0"
> >>
> >> renewTicket=true
> >>
> >> doNotPrompt=true
> >>
> >> debug=true
> >>
> >> keyTab="/etc/krb5.keytab"
> >>
> >> serviceName="zookeeper"
> >>
> >> principal="zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka";
> >>
> >> };
> >>
> >>
> >> Using kinit command I am able to generate the TGT for both principals.
> As
> >> per the zookeeper server log, the TGT can be generated as expected. The
> >> keytab file is accessible to all system users for now.
> >>
> >> aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
> >> zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab
> >>
> >> aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
> >> host/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab
> >>
> >>
> >> --
> >> Thanks,
> >> Aparajita
> >>
>
--
Thanks,
Aparajita
Re: Zookeeper client fails during SASL authentication
Posted by Jörn Franke <jo...@gmail.com>.
Kerberos can be quite a beast for any application. I managed to use Kerberos authentication for Zookeeper a couple of times. Usually the error messages in Java are meaningless.
Did you register the service principal correctly in your AD/KDC?
If AD then did you make sure that the attribute for the user is activated to enable AES256 Kerberos auth?
Do you have unlimited crypto policies installed with your JDK?
Is the keytab accessible to zk?
Did you create keytab with AES256 encryption type?
> Am 10.06.2020 um 14:35 schrieb Aparajita Singh <ap...@gmail.com>:
>
>
>>
>>
>> Hi,
>>
>> I am trying to migrate an unauthenticated zookeeper cluster to a kerberos
>> authenticated one. For the time being SSL is disabled. I have configured
>> the server and client as described below but when SASL is enabled I am
>> unable to retreive data using zookeeper shell client from the zookeeper
>> server. Could I get some help in understanding why this is failing?
>>
>> server.log snippet
>>
>> 2020-06-10 17:09:01,263 - INFO [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
>> connection from /127.0.0.1:44994
>>
>> 2020-06-10 17:09:01,264 - INFO [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command from /
>> 127.0.0.1:44994
>>
>> 2020-06-10 17:09:01,265 - INFO [Thread-5:NIOServerCnxn@1007] - Closed
>> socket connection for client /127.0.0.1:44994 (no session established for
>> client)
>>
>> 2020-06-10 17:09:26,647 - INFO [main:Environment@100] - Client
>> environment:zookeeper.version=3.4.6-169--1, built on 02/10/2016 05:49 GMT
>>
>> 2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client
>> environment:host.name=stage-kdc-zk-ivy
>>
>> 2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client
>> environment:java.version=1.8.0_172
>>
>> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
>> environment:java.vendor=Oracle Corporation
>>
>> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
>> environment:java.home=/usr/lib/jvm/oracle-java8-jdk-amd64/jre
>>
>> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
>> environment:java.class.path=/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/classes:/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../src/java/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/conf::/usr/hdp/2.4.0.0-169/zookeeper/conf:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper.jar:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/share/zookeeper/*
>>
>> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>
>> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
>> environment:java.io.tmpdir=/tmp
>>
>> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
>> environment:java.compiler=<NA>
>>
>> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
>> environment:os.name=Linux
>>
>> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
>> environment:os.arch=amd64
>>
>> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
>> environment:os.version=4.9.0-9-amd64
>>
>> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
>> environment:user.name=root
>>
>> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
>> environment:user.home=/root
>>
>> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
>> environment:user.dir=/home/aparajita.singh
>>
>> 2020-06-10 17:09:26,653 - INFO [main:ZooKeeper@438] - Initiating client
>> connection, connectString=stage-kdc-zk-ivy sessionTimeout=30000
>> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@379619aa
>>
>> 2020-06-10 17:09:26,752 - INFO
>> [main-SendThread(stage-kdc-zk-ivy:2181):Login@293] - successfully logged
>> in.
>>
>> 2020-06-10 17:09:26,753 - INFO [Thread-0:Login$1@127] - TGT refresh
>> thread started.
>>
>> 2020-06-10 17:09:26,757 - INFO
>> [main-SendThread(stage-kdc-zk-ivy:2181):ZooKeeperSaslClient$1@285] -
>> Client will use GSSAPI as SASL mechanism.
>>
>> 2020-06-10 17:09:26,758 - INFO [Thread-0:Login@301] - TGT valid starting
>> at: Wed Jun 10 15:17:21 IST 2020
>>
>> 2020-06-10 17:09:26,758 - INFO [Thread-0:Login@302] - TGT expires:
>> Thu Jun 11 15:17:21 IST 2020
>>
>> 2020-06-10 17:09:26,758 - INFO [Thread-0:Login$1@181] - TGT refresh
>> sleeping until: Thu Jun 11 11:17:04 IST 2020
>>
>> 2020-06-10 17:09:26,799 - INFO
>> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1019] -
>> Opening socket connection to server stage-kdc-zk-ivy/10.33.203.225:2181.
>> Will attempt to SASL-authenticate using Login Context section 'Client'
>>
>> 2020-06-10 17:09:26,854 - INFO [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
>> connection from /10.33.203.225:45018
>>
>> 2020-06-10 17:09:26,854 - INFO
>> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@864] -
>> Socket connection established to stage-kdc-zk-ivy/10.33.203.225:2181,
>> initiating session
>>
>> 2020-06-10 17:09:26,856 - INFO [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
>> establish new session at /10.33.203.225:45018
>>
>> 2020-06-10 17:09:26,859 - INFO [CommitProcessor:88:ZooKeeperServer@617]
>> - Established session 0x58729e0540980002 with negotiated timeout 30000 for
>> client /10.33.203.225:45018
>>
>> 2020-06-10 17:09:26,861 - INFO
>> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1279] -
>> Session establishment complete on server stage-kdc-zk-ivy/
>> 10.33.203.225:2181, sessionid = 0x58729e0540980002, negotiated timeout =
>> 30000
>>
>> 2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969] - Client failed to SASL
>> authenticate: javax.security.sasl.SaslException: GSS initiate failed
>> [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism
>> level: Invalid argument (400) - Cannot find key of appropriate type to
>> decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)]
>>
>> 2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@975] - Closing client connection due
>> to SASL authentication failure.
>>
>> 2020-06-10 17:09:27,007 - INFO [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
>> client /10.33.203.225:45018 which had sessionid 0x58729e0540980002
>>
>> 2020-06-10 17:09:27,008 - ERROR [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@178] - Unexpected Exception:
>>
>> java.nio.channels.CancelledKeyException
>>
>> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
>>
>> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
>>
>> at
>> org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
>>
>> at
>> org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
>>
>> at
>> org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer.java:936)
>>
>> at
>> org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:373)
>>
>> at
>> org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:200)
>>
>> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:244)
>>
>> at
>> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>>
>> at java.lang.Thread.run(Thread.java:748)
>>
>> 2020-06-10 17:09:27,008 - INFO
>> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1142] -
>> Unable to read additional data from server sessionid 0x58729e0540980002,
>> likely server has closed socket, closing socket connection and attempting
>> reconnect
>>
>> 2020-06-10 17:09:27,008 - WARN [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@346] - Exception causing close of
>> session 0x58729e0540980002 due to java.nio.channels.CancelledKeyException
>>
>> 2020-06-10 17:10:01,317 - INFO [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
>> connection from /127.0.0.1:45004
>>
>> 2020-06-10 17:10:01,318 - INFO [NIOServerCxn.Factory:
>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command from /
>> 127.0.0.1:45004
>>
>>
>>
>> zookeeper shell client output
>>
>> aparajita.singh@stage-kdc-zk-ivy:~$ sudo
>> /usr/hdp/2.4.0.0-169/zookeeper/bin/zookeeper-client -server
>> stage-kdc-zk-ivy get /test2
>>
>> log4j:WARN Large window sizes are not allowed.
>>
>> log4j:WARN MaxIndex reduced to 13.
>>
>> Connecting to stage-kdc-zk-ivy
>>
>> Debug is true storeKey false useTicketCache true useKeyTab true
>> doNotPrompt true ticketCache is /tmp/krb5cc_0 isInitiator true KeyTab is
>> /etc/krb5.keytab refreshKrb5Config is false principal is
>> zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka tryFirstPass is false
>> useFirstPass is false storePass is false clearPass is false
>>
>> Acquire TGT from Cache
>>
>> Principal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka
>>
>> null credentials from Ticket Cache
>>
>> principal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka
>>
>> Will use keytab
>>
>> Commit Succeeded
>>
>>
>>
>> WATCHER::
>>
>>
>> WatchedEvent state:SyncConnected type:None path:null
>>
>>
>> WATCHER::
>>
>>
>> WatchedEvent state:Disconnected type:None path:null
>>
>> Exception in thread "main"
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /test2
>>
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>
>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>>
>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)
>>
>> at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:717)
>>
>> at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)
>>
>> at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:354)
>>
>> at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)
>>
>> zoo.cfg
>>
>> #setACL=False
>>
>> autopurge.snapRetainCount=30
>>
>> tickTime=2000
>>
>> dataDir=/grid/1/var/lib/zookeeper
>>
>> zookeeper_jmx_port=9009
>>
>> initLimit=100
>>
>> syncLimit=5
>>
>> autopurge.purgeInterval=24
>>
>> clientPort=2181
>>
>> globalOutstandingLimit=5000
>>
>> maxClientCnxns=2000
>>
>> server.99=stage-kdc-zk-harley:2888:3888
>>
>> server.88=stage-kdc-zk-ivy:2888:3888
>>
>> server.77=stage-kdc-zk-2face:2888:3888
>>
>>
>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>>
>> requireClientAuthScheme=sasl
>>
>>
>> quorum.auth.enableSasl=true
>>
>> quorum.auth.learnerRequireSasl=true
>>
>> quorum.auth.serverRequireSasl=true
>>
>> quorum.auth.kerberos.servicePrincipal=host/stage-kdc-zk-ivy@stage.fdp.kafka
>>
>> quorum.cnxn.threads.size=20
>>
>>
>>
>> java.env
>>
>> SERVER_JVMFLAGS="${SERVER_JVMFLAGS}
>> -Djava.security.auth.login.config=/home/aparajita.singh/jaas/jaas.conf
>> -Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>> -Dsun.security.krb5.debug=true"
>>
>> CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS}
>> -Djava.security.auth.login.config=/home/aparajita.singh/jaas/client.conf
>> -Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>> -Dsun.security.krb5.debug=true"
>>
>>
>> /home/aparajita.singh/jaas/jaas.conf
>>
>> // Zookeeper server authentication
>>
>> Server {
>>
>> com.sun.security.auth.module.Krb5LoginModule required
>>
>> useKeyTab=true
>>
>> useTicketCache=false
>>
>> //ticketCache="/tmp/krb5cc_0"
>>
>> renewTicket=true
>>
>> doNotPrompt=true
>>
>> debug=true
>>
>> keyTab="/etc/krb5.keytab"
>>
>> serviceName="host"
>>
>> principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
>>
>> };
>>
>>
>> // Zookeeper quorum server authentication
>>
>> QuorumServer {
>>
>> com.sun.security.auth.module.Krb5LoginModule required
>>
>> useKeyTab=true
>>
>> useTicketCache=false
>>
>> //ticketCache="/tmp/krb5cc_0"
>>
>> renewTicket=true
>>
>> doNotPrompt=true
>>
>> debug=true
>>
>> keyTab="/etc/krb5.keytab"
>>
>> serviceName="host"
>>
>> principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
>>
>> };
>>
>>
>> // Zookeeper learner authentication
>>
>> QuorumLearner {
>>
>> com.sun.security.auth.module.Krb5LoginModule required
>>
>> useKeyTab=true
>>
>> useTicketCache=false
>>
>> //ticketCache="/tmp/krb5cc_0"
>>
>> renewTicket=true
>>
>> doNotPrompt=true
>>
>> debug=true
>>
>> keyTab="/etc/krb5.keytab"
>>
>> serviceName="host"
>>
>> principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
>>
>> };
>>
>>
>>
>> /home/aparajita.singh/jaas/client.conf
>>
>> // Zookeeper client authentication
>>
>> Client {
>>
>> com.sun.security.auth.module.Krb5LoginModule required
>>
>> useKeyTab=true
>>
>> useTicketCache=true
>>
>> ticketCache="/tmp/krb5cc_0"
>>
>> renewTicket=true
>>
>> doNotPrompt=true
>>
>> debug=true
>>
>> keyTab="/etc/krb5.keytab"
>>
>> serviceName="zookeeper"
>>
>> principal="zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka";
>>
>> };
>>
>>
>> Using kinit command I am able to generate the TGT for both principals. As
>> per the zookeeper server log, the TGT can be generated as expected. The
>> keytab file is accessible to all system users for now.
>>
>> aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
>> zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab
>>
>> aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
>> host/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab
>>
>>
>> --
>> Thanks,
>> Aparajita
>>
Re: Zookeeper client fails during SASL authentication
Posted by Arpit Jain <ja...@gmail.com>.
Hi,
I tried it a few months ago and managed to do it. I am not either an expert
on this but managed to do the SASL authentication between ZK and client
I ran the Kerberos server using this image
https://hub.docker.com/r/gcavalcante8808/krb5-server/.
Thanks
On Thu, Jun 11, 2020 at 9:12 AM Szalay-Bekő Máté <sz...@gmail.com>
wrote:
> Hello Aparajita,
>
> After a quick glance on your configs and logs, I haven't found any problem
> with your zookeeper configs. I am not sure if you know this page, using
> these steps worked for me to setup a kerberized zookeeper:
> https://github.com/ekoontz/zookeeper/wiki
> I guess you are also familiar with our wiki:
>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
>
> Based on your logs the problem is here:
> 2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969] - Client failed to SASL
> > authenticate: javax.security.sasl.SaslException: GSS initiate failed
> > [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism
> > level: Invalid argument (400) - Cannot find key of appropriate type to
> > decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)]
> >
>
> This is a kerberos / jaas related issue, I don't think it is zookeeper
> related. a few thing you might wish to check:
> - make sure you have "Java Cryptography Extension (JCE) Unlimited Strength
> Jurisdiction Policy Files" installed (I think you need them for AES256?)
> and your java security configs are OK
> - run "klist -e -k /etc/krb5.keytab" to see if what encryptions you have
> in the keytabs
> - check if you have full export support in JCE by "java KeyLengthDetector"
> - Maybe you can try with different encryption types in kerberos configs /
> during keytab generation.
> - trying to use a different java version (latest JDK patches have some
> known kerberos backward-incompatibilities)
>
> Unfortunately I am not a kerberos expert, so I don't know much about these
> issues, I just used google to find some hints :)
> Maybe someone else in the community with deeper kerberos knowledge can help
> you more.
>
> Kind regards,
> Mate
>
> On Thu, Jun 11, 2020 at 9:47 AM Aparajita Singh <ap...@gmail.com>
> wrote:
>
> > gentle reminder
> > (unquoting the previous email)
> >
> > --
> >
> > Hi,
> >
> > I am trying to migrate an unauthenticated zookeeper cluster to a kerberos
> > authenticated one. For the time being SSL is disabled. I have configured
> > the server and client as described below but when SASL is enabled I am
> > unable to retreive data using zookeeper shell client from the zookeeper
> > server. Could I get some help in understanding why this is failing?
> >
> >
> > *server.log snippet*
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *2020-06-10 17:09:01,263 - INFO
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197
> > <ht...@197>] - Accepted
> socket
> > connection from /127.0.0.1:44994 <http://127.0.0.1:44994>2020-06-10
> > 17:09:01,264 - INFO
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827
> > <ht...@827>] - Processing mntr
> command
> > from /127.0.0.1:44994 <http://127.0.0.1:44994>2020-06-10 17:09:01,265 -
> > INFO [Thread-5:NIOServerCnxn@1007] - Closed socket connection for
> client
> > /127.0.0.1:44994 <http://127.0.0.1:44994> (no session established for
> > client)2020-06-10 17:09:26,647 - INFO [main:Environment@100] - Client
> > environment:zookeeper.version=3.4.6-169--1, built on 02/10/2016 05:49
> > GMT2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client
> > environment:host.name <http://host.name>=stage-kdc-zk-ivy2020-06-10
> > 17:09:26,649 - INFO [main:Environment@100] - Client
> > environment:java.version=1.8.0_1722020-06-10 17:09:26,651 - INFO
> > [main:Environment@100] - Client environment:java.vendor=Oracle
> > Corporation2020-06-10 17:09:26,651 - INFO [main:Environment@100] -
> Client
> > environment:java.home=/usr/lib/jvm/oracle-java8-jdk-amd64/jre2020-06-10
> > 17:09:26,651 - INFO [main:Environment@100] - Client
> >
> >
> environment:java.class.path=/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/classes:/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/hdp/
> > 2.4.0.
> >
> 0-169/zookeeper/bin/../lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../src/java/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/conf::/usr/hdp/2.4.0.0-169/zookeeper/conf:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper.jar:/usr/hdp/
> > 2.4.0.
> >
> 0-169/zookeeper/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/hdp/
> > 2.4.0.
> >
> 0-169/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-codec-1.6.jar:/usr/hdp/
> > 2.4.0.
> >
> 0-169/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/share/zookeeper/*2020-06-10
> > 17:09:26,651 - INFO [main:Environment@100] - Client
> >
> >
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib2020-06-10
> > 17:09:26,651 - INFO [main:Environment@100] - Client
> > environment:java.io.tmpdir=/tmp2020-06-10 17:09:26,651 - INFO
> > [main:Environment@100] - Client
> environment:java.compiler=<NA>2020-06-10
> > 17:09:26,651 - INFO [main:Environment@100] - Client environment:os.name
> > <http://os.name>=Linux2020-06-10 17:09:26,652 - INFO
> > [main:Environment@100] - Client environment:os.arch=amd642020-06-10
> > 17:09:26,652 - INFO [main:Environment@100] - Client
> > environment:os.version=4.9.0-9-amd642020-06-10 17:09:26,652 - INFO
> > [main:Environment@100] - Client environment:user.name
> > <http://user.name>=root2020-06-10 17:09:26,652 - INFO
> > [main:Environment@100] - Client environment:user.home=/root2020-06-10
> > 17:09:26,652 - INFO [main:Environment@100] - Client
> > environment:user.dir=/home/aparajita.singh2020-06-10 17:09:26,653 - INFO
> > [main:ZooKeeper@438] - Initiating client connection,
> > connectString=stage-kdc-zk-ivy sessionTimeout=30000
> > watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@379619aa2020-06-10
> > 17:09:26,752 - INFO [main-SendThread(stage-kdc-zk-ivy:2181):Login@293]
> -
> > successfully logged in.2020-06-10 17:09:26,753 - INFO
> > [Thread-0:Login$1@127] - TGT refresh thread started.2020-06-10
> > 17:09:26,757 - INFO
> > [main-SendThread(stage-kdc-zk-ivy:2181):ZooKeeperSaslClient$1@285] -
> > Client will use GSSAPI as SASL mechanism.2020-06-10 17:09:26,758 - INFO
> > [Thread-0:Login@301] - TGT valid starting at: Wed Jun 10
> 15:17:21
> > IST 20202020-06-10 17:09:26,758 - INFO [Thread-0:Login@302] - TGT
> > expires:
> > Thu Jun 11 15:17:21 IST 20202020-06-10 17:09:26,758 -
> INFO
> > [Thread-0:Login$1@181] - TGT refresh sleeping until: Thu Jun 11
> 11:17:04
> > IST 20202020-06-10 17:09:26,799 - INFO
> > [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1019] -
> > Opening socket connection to server stage-kdc-zk-ivy/10.33.203.225:2181
> > <http://10.33.203.225:2181>. Will attempt to SASL-authenticate using
> Login
> > Context section 'Client'2020-06-10 17:09:26,854 - INFO
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197
> > <ht...@197>] - Accepted
> socket
> > connection from /10.33.203.225:45018 <http://10.33.203.225:45018
> > >2020-06-10
> > 17:09:26,854 - INFO
> > [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@864] -
> > Socket connection established to stage-kdc-zk-ivy/10.33.203.225:2181
> > <http://10.33.203.225:2181>, initiating session2020-06-10 17:09:26,856 -
> > INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868
> > <ht...@868>] - Client attempting
> to
> > establish new session at /10.33.203.225:45018
> > <http://10.33.203.225:45018>2020-06-10 17:09:26,859 - INFO
> > [CommitProcessor:88:ZooKeeperServer@617] - Established session
> > 0x58729e0540980002 with negotiated timeout 30000 for client
> > /10.33.203.225:45018 <http://10.33.203.225:45018>2020-06-10
> 17:09:26,861 -
> > INFO [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1279
> ]
> > -
> > Session establishment complete on server
> > stage-kdc-zk-ivy/10.33.203.225:2181 <http://10.33.203.225:2181>,
> sessionid
> > = 0x58729e0540980002, negotiated timeout = 300002020-06-10 17:09:27,007 -
> > WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969
> > <ht...@969>] - Client failed to
> SASL
> > authenticate: javax.security.sasl.SaslException: GSS initiate failed
> > [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism
> > level: Invalid argument (400) - Cannot find key of appropriate type to
> > decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)]2020-06-10
> 17:09:27,007
> > - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@975
> > <ht...@975>] - Closing client
> > connection due to SASL authentication failure.2020-06-10 17:09:27,007 -
> > INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007
> > <ht...@1007>] - Closed socket
> > connection for client /10.33.203.225:45018 <http://10.33.203.225:45018>
> > which had sessionid 0x58729e05409800022020-06-10 17:09:27,008 - ERROR
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@178
> > <ht...@178>] - Unexpected Exception:
> > java.nio.channels.CancelledKeyExceptionat
> > sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)at
> > sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)at
> >
> >
> org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)at
> >
> >
> org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)at
> >
> >
> org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer.java:936)at
> >
> >
> org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:373)at
> >
> >
> org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:200)at
> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:244)at
> >
> >
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)at
> > java.lang.Thread.run(Thread.java:748)2020-06-10 17:09:27,008 - INFO
> > [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1142] -
> > Unable to read additional data from server sessionid 0x58729e0540980002,
> > likely server has closed socket, closing socket connection and attempting
> > reconnect2020-06-10 17:09:27,008 - WARN
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@346
> > <ht...@346>] - Exception causing
> close
> > of session 0x58729e0540980002 due to
> > java.nio.channels.CancelledKeyException2020-06-10 17:10:01,317 - INFO
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197
> > <ht...@197>] - Accepted
> socket
> > connection from /127.0.0.1:45004 <http://127.0.0.1:45004>2020-06-10
> > 17:10:01,318 - INFO
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827
> > <ht...@827>] - Processing mntr
> command
> > from /127.0.0.1:45004 <http://127.0.0.1:45004>*
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *zookeeper shell client outputaparajita.singh@stage-kdc-zk-ivy:~$ sudo
> > /usr/hdp/2.4.0.0-169/zookeeper/bin/zookeeper-client -server
> > stage-kdc-zk-ivy get /test2log4j:WARN Large window sizes are not
> > allowed.log4j:WARN MaxIndex reduced to 13.Connecting to
> > stage-kdc-zk-ivyDebug is true storeKey false useTicketCache true
> useKeyTab
> > true doNotPrompt true ticketCache is /tmp/krb5cc_0 isInitiator true
> KeyTab
> > is /etc/krb5.keytab refreshKrb5Config is false principal is
> > zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka tryFirstPass is false
> > useFirstPass is false storePass is false clearPass is falseAcquire TGT
> from
> > CachePrincipal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafkanull
> > credentials from Ticket Cacheprincipal is
> > zookeeper/stage-kdc-zk-ivy@stage.fdp.kafkaWill use keytabCommit
> Succeeded
> > WATCHER::WatchedEvent state:SyncConnected type:None
> > path:nullWATCHER::WatchedEvent state:Disconnected type:None
> > path:nullException in thread "main"
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /test2at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:99)at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)at
> > org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)at
> > org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)at
> > org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:717)at
> > org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)at
> > org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:354)at
> >
> >
> org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)zoo.cfg#setACL=Falseautopurge.snapRetainCount=30tickTime=2000dataDir=/grid/1/var/lib/zookeeperzookeeper_jmx_port=9009initLimit=100syncLimit=5autopurge.purgeInterval=24clientPort=2181globalOutstandingLimit=5000maxClientCnxns=2000server.99=stage-kdc-zk-harley:2888:3888server.88=stage-kdc-zk-ivy:2888:3888server.77=stage-kdc-zk-2face:2888:3888authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProviderrequireClientAuthScheme=saslquorum.auth.enableSasl=truequorum.auth.learnerRequireSasl=truequorum.auth.serverRequireSasl=truequorum.auth.kerberos.servicePrincipal=host/stage-kdc-zk-ivy@stage.fdp.kafkaquorum.cnxn.threads.size
> > =20*
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *java.envSERVER_JVMFLAGS="${SERVER_JVMFLAGS}
> > -Djava.security.auth.login.config=/home/aparajita.singh/jaas/jaas.conf
> >
> >
> -Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> > -Dsun.security.krb5.debug=true"CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS}
> > -Djava.security.auth.login.config=/home/aparajita.singh/jaas/client.conf
> >
> >
> -Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> > -Dsun.security.krb5.debug=true"/home/aparajita.singh/jaas/jaas.conf//
> > Zookeeper server authenticationServer {
> > com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true
> > useTicketCache=false //ticketCache="/tmp/krb5cc_0" renewTicket=true
> > doNotPrompt=true debug=true keyTab="/etc/krb5.keytab"
> > serviceName="host" principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
> > }; // Zookeeper quorum server authenticationQuorumServer {
> > com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true
> > useTicketCache=false //ticketCache="/tmp/krb5cc_0" renewTicket=true
> > doNotPrompt=true debug=true keyTab="/etc/krb5.keytab"
> > serviceName="host" principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
> > }; // Zookeeper learner authenticationQuorumLearner {
> > com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true
> > useTicketCache=false //ticketCache="/tmp/krb5cc_0" renewTicket=true
> > doNotPrompt=true debug=true keyTab="/etc/krb5.keytab"
> > serviceName="host" principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
> > }; /home/aparajita.singh/jaas/client.conf// Zookeeper client
> > authenticationClient { com.sun.security.auth.module.Krb5LoginModule
> > required useKeyTab=true useTicketCache=true
> > ticketCache="/tmp/krb5cc_0" renewTicket=true doNotPrompt=true
> > debug=true keyTab="/etc/krb5.keytab" serviceName="zookeeper"
> > principal="zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka"; }; *
> > Using kinit command I am able to generate the TGT for both principals. As
> > per the zookeeper server log, the TGT can be generated as expected. The
> > keytab file is accessible to all system users for now. The below commands
> > don't give any output and the lack of error indicates that the ticket was
> > generated successfully. klist command also shows the latest ticket
> > generated as expected.
> >
> > *aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
> > zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab
> > aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
> > host/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab *
> >
> >
> > Thanks,
> > Aparajita
> >
>
Re: Zookeeper client fails during SASL authentication
Posted by Szalay-Bekő Máté <sz...@gmail.com>.
Hello Aparajita,
After a quick glance on your configs and logs, I haven't found any problem
with your zookeeper configs. I am not sure if you know this page, using
these steps worked for me to setup a kerberized zookeeper:
https://github.com/ekoontz/zookeeper/wiki
I guess you are also familiar with our wiki:
https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
Based on your logs the problem is here:
2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969] - Client failed to SASL
> authenticate: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism
> level: Invalid argument (400) - Cannot find key of appropriate type to
> decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)]
>
This is a kerberos / jaas related issue, I don't think it is zookeeper
related. a few thing you might wish to check:
- make sure you have "Java Cryptography Extension (JCE) Unlimited Strength
Jurisdiction Policy Files" installed (I think you need them for AES256?)
and your java security configs are OK
- run "klist -e -k /etc/krb5.keytab" to see if what encryptions you have
in the keytabs
- check if you have full export support in JCE by "java KeyLengthDetector"
- Maybe you can try with different encryption types in kerberos configs /
during keytab generation.
- trying to use a different java version (latest JDK patches have some
known kerberos backward-incompatibilities)
Unfortunately I am not a kerberos expert, so I don't know much about these
issues, I just used google to find some hints :)
Maybe someone else in the community with deeper kerberos knowledge can help
you more.
Kind regards,
Mate
On Thu, Jun 11, 2020 at 9:47 AM Aparajita Singh <ap...@gmail.com>
wrote:
> gentle reminder
> (unquoting the previous email)
>
> --
>
> Hi,
>
> I am trying to migrate an unauthenticated zookeeper cluster to a kerberos
> authenticated one. For the time being SSL is disabled. I have configured
> the server and client as described below but when SASL is enabled I am
> unable to retreive data using zookeeper shell client from the zookeeper
> server. Could I get some help in understanding why this is failing?
>
>
> *server.log snippet*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *2020-06-10 17:09:01,263 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197
> <ht...@197>] - Accepted socket
> connection from /127.0.0.1:44994 <http://127.0.0.1:44994>2020-06-10
> 17:09:01,264 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827
> <ht...@827>] - Processing mntr command
> from /127.0.0.1:44994 <http://127.0.0.1:44994>2020-06-10 17:09:01,265 -
> INFO [Thread-5:NIOServerCnxn@1007] - Closed socket connection for client
> /127.0.0.1:44994 <http://127.0.0.1:44994> (no session established for
> client)2020-06-10 17:09:26,647 - INFO [main:Environment@100] - Client
> environment:zookeeper.version=3.4.6-169--1, built on 02/10/2016 05:49
> GMT2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client
> environment:host.name <http://host.name>=stage-kdc-zk-ivy2020-06-10
> 17:09:26,649 - INFO [main:Environment@100] - Client
> environment:java.version=1.8.0_1722020-06-10 17:09:26,651 - INFO
> [main:Environment@100] - Client environment:java.vendor=Oracle
> Corporation2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
> environment:java.home=/usr/lib/jvm/oracle-java8-jdk-amd64/jre2020-06-10
> 17:09:26,651 - INFO [main:Environment@100] - Client
>
> environment:java.class.path=/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/classes:/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/hdp/
> 2.4.0.
> 0-169/zookeeper/bin/../lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../src/java/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/conf::/usr/hdp/2.4.0.0-169/zookeeper/conf:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper.jar:/usr/hdp/
> 2.4.0.
> 0-169/zookeeper/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/hdp/
> 2.4.0.
> 0-169/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-codec-1.6.jar:/usr/hdp/
> 2.4.0.
> 0-169/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/share/zookeeper/*2020-06-10
> 17:09:26,651 - INFO [main:Environment@100] - Client
>
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib2020-06-10
> 17:09:26,651 - INFO [main:Environment@100] - Client
> environment:java.io.tmpdir=/tmp2020-06-10 17:09:26,651 - INFO
> [main:Environment@100] - Client environment:java.compiler=<NA>2020-06-10
> 17:09:26,651 - INFO [main:Environment@100] - Client environment:os.name
> <http://os.name>=Linux2020-06-10 17:09:26,652 - INFO
> [main:Environment@100] - Client environment:os.arch=amd642020-06-10
> 17:09:26,652 - INFO [main:Environment@100] - Client
> environment:os.version=4.9.0-9-amd642020-06-10 17:09:26,652 - INFO
> [main:Environment@100] - Client environment:user.name
> <http://user.name>=root2020-06-10 17:09:26,652 - INFO
> [main:Environment@100] - Client environment:user.home=/root2020-06-10
> 17:09:26,652 - INFO [main:Environment@100] - Client
> environment:user.dir=/home/aparajita.singh2020-06-10 17:09:26,653 - INFO
> [main:ZooKeeper@438] - Initiating client connection,
> connectString=stage-kdc-zk-ivy sessionTimeout=30000
> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@379619aa2020-06-10
> 17:09:26,752 - INFO [main-SendThread(stage-kdc-zk-ivy:2181):Login@293] -
> successfully logged in.2020-06-10 17:09:26,753 - INFO
> [Thread-0:Login$1@127] - TGT refresh thread started.2020-06-10
> 17:09:26,757 - INFO
> [main-SendThread(stage-kdc-zk-ivy:2181):ZooKeeperSaslClient$1@285] -
> Client will use GSSAPI as SASL mechanism.2020-06-10 17:09:26,758 - INFO
> [Thread-0:Login@301] - TGT valid starting at: Wed Jun 10 15:17:21
> IST 20202020-06-10 17:09:26,758 - INFO [Thread-0:Login@302] - TGT
> expires:
> Thu Jun 11 15:17:21 IST 20202020-06-10 17:09:26,758 - INFO
> [Thread-0:Login$1@181] - TGT refresh sleeping until: Thu Jun 11 11:17:04
> IST 20202020-06-10 17:09:26,799 - INFO
> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1019] -
> Opening socket connection to server stage-kdc-zk-ivy/10.33.203.225:2181
> <http://10.33.203.225:2181>. Will attempt to SASL-authenticate using Login
> Context section 'Client'2020-06-10 17:09:26,854 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197
> <ht...@197>] - Accepted socket
> connection from /10.33.203.225:45018 <http://10.33.203.225:45018
> >2020-06-10
> 17:09:26,854 - INFO
> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@864] -
> Socket connection established to stage-kdc-zk-ivy/10.33.203.225:2181
> <http://10.33.203.225:2181>, initiating session2020-06-10 17:09:26,856 -
> INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868
> <ht...@868>] - Client attempting to
> establish new session at /10.33.203.225:45018
> <http://10.33.203.225:45018>2020-06-10 17:09:26,859 - INFO
> [CommitProcessor:88:ZooKeeperServer@617] - Established session
> 0x58729e0540980002 with negotiated timeout 30000 for client
> /10.33.203.225:45018 <http://10.33.203.225:45018>2020-06-10 17:09:26,861 -
> INFO [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1279]
> -
> Session establishment complete on server
> stage-kdc-zk-ivy/10.33.203.225:2181 <http://10.33.203.225:2181>, sessionid
> = 0x58729e0540980002, negotiated timeout = 300002020-06-10 17:09:27,007 -
> WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969
> <ht...@969>] - Client failed to SASL
> authenticate: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism
> level: Invalid argument (400) - Cannot find key of appropriate type to
> decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)]2020-06-10 17:09:27,007
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@975
> <ht...@975>] - Closing client
> connection due to SASL authentication failure.2020-06-10 17:09:27,007 -
> INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007
> <ht...@1007>] - Closed socket
> connection for client /10.33.203.225:45018 <http://10.33.203.225:45018>
> which had sessionid 0x58729e05409800022020-06-10 17:09:27,008 - ERROR
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@178
> <ht...@178>] - Unexpected Exception:
> java.nio.channels.CancelledKeyExceptionat
> sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)at
> sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)at
>
> org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)at
>
> org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)at
>
> org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer.java:936)at
>
> org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:373)at
>
> org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:200)at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:244)at
>
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)at
> java.lang.Thread.run(Thread.java:748)2020-06-10 17:09:27,008 - INFO
> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1142] -
> Unable to read additional data from server sessionid 0x58729e0540980002,
> likely server has closed socket, closing socket connection and attempting
> reconnect2020-06-10 17:09:27,008 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@346
> <ht...@346>] - Exception causing close
> of session 0x58729e0540980002 due to
> java.nio.channels.CancelledKeyException2020-06-10 17:10:01,317 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197
> <ht...@197>] - Accepted socket
> connection from /127.0.0.1:45004 <http://127.0.0.1:45004>2020-06-10
> 17:10:01,318 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827
> <ht...@827>] - Processing mntr command
> from /127.0.0.1:45004 <http://127.0.0.1:45004>*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *zookeeper shell client outputaparajita.singh@stage-kdc-zk-ivy:~$ sudo
> /usr/hdp/2.4.0.0-169/zookeeper/bin/zookeeper-client -server
> stage-kdc-zk-ivy get /test2log4j:WARN Large window sizes are not
> allowed.log4j:WARN MaxIndex reduced to 13.Connecting to
> stage-kdc-zk-ivyDebug is true storeKey false useTicketCache true useKeyTab
> true doNotPrompt true ticketCache is /tmp/krb5cc_0 isInitiator true KeyTab
> is /etc/krb5.keytab refreshKrb5Config is false principal is
> zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka tryFirstPass is false
> useFirstPass is false storePass is false clearPass is falseAcquire TGT from
> CachePrincipal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafkanull
> credentials from Ticket Cacheprincipal is
> zookeeper/stage-kdc-zk-ivy@stage.fdp.kafkaWill use keytabCommit Succeeded
> WATCHER::WatchedEvent state:SyncConnected type:None
> path:nullWATCHER::WatchedEvent state:Disconnected type:None
> path:nullException in thread "main"
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /test2at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)at
> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)at
> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)at
> org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:717)at
> org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)at
> org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:354)at
>
> org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)zoo.cfg#setACL=Falseautopurge.snapRetainCount=30tickTime=2000dataDir=/grid/1/var/lib/zookeeperzookeeper_jmx_port=9009initLimit=100syncLimit=5autopurge.purgeInterval=24clientPort=2181globalOutstandingLimit=5000maxClientCnxns=2000server.99=stage-kdc-zk-harley:2888:3888server.88=stage-kdc-zk-ivy:2888:3888server.77=stage-kdc-zk-2face:2888:3888authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProviderrequireClientAuthScheme=saslquorum.auth.enableSasl=truequorum.auth.learnerRequireSasl=truequorum.auth.serverRequireSasl=truequorum.auth.kerberos.servicePrincipal=host/stage-kdc-zk-ivy@stage.fdp.kafkaquorum.cnxn.threads.size
> =20*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *java.envSERVER_JVMFLAGS="${SERVER_JVMFLAGS}
> -Djava.security.auth.login.config=/home/aparajita.singh/jaas/jaas.conf
>
> -Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> -Dsun.security.krb5.debug=true"CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS}
> -Djava.security.auth.login.config=/home/aparajita.singh/jaas/client.conf
>
> -Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> -Dsun.security.krb5.debug=true"/home/aparajita.singh/jaas/jaas.conf//
> Zookeeper server authenticationServer {
> com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true
> useTicketCache=false //ticketCache="/tmp/krb5cc_0" renewTicket=true
> doNotPrompt=true debug=true keyTab="/etc/krb5.keytab"
> serviceName="host" principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
> }; // Zookeeper quorum server authenticationQuorumServer {
> com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true
> useTicketCache=false //ticketCache="/tmp/krb5cc_0" renewTicket=true
> doNotPrompt=true debug=true keyTab="/etc/krb5.keytab"
> serviceName="host" principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
> }; // Zookeeper learner authenticationQuorumLearner {
> com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true
> useTicketCache=false //ticketCache="/tmp/krb5cc_0" renewTicket=true
> doNotPrompt=true debug=true keyTab="/etc/krb5.keytab"
> serviceName="host" principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
> }; /home/aparajita.singh/jaas/client.conf// Zookeeper client
> authenticationClient { com.sun.security.auth.module.Krb5LoginModule
> required useKeyTab=true useTicketCache=true
> ticketCache="/tmp/krb5cc_0" renewTicket=true doNotPrompt=true
> debug=true keyTab="/etc/krb5.keytab" serviceName="zookeeper"
> principal="zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka"; }; *
> Using kinit command I am able to generate the TGT for both principals. As
> per the zookeeper server log, the TGT can be generated as expected. The
> keytab file is accessible to all system users for now. The below commands
> don't give any output and the lack of error indicates that the ticket was
> generated successfully. klist command also shows the latest ticket
> generated as expected.
>
> *aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
> zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab
> aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
> host/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab *
>
>
> Thanks,
> Aparajita
>
Re: Zookeeper client fails during SASL authentication
Posted by Aparajita Singh <ap...@gmail.com>.
gentle reminder
(unquoting the previous email)
--
Hi,
I am trying to migrate an unauthenticated zookeeper cluster to a kerberos
authenticated one. For the time being SSL is disabled. I have configured
the server and client as described below but when SASL is enabled I am
unable to retreive data using zookeeper shell client from the zookeeper
server. Could I get some help in understanding why this is failing?
*server.log snippet*
*2020-06-10 17:09:01,263 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197
<ht...@197>] - Accepted socket
connection from /127.0.0.1:44994 <http://127.0.0.1:44994>2020-06-10
17:09:01,264 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827
<ht...@827>] - Processing mntr command
from /127.0.0.1:44994 <http://127.0.0.1:44994>2020-06-10 17:09:01,265 -
INFO [Thread-5:NIOServerCnxn@1007] - Closed socket connection for client
/127.0.0.1:44994 <http://127.0.0.1:44994> (no session established for
client)2020-06-10 17:09:26,647 - INFO [main:Environment@100] - Client
environment:zookeeper.version=3.4.6-169--1, built on 02/10/2016 05:49
GMT2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client
environment:host.name <http://host.name>=stage-kdc-zk-ivy2020-06-10
17:09:26,649 - INFO [main:Environment@100] - Client
environment:java.version=1.8.0_1722020-06-10 17:09:26,651 - INFO
[main:Environment@100] - Client environment:java.vendor=Oracle
Corporation2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.home=/usr/lib/jvm/oracle-java8-jdk-amd64/jre2020-06-10
17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.class.path=/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/classes:/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../src/java/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/conf::/usr/hdp/2.4.0.0-169/zookeeper/conf:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper.jar:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/share/zookeeper/*2020-06-10
17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib2020-06-10
17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.io.tmpdir=/tmp2020-06-10 17:09:26,651 - INFO
[main:Environment@100] - Client environment:java.compiler=<NA>2020-06-10
17:09:26,651 - INFO [main:Environment@100] - Client environment:os.name
<http://os.name>=Linux2020-06-10 17:09:26,652 - INFO
[main:Environment@100] - Client environment:os.arch=amd642020-06-10
17:09:26,652 - INFO [main:Environment@100] - Client
environment:os.version=4.9.0-9-amd642020-06-10 17:09:26,652 - INFO
[main:Environment@100] - Client environment:user.name
<http://user.name>=root2020-06-10 17:09:26,652 - INFO
[main:Environment@100] - Client environment:user.home=/root2020-06-10
17:09:26,652 - INFO [main:Environment@100] - Client
environment:user.dir=/home/aparajita.singh2020-06-10 17:09:26,653 - INFO
[main:ZooKeeper@438] - Initiating client connection,
connectString=stage-kdc-zk-ivy sessionTimeout=30000
watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@379619aa2020-06-10
17:09:26,752 - INFO [main-SendThread(stage-kdc-zk-ivy:2181):Login@293] -
successfully logged in.2020-06-10 17:09:26,753 - INFO
[Thread-0:Login$1@127] - TGT refresh thread started.2020-06-10
17:09:26,757 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ZooKeeperSaslClient$1@285] -
Client will use GSSAPI as SASL mechanism.2020-06-10 17:09:26,758 - INFO
[Thread-0:Login@301] - TGT valid starting at: Wed Jun 10 15:17:21
IST 20202020-06-10 17:09:26,758 - INFO [Thread-0:Login@302] - TGT expires:
Thu Jun 11 15:17:21 IST 20202020-06-10 17:09:26,758 - INFO
[Thread-0:Login$1@181] - TGT refresh sleeping until: Thu Jun 11 11:17:04
IST 20202020-06-10 17:09:26,799 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1019] -
Opening socket connection to server stage-kdc-zk-ivy/10.33.203.225:2181
<http://10.33.203.225:2181>. Will attempt to SASL-authenticate using Login
Context section 'Client'2020-06-10 17:09:26,854 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197
<ht...@197>] - Accepted socket
connection from /10.33.203.225:45018 <http://10.33.203.225:45018>2020-06-10
17:09:26,854 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@864] -
Socket connection established to stage-kdc-zk-ivy/10.33.203.225:2181
<http://10.33.203.225:2181>, initiating session2020-06-10 17:09:26,856 -
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868
<ht...@868>] - Client attempting to
establish new session at /10.33.203.225:45018
<http://10.33.203.225:45018>2020-06-10 17:09:26,859 - INFO
[CommitProcessor:88:ZooKeeperServer@617] - Established session
0x58729e0540980002 with negotiated timeout 30000 for client
/10.33.203.225:45018 <http://10.33.203.225:45018>2020-06-10 17:09:26,861 -
INFO [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1279] -
Session establishment complete on server
stage-kdc-zk-ivy/10.33.203.225:2181 <http://10.33.203.225:2181>, sessionid
= 0x58729e0540980002, negotiated timeout = 300002020-06-10 17:09:27,007 -
WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969
<ht...@969>] - Client failed to SASL
authenticate: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: Failure unspecified at GSS-API level (Mechanism
level: Invalid argument (400) - Cannot find key of appropriate type to
decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)]2020-06-10 17:09:27,007
- WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@975
<ht...@975>] - Closing client
connection due to SASL authentication failure.2020-06-10 17:09:27,007 -
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007
<ht...@1007>] - Closed socket
connection for client /10.33.203.225:45018 <http://10.33.203.225:45018>
which had sessionid 0x58729e05409800022020-06-10 17:09:27,008 - ERROR
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@178
<ht...@178>] - Unexpected Exception:
java.nio.channels.CancelledKeyExceptionat
sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)at
sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)at
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)at
org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)at
org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer.java:936)at
org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:373)at
org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:200)at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:244)at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)at
java.lang.Thread.run(Thread.java:748)2020-06-10 17:09:27,008 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1142] -
Unable to read additional data from server sessionid 0x58729e0540980002,
likely server has closed socket, closing socket connection and attempting
reconnect2020-06-10 17:09:27,008 - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@346
<ht...@346>] - Exception causing close
of session 0x58729e0540980002 due to
java.nio.channels.CancelledKeyException2020-06-10 17:10:01,317 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197
<ht...@197>] - Accepted socket
connection from /127.0.0.1:45004 <http://127.0.0.1:45004>2020-06-10
17:10:01,318 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827
<ht...@827>] - Processing mntr command
from /127.0.0.1:45004 <http://127.0.0.1:45004>*
*zookeeper shell client outputaparajita.singh@stage-kdc-zk-ivy:~$ sudo
/usr/hdp/2.4.0.0-169/zookeeper/bin/zookeeper-client -server
stage-kdc-zk-ivy get /test2log4j:WARN Large window sizes are not
allowed.log4j:WARN MaxIndex reduced to 13.Connecting to
stage-kdc-zk-ivyDebug is true storeKey false useTicketCache true useKeyTab
true doNotPrompt true ticketCache is /tmp/krb5cc_0 isInitiator true KeyTab
is /etc/krb5.keytab refreshKrb5Config is false principal is
zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka tryFirstPass is false
useFirstPass is false storePass is false clearPass is falseAcquire TGT from
CachePrincipal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafkanull
credentials from Ticket Cacheprincipal is
zookeeper/stage-kdc-zk-ivy@stage.fdp.kafkaWill use keytabCommit Succeeded
WATCHER::WatchedEvent state:SyncConnected type:None
path:nullWATCHER::WatchedEvent state:Disconnected type:None
path:nullException in thread "main"
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /test2at
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)at
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)at
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)at
org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:717)at
org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)at
org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:354)at
org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)zoo.cfg#setACL=Falseautopurge.snapRetainCount=30tickTime=2000dataDir=/grid/1/var/lib/zookeeperzookeeper_jmx_port=9009initLimit=100syncLimit=5autopurge.purgeInterval=24clientPort=2181globalOutstandingLimit=5000maxClientCnxns=2000server.99=stage-kdc-zk-harley:2888:3888server.88=stage-kdc-zk-ivy:2888:3888server.77=stage-kdc-zk-2face:2888:3888authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProviderrequireClientAuthScheme=saslquorum.auth.enableSasl=truequorum.auth.learnerRequireSasl=truequorum.auth.serverRequireSasl=truequorum.auth.kerberos.servicePrincipal=host/stage-kdc-zk-ivy@stage.fdp.kafkaquorum.cnxn.threads.size=20*
*java.envSERVER_JVMFLAGS="${SERVER_JVMFLAGS}
-Djava.security.auth.login.config=/home/aparajita.singh/jaas/jaas.conf
-Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
-Dsun.security.krb5.debug=true"CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS}
-Djava.security.auth.login.config=/home/aparajita.singh/jaas/client.conf
-Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
-Dsun.security.krb5.debug=true"/home/aparajita.singh/jaas/jaas.conf//
Zookeeper server authenticationServer {
com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true
useTicketCache=false //ticketCache="/tmp/krb5cc_0" renewTicket=true
doNotPrompt=true debug=true keyTab="/etc/krb5.keytab"
serviceName="host" principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
}; // Zookeeper quorum server authenticationQuorumServer {
com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true
useTicketCache=false //ticketCache="/tmp/krb5cc_0" renewTicket=true
doNotPrompt=true debug=true keyTab="/etc/krb5.keytab"
serviceName="host" principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
}; // Zookeeper learner authenticationQuorumLearner {
com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true
useTicketCache=false //ticketCache="/tmp/krb5cc_0" renewTicket=true
doNotPrompt=true debug=true keyTab="/etc/krb5.keytab"
serviceName="host" principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
}; /home/aparajita.singh/jaas/client.conf// Zookeeper client
authenticationClient { com.sun.security.auth.module.Krb5LoginModule
required useKeyTab=true useTicketCache=true
ticketCache="/tmp/krb5cc_0" renewTicket=true doNotPrompt=true
debug=true keyTab="/etc/krb5.keytab" serviceName="zookeeper"
principal="zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka"; }; *
Using kinit command I am able to generate the TGT for both principals. As
per the zookeeper server log, the TGT can be generated as expected. The
keytab file is accessible to all system users for now. The below commands
don't give any output and the lack of error indicates that the ticket was
generated successfully. klist command also shows the latest ticket
generated as expected.
*aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab
aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
host/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab *
Thanks,
Aparajita
Re: Zookeeper client fails during SASL authentication
Posted by Aparajita Singh <ap...@gmail.com>.
>
> Hi,
>
> I am trying to migrate an unauthenticated zookeeper cluster to a kerberos
> authenticated one. For the time being SSL is disabled. I have configured
> the server and client as described below but when SASL is enabled I am
> unable to retreive data using zookeeper shell client from the zookeeper
> server. Could I get some help in understanding why this is failing?
>
> server.log snippet
>
> 2020-06-10 17:09:01,263 - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> connection from /127.0.0.1:44994
>
> 2020-06-10 17:09:01,264 - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command from /
> 127.0.0.1:44994
>
> 2020-06-10 17:09:01,265 - INFO [Thread-5:NIOServerCnxn@1007] - Closed
> socket connection for client /127.0.0.1:44994 (no session established for
> client)
>
> 2020-06-10 17:09:26,647 - INFO [main:Environment@100] - Client
> environment:zookeeper.version=3.4.6-169--1, built on 02/10/2016 05:49 GMT
>
> 2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client
> environment:host.name=stage-kdc-zk-ivy
>
> 2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client
> environment:java.version=1.8.0_172
>
> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
> environment:java.vendor=Oracle Corporation
>
> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
> environment:java.home=/usr/lib/jvm/oracle-java8-jdk-amd64/jre
>
> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
> environment:java.class.path=/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/classes:/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../src/java/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/conf::/usr/hdp/2.4.0.0-169/zookeeper/conf:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper.jar:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/share/zookeeper/*
>
> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>
> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
> environment:java.io.tmpdir=/tmp
>
> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
> environment:java.compiler=<NA>
>
> 2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
> environment:os.name=Linux
>
> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
> environment:os.arch=amd64
>
> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
> environment:os.version=4.9.0-9-amd64
>
> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
> environment:user.name=root
>
> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
> environment:user.home=/root
>
> 2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
> environment:user.dir=/home/aparajita.singh
>
> 2020-06-10 17:09:26,653 - INFO [main:ZooKeeper@438] - Initiating client
> connection, connectString=stage-kdc-zk-ivy sessionTimeout=30000
> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@379619aa
>
> 2020-06-10 17:09:26,752 - INFO
> [main-SendThread(stage-kdc-zk-ivy:2181):Login@293] - successfully logged
> in.
>
> 2020-06-10 17:09:26,753 - INFO [Thread-0:Login$1@127] - TGT refresh
> thread started.
>
> 2020-06-10 17:09:26,757 - INFO
> [main-SendThread(stage-kdc-zk-ivy:2181):ZooKeeperSaslClient$1@285] -
> Client will use GSSAPI as SASL mechanism.
>
> 2020-06-10 17:09:26,758 - INFO [Thread-0:Login@301] - TGT valid starting
> at: Wed Jun 10 15:17:21 IST 2020
>
> 2020-06-10 17:09:26,758 - INFO [Thread-0:Login@302] - TGT expires:
> Thu Jun 11 15:17:21 IST 2020
>
> 2020-06-10 17:09:26,758 - INFO [Thread-0:Login$1@181] - TGT refresh
> sleeping until: Thu Jun 11 11:17:04 IST 2020
>
> 2020-06-10 17:09:26,799 - INFO
> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1019] -
> Opening socket connection to server stage-kdc-zk-ivy/10.33.203.225:2181.
> Will attempt to SASL-authenticate using Login Context section 'Client'
>
> 2020-06-10 17:09:26,854 - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> connection from /10.33.203.225:45018
>
> 2020-06-10 17:09:26,854 - INFO
> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@864] -
> Socket connection established to stage-kdc-zk-ivy/10.33.203.225:2181,
> initiating session
>
> 2020-06-10 17:09:26,856 - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to
> establish new session at /10.33.203.225:45018
>
> 2020-06-10 17:09:26,859 - INFO [CommitProcessor:88:ZooKeeperServer@617]
> - Established session 0x58729e0540980002 with negotiated timeout 30000 for
> client /10.33.203.225:45018
>
> 2020-06-10 17:09:26,861 - INFO
> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1279] -
> Session establishment complete on server stage-kdc-zk-ivy/
> 10.33.203.225:2181, sessionid = 0x58729e0540980002, negotiated timeout =
> 30000
>
> 2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969] - Client failed to SASL
> authenticate: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism
> level: Invalid argument (400) - Cannot find key of appropriate type to
> decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)]
>
> 2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@975] - Closing client connection due
> to SASL authentication failure.
>
> 2020-06-10 17:09:27,007 - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
> client /10.33.203.225:45018 which had sessionid 0x58729e0540980002
>
> 2020-06-10 17:09:27,008 - ERROR [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@178] - Unexpected Exception:
>
> java.nio.channels.CancelledKeyException
>
> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
>
> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
>
> at
> org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
>
> at
> org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
>
> at
> org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer.java:936)
>
> at
> org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:373)
>
> at
> org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:200)
>
> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:244)
>
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>
> at java.lang.Thread.run(Thread.java:748)
>
> 2020-06-10 17:09:27,008 - INFO
> [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1142] -
> Unable to read additional data from server sessionid 0x58729e0540980002,
> likely server has closed socket, closing socket connection and attempting
> reconnect
>
> 2020-06-10 17:09:27,008 - WARN [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@346] - Exception causing close of
> session 0x58729e0540980002 due to java.nio.channels.CancelledKeyException
>
> 2020-06-10 17:10:01,317 - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> connection from /127.0.0.1:45004
>
> 2020-06-10 17:10:01,318 - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command from /
> 127.0.0.1:45004
>
>
>
> zookeeper shell client output
>
> aparajita.singh@stage-kdc-zk-ivy:~$ sudo
> /usr/hdp/2.4.0.0-169/zookeeper/bin/zookeeper-client -server
> stage-kdc-zk-ivy get /test2
>
> log4j:WARN Large window sizes are not allowed.
>
> log4j:WARN MaxIndex reduced to 13.
>
> Connecting to stage-kdc-zk-ivy
>
> Debug is true storeKey false useTicketCache true useKeyTab true
> doNotPrompt true ticketCache is /tmp/krb5cc_0 isInitiator true KeyTab is
> /etc/krb5.keytab refreshKrb5Config is false principal is
> zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka tryFirstPass is false
> useFirstPass is false storePass is false clearPass is false
>
> Acquire TGT from Cache
>
> Principal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka
>
> null credentials from Ticket Cache
>
> principal is zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka
>
> Will use keytab
>
> Commit Succeeded
>
>
>
> WATCHER::
>
>
> WatchedEvent state:SyncConnected type:None path:null
>
>
> WATCHER::
>
>
> WatchedEvent state:Disconnected type:None path:null
>
> Exception in thread "main"
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /test2
>
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)
>
> at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:717)
>
> at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)
>
> at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:354)
>
> at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)
>
> zoo.cfg
>
> #setACL=False
>
> autopurge.snapRetainCount=30
>
> tickTime=2000
>
> dataDir=/grid/1/var/lib/zookeeper
>
> zookeeper_jmx_port=9009
>
> initLimit=100
>
> syncLimit=5
>
> autopurge.purgeInterval=24
>
> clientPort=2181
>
> globalOutstandingLimit=5000
>
> maxClientCnxns=2000
>
> server.99=stage-kdc-zk-harley:2888:3888
>
> server.88=stage-kdc-zk-ivy:2888:3888
>
> server.77=stage-kdc-zk-2face:2888:3888
>
>
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>
> requireClientAuthScheme=sasl
>
>
> quorum.auth.enableSasl=true
>
> quorum.auth.learnerRequireSasl=true
>
> quorum.auth.serverRequireSasl=true
>
> quorum.auth.kerberos.servicePrincipal=host/stage-kdc-zk-ivy@stage.fdp.kafka
>
> quorum.cnxn.threads.size=20
>
>
>
> java.env
>
> SERVER_JVMFLAGS="${SERVER_JVMFLAGS}
> -Djava.security.auth.login.config=/home/aparajita.singh/jaas/jaas.conf
> -Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> -Dsun.security.krb5.debug=true"
>
> CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS}
> -Djava.security.auth.login.config=/home/aparajita.singh/jaas/client.conf
> -Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> -Dsun.security.krb5.debug=true"
>
>
> /home/aparajita.singh/jaas/jaas.conf
>
> // Zookeeper server authentication
>
> Server {
>
> com.sun.security.auth.module.Krb5LoginModule required
>
> useKeyTab=true
>
> useTicketCache=false
>
> //ticketCache="/tmp/krb5cc_0"
>
> renewTicket=true
>
> doNotPrompt=true
>
> debug=true
>
> keyTab="/etc/krb5.keytab"
>
> serviceName="host"
>
> principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
>
> };
>
>
> // Zookeeper quorum server authentication
>
> QuorumServer {
>
> com.sun.security.auth.module.Krb5LoginModule required
>
> useKeyTab=true
>
> useTicketCache=false
>
> //ticketCache="/tmp/krb5cc_0"
>
> renewTicket=true
>
> doNotPrompt=true
>
> debug=true
>
> keyTab="/etc/krb5.keytab"
>
> serviceName="host"
>
> principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
>
> };
>
>
> // Zookeeper learner authentication
>
> QuorumLearner {
>
> com.sun.security.auth.module.Krb5LoginModule required
>
> useKeyTab=true
>
> useTicketCache=false
>
> //ticketCache="/tmp/krb5cc_0"
>
> renewTicket=true
>
> doNotPrompt=true
>
> debug=true
>
> keyTab="/etc/krb5.keytab"
>
> serviceName="host"
>
> principal="host/stage-kdc-zk-ivy@stage.fdp.kafka";
>
> };
>
>
>
> /home/aparajita.singh/jaas/client.conf
>
> // Zookeeper client authentication
>
> Client {
>
> com.sun.security.auth.module.Krb5LoginModule required
>
> useKeyTab=true
>
> useTicketCache=true
>
> ticketCache="/tmp/krb5cc_0"
>
> renewTicket=true
>
> doNotPrompt=true
>
> debug=true
>
> keyTab="/etc/krb5.keytab"
>
> serviceName="zookeeper"
>
> principal="zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka";
>
> };
>
>
> Using kinit command I am able to generate the TGT for both principals. As
> per the zookeeper server log, the TGT can be generated as expected. The
> keytab file is accessible to all system users for now.
>
> aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
> zookeeper/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab
>
> aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
> host/stage-kdc-zk-ivy@stage.fdp.kafka -k -t /etc/krb5.keytab
>
>
> --
> Thanks,
> Aparajita
>