You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Oleg Zhurakousky <oz...@hortonworks.com> on 2016/02/23 20:01:59 UTC
Kerberized Kafka setup issues
Hey guys, first post here so bare with me
Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here http://kafka.apache.org/documentation.html#security_sasl and i seem to be very close, but not quite there yet.
ZOOKEEPER
Starting Zookeeper seems to be OK (below is the relevant part of the log)
. . .
[2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab refreshKrb5Config is false principal is zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com> tryFirstPass is false useFirstPass is false storePass is false clearPass is false
principal is zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
Will use keytab
Commit Succeeded
[2016-02-23 13:22:40,541] INFO successfully logged in. (org.apache.zookeeper.Login)
[2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-02-23 13:22:40,544] INFO TGT refresh thread started. (org.apache.zookeeper.Login)
[2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23 13:22:40 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23 23:22:40 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23 21:47:35 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:23:09,012] INFO Accepted socket connection from /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-02-23 13:23:09,025] INFO Client attempting to establish new session at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-02-23 13:23:09,026] INFO Creating new log file: log.57 (org.apache.zookeeper.server.persistence.FileTxnLog)
. . .
KAFKA
Starting Kafka server is not going well yet although I see that interaction with Kerberos is successful (see relevant log below. the error is at the bottom)
. . .
[2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
[2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2016-02-23 13:26:11,519] INFO JAAS File name: /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf (org.I0Itec.zkclient.ZkClient)
[2016-02-23 13:26:11,520] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2016-02-23 13:26:11,527] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20 (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72 (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,527] INFO Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,527] INFO Client environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka. . . . . .
[2016-02-23 13:26:11,531] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1 (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,532] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@647fd8ce (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated (org.I0Itec.zkclient.ZkClient)
Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab refreshKrb5Config is false principal is kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com> tryFirstPass is false useFirstPass is false storePass is false clearPass is false
principal is kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
Will use keytab
Commit Succeeded
[2016-02-23 13:26:11,734] INFO successfully logged in. (org.apache.zookeeper.Login)
[2016-02-23 13:26:11,735] INFO TGT refresh thread started. (org.apache.zookeeper.Login)
[2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2016-02-23 13:26:11,743] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2016-02-23 13:26:11,748] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23 13:26:11 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23 23:26:11 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23 21:40:22 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:26:11,761] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2016-02-23 13:26:11,773] ERROR An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
[2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed) (org.I0Itec.zkclient.ZkClient)
[2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
. . .
Any pointers?
Cheers
Oleg
RE: Kerberized Kafka setup issues
Posted by Martin Gainty <mg...@hotmail.com>.
Glad to hear you worked it out Oleg!
Martin
_______________________________
> Subject: Re: Kerberized Kafka setup issues
> From: ozhurakousky@hortonworks.com
> To: users@kafka.apache.org
> Date: Wed, 24 Feb 2016 16:27:05 +0000
>
> Guys, thank you so much for helping, but the error was all on my end. This morning I’ve looked at my krb5.conf and noticed this:
> [domain_realm]
> .ubuntu.oleg.com = OLEG.COM
> ubuntu.oleg.com = OLEG.COM
> instead of
> [domain_realm]
> .oleg.com = OLEG.COM
> oleg.com = OLEG.COM
>
> Once I changed it all went fine!
>
> Cheers
> Oleg
> > On Feb 23, 2016, at 6:09 PM, Oleg Zhurakousky <oz...@hortonworks.com> wrote:
> >
> > Well, I am running on the same machine, so I say yes
> >
> > Sent from my iPhone
> >
> >> On Feb 23, 2016, at 18:05, Martin Gainty <mg...@hotmail.com> wrote:
> >>
> >> one more thing to check:
> >>
> >> specifically are the /etc/krb5.conf credentials the same you use to authenticate to ubuntu.oleg.com
> >>
> >> ?
> >> Martin
> >> __________________
> >>
> >>
> >>
> >>> Subject: Re: Kerberized Kafka setup issues
> >>> From: ozhurakousky@hortonworks.com
> >>> To: users@kafka.apache.org
> >>> Date: Tue, 23 Feb 2016 21:58:48 +0000
> >>>
> >>> Harsh
> >>>
> >>> I followed this blog (http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption) and got an environment via vagrant setup, no issues. I’ll poke around what the differences are and if find the issue will post.
> >>> Thanks for your help anyway.
> >>>
> >>> Cheers
> >>> Oleg
> >>> On Feb 23, 2016, at 4:06 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
> >>>
> >>> Yeah, I noticed the localhost as well, but I’ve changed it since to FQDN and it is still the same including 'sname is zookeeper/localhost@OLEG.COM<ma...@oleg.com>’
> >>>
> >>> Oleg
> >>>
> >>> On Feb 23, 2016, at 4:00 PM, Harsha <ka...@harsha.io>> wrote:
> >>>
> >>> whats your zookeeper.connect in server.properties looks like. Did you
> >>> use the hostname or localhost
> >>> -Harsha
> >>>
> >>> On Tue, Feb 23, 2016, at 12:01 PM, Oleg Zhurakousky wrote:
> >>> Still digging, but here is more info that may help
> >>>
> >>> 2016-02-23 14:59:24,240] INFO zookeeper state changed (SyncConnected)
> >>> (org.I0Itec.zkclient.ZkClient)
> >>> Found ticket for kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com> to go to
> >>> krbtgt/OLEG.COM@OLEG.COM<ma...@oleg.com> expiring on Wed Feb 24 00:59:24 EST 2016
> >>> Entered Krb5Context.initSecContext with state=STATE_NEW
> >>> Found ticket for kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com> to go to
> >>> krbtgt/OLEG.COM@OLEG.COM<ma...@oleg.com> expiring on Wed Feb 24 00:59:24 EST 2016
> >>> Service ticket not found in the subject
> >>> Credentials acquireServiceCreds: same realm
> >>> Using builtin default etypes for default_tgs_enctypes
> >>> default etypes for default_tgs_enctypes: 17 16 23.
> >>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
> >>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
> >>> KrbKdcReq send: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88, timeout=30000, number of retries =3, #bytes=660
> >>> KDCCommunication: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88, timeout=30000,Attempt =1, #bytes=660
> >>> KrbKdcReq send: #bytes read=183
> >>> KdcAccessibility: remove ubuntu.oleg.com<http://ubuntu.oleg.com>
> >>> KDCRep: init() encoding tag is 126 req type is 13
> >>> KRBError:
> >>> cTime is Sat Aug 01 11:32:55 EDT 1998 901985575000
> >>> sTime is Tue Feb 23 14:59:24 EST 2016 1456257564000
> >>> suSec is 248635
> >>> error code is 7
> >>> error Message is Server not found in Kerberos database
> >>> cname is kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> >>> sname is zookeeper/localhost@OLEG.COM<ma...@oleg.com>
> >>> msgType is 30
> >>>
> >>> On Feb 23, 2016, at 2:46 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
> >>>
> >>> No joy. the same error
> >>>
> >>> KafkaServer {
> >>> com.sun.security.auth.module.Krb5LoginModule required
> >>> debug=true
> >>> useKeyTab=true
> >>> storeKey=true
> >>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
> >>> };
> >>> Client {
> >>> com.sun.security.auth.module.Krb5LoginModule required
> >>> debug=true
> >>> useKeyTab=true
> >>> serviceName=zookeeper
> >>> storeKey=true
> >>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
> >>> };
> >>> On Feb 23, 2016, at 2:41 PM, Harsha <ma...@harsha.io>> wrote:
> >>>
> >>> My bad it should be under Client section
> >>>
> >>> Client {
> >>> com.sun.security.auth.module.Krb5LoginModule required
> >>> debug=true
> >>> useKeyTab=true
> >>> storeKey=true
> >>> serviceName=zookeeper
> >>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
> >>> };
> >>>
> >>> -Harsha
> >>>
> >>> On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote:
> >>> can you try adding "serviceName=zookeeper" to KafkaServer section like
> >>> KafkaServer {
> >>> com.sun.security.auth.module.Krb5LoginModule required
> >>> debug=true
> >>> useKeyTab=true
> >>> storeKey=true
> >>> serviceName=zookeeper
> >>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
> >>> };
> >>>
> >>> On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
> >>> More info
> >>>
> >>> I am starting both services as myself ‘oleg’. Validated that both key tab
> >>> files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
> >>> and Kafka as ‘kafka’
> >>>
> >>> Oleg
> >>>
> >>> On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
> >>>
> >>> Harsha
> >>>
> >>> Thanks for following up. Here is is:
> >>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf
> >>> KafkaServer {
> >>> com.sun.security.auth.module.Krb5LoginModule required
> >>> debug=true
> >>> useKeyTab=true
> >>> storeKey=true
> >>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
> >>> };
> >>> Client {
> >>> com.sun.security.auth.module.Krb5LoginModule required
> >>> debug=true
> >>> useKeyTab=true
> >>> storeKey=true
> >>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
> >>> };
> >>>
> >>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf
> >>> Server {
> >>> com.sun.security.auth.module.Krb5LoginModule required
> >>> debug=true
> >>> useKeyTab=true
> >>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
> >>> storeKey=true
> >>> useTicketCache=false
> >>> principal="zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
> >>> };
> >>>
> >>> Cheers
> >>> Oleg
> >>>
> >>> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io>> wrote:
> >>>
> >>> Oleg,
> >>> Can you post your jaas configs. Its important that serviceName
> >>> must match the principal name with which zookeeper is running.
> >>> Whats the principal name zookeeper service is running with.
> >>> -Harsha
> >>>
> >>> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
> >>> Hey guys, first post here so bare with me
> >>>
> >>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
> >>> http://kafka.apache.org/documentation.html#security_sasl and i seem to be
> >>> very close, but not quite there yet.
> >>>
> >>> ZOOKEEPER
> >>> Starting Zookeeper seems to be OK (below is the relevant part of the log)
> >>> . . .
> >>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
> >>> (org.apache.zookeeper.server.ZooKeeperServer)
> >>> Debug is true storeKey true useTicketCache false useKeyTab true
> >>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
> >>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
> >>> refreshKrb5Config is false principal is
> >>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> >>> tryFirstPass is false useFirstPass is false storePass is false clearPass
> >>> is false
> >>> principal is
> >>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> >>> Will use keytab
> >>> Commit Succeeded
> >>>
> >>> [2016-02-23 13:22:40,541] INFO successfully logged in.
> >>> (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
> >>> (org.apache.zookeeper.server.NIOServerCnxnFactory)
> >>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
> >>> (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23
> >>> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23
> >>> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
> >>> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
> >>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> >>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new session
> >>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
> >>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
> >>> (org.apache.zookeeper.server.persistence.FileTxnLog)
> >>> . . .
> >>>
> >>>
> >>> KAFKA
> >>> Starting Kafka server is not going well yet although I see that
> >>> interaction with Kerberos is successful (see relevant log below. the
> >>> error is at the bottom)
> >>> . . .
> >>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
> >>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
> >>> (kafka.server.KafkaServer)
> >>> [2016-02-23 13:26:11,519] INFO JAAS File name:
> >>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
> >>> (org.I0Itec.zkclient.ZkClient)
> >>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
> >>> (org.I0Itec.zkclient.ZkEventThread)
> >>> [2016-02-23 13:26:11,527] INFO Client
> >>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> >>> GMT (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
> >>> Corporation (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,527] INFO Client
> >>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,527] INFO Client
> >>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
> >>> . . . . .
> >>> [2016-02-23 13:26:11,531] INFO Client
> >>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client
> >>> environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client
> >>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,532] INFO Initiating client connection,
> >>> connectString=localhost:2181 sessionTimeout=6000
> >>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
> >>> (org.I0Itec.zkclient.ZkClient)
> >>> Debug is true storeKey true useTicketCache false useKeyTab true
> >>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
> >>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
> >>> refreshKrb5Config is false principal is
> >>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> >>> tryFirstPass is false useFirstPass is false storePass is false clearPass
> >>> is false
> >>> principal is
> >>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> >>> Will use keytab
> >>> Commit Succeeded
> >>>
> >>> [2016-02-23 13:26:11,734] INFO successfully logged in.
> >>> (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
> >>> (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
> >>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
> >>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
> >>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
> >>> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
> >>> [2016-02-23 13:26:11,748] INFO Socket connection established to
> >>> localhost/127.0.0.1:2181, initiating session
> >>> (org.apache.zookeeper.ClientCnxn)
> >>> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23
> >>> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23
> >>> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
> >>> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:26:11,761] INFO Session establishment complete on server
> >>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
> >>> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> >>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
> >>> (org.I0Itec.zkclient.ZkClient)
> >>> [2016-02-23 13:26:11,773] ERROR An error:
> >>> (java.security.PrivilegedActionException:
> >>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> >>> GSSException: No valid credentials provided (Mechanism level: Server not
> >>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
> >>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
> >>> Client will go to AUTH_FAILED state.
> >>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
> >>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
> >>> member failed: javax.security.sasl.SaslException: An error:
> >>> (java.security.PrivilegedActionException:
> >>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> >>> GSSException: No valid credentials provided (Mechanism level: Server not
> >>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
> >>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
> >>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
> >>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
> >>> (org.I0Itec.zkclient.ZkClient)
> >>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
> >>> (org.I0Itec.zkclient.ZkEventThread)
> >>> . . .
> >>>
> >>> Any pointers?
> >>>
> >>> Cheers
> >>> Oleg
> >>
> >
>
Re: Kerberized Kafka setup issues
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
Guys, thank you so much for helping, but the error was all on my end. This morning I’ve looked at my krb5.conf and noticed this:
[domain_realm]
.ubuntu.oleg.com = OLEG.COM
ubuntu.oleg.com = OLEG.COM
instead of
[domain_realm]
.oleg.com = OLEG.COM
oleg.com = OLEG.COM
Once I changed it all went fine!
Cheers
Oleg
> On Feb 23, 2016, at 6:09 PM, Oleg Zhurakousky <oz...@hortonworks.com> wrote:
>
> Well, I am running on the same machine, so I say yes
>
> Sent from my iPhone
>
>> On Feb 23, 2016, at 18:05, Martin Gainty <mg...@hotmail.com> wrote:
>>
>> one more thing to check:
>>
>> specifically are the /etc/krb5.conf credentials the same you use to authenticate to ubuntu.oleg.com
>>
>> ?
>> Martin
>> __________________
>>
>>
>>
>>> Subject: Re: Kerberized Kafka setup issues
>>> From: ozhurakousky@hortonworks.com
>>> To: users@kafka.apache.org
>>> Date: Tue, 23 Feb 2016 21:58:48 +0000
>>>
>>> Harsh
>>>
>>> I followed this blog (http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption) and got an environment via vagrant setup, no issues. I’ll poke around what the differences are and if find the issue will post.
>>> Thanks for your help anyway.
>>>
>>> Cheers
>>> Oleg
>>> On Feb 23, 2016, at 4:06 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
>>>
>>> Yeah, I noticed the localhost as well, but I’ve changed it since to FQDN and it is still the same including 'sname is zookeeper/localhost@OLEG.COM<ma...@oleg.com>’
>>>
>>> Oleg
>>>
>>> On Feb 23, 2016, at 4:00 PM, Harsha <ka...@harsha.io>> wrote:
>>>
>>> whats your zookeeper.connect in server.properties looks like. Did you
>>> use the hostname or localhost
>>> -Harsha
>>>
>>> On Tue, Feb 23, 2016, at 12:01 PM, Oleg Zhurakousky wrote:
>>> Still digging, but here is more info that may help
>>>
>>> 2016-02-23 14:59:24,240] INFO zookeeper state changed (SyncConnected)
>>> (org.I0Itec.zkclient.ZkClient)
>>> Found ticket for kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com> to go to
>>> krbtgt/OLEG.COM@OLEG.COM<ma...@oleg.com> expiring on Wed Feb 24 00:59:24 EST 2016
>>> Entered Krb5Context.initSecContext with state=STATE_NEW
>>> Found ticket for kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com> to go to
>>> krbtgt/OLEG.COM@OLEG.COM<ma...@oleg.com> expiring on Wed Feb 24 00:59:24 EST 2016
>>> Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
>>> Using builtin default etypes for default_tgs_enctypes
>>> default etypes for default_tgs_enctypes: 17 16 23.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
>>> KrbKdcReq send: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88, timeout=30000, number of retries =3, #bytes=660
>>> KDCCommunication: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88, timeout=30000,Attempt =1, #bytes=660
>>> KrbKdcReq send: #bytes read=183
>>> KdcAccessibility: remove ubuntu.oleg.com<http://ubuntu.oleg.com>
>>> KDCRep: init() encoding tag is 126 req type is 13
>>> KRBError:
>>> cTime is Sat Aug 01 11:32:55 EDT 1998 901985575000
>>> sTime is Tue Feb 23 14:59:24 EST 2016 1456257564000
>>> suSec is 248635
>>> error code is 7
>>> error Message is Server not found in Kerberos database
>>> cname is kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>> sname is zookeeper/localhost@OLEG.COM<ma...@oleg.com>
>>> msgType is 30
>>>
>>> On Feb 23, 2016, at 2:46 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
>>>
>>> No joy. the same error
>>>
>>> KafkaServer {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> storeKey=true
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
>>> };
>>> Client {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> serviceName=zookeeper
>>> storeKey=true
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
>>> };
>>> On Feb 23, 2016, at 2:41 PM, Harsha <ma...@harsha.io>> wrote:
>>>
>>> My bad it should be under Client section
>>>
>>> Client {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> storeKey=true
>>> serviceName=zookeeper
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
>>> };
>>>
>>> -Harsha
>>>
>>> On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote:
>>> can you try adding "serviceName=zookeeper" to KafkaServer section like
>>> KafkaServer {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> storeKey=true
>>> serviceName=zookeeper
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
>>> };
>>>
>>> On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
>>> More info
>>>
>>> I am starting both services as myself ‘oleg’. Validated that both key tab
>>> files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
>>> and Kafka as ‘kafka’
>>>
>>> Oleg
>>>
>>> On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
>>>
>>> Harsha
>>>
>>> Thanks for following up. Here is is:
>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf
>>> KafkaServer {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> storeKey=true
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
>>> };
>>> Client {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> storeKey=true
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
>>> };
>>>
>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf
>>> Server {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
>>> storeKey=true
>>> useTicketCache=false
>>> principal="zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
>>> };
>>>
>>> Cheers
>>> Oleg
>>>
>>> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io>> wrote:
>>>
>>> Oleg,
>>> Can you post your jaas configs. Its important that serviceName
>>> must match the principal name with which zookeeper is running.
>>> Whats the principal name zookeeper service is running with.
>>> -Harsha
>>>
>>> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
>>> Hey guys, first post here so bare with me
>>>
>>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
>>> http://kafka.apache.org/documentation.html#security_sasl and i seem to be
>>> very close, but not quite there yet.
>>>
>>> ZOOKEEPER
>>> Starting Zookeeper seems to be OK (below is the relevant part of the log)
>>> . . .
>>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
>>> (org.apache.zookeeper.server.ZooKeeperServer)
>>> Debug is true storeKey true useTicketCache false useKeyTab true
>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
>>> refreshKrb5Config is false principal is
>>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>>> is false
>>> principal is
>>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>> Will use keytab
>>> Commit Succeeded
>>>
>>> [2016-02-23 13:22:40,541] INFO successfully logged in.
>>> (org.apache.zookeeper.Login)
>>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
>>> (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
>>> (org.apache.zookeeper.Login)
>>> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23
>>> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23
>>> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
>>> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
>>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new session
>>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
>>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
>>> (org.apache.zookeeper.server.persistence.FileTxnLog)
>>> . . .
>>>
>>>
>>> KAFKA
>>> Starting Kafka server is not going well yet although I see that
>>> interaction with Kerberos is successful (see relevant log below. the
>>> error is at the bottom)
>>> . . .
>>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
>>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
>>> (kafka.server.KafkaServer)
>>> [2016-02-23 13:26:11,519] INFO JAAS File name:
>>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
>>> (org.I0Itec.zkclient.ZkClient)
>>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
>>> (org.I0Itec.zkclient.ZkEventThread)
>>> [2016-02-23 13:26:11,527] INFO Client
>>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
>>> GMT (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
>>> Corporation (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,527] INFO Client
>>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,527] INFO Client
>>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
>>> . . . . .
>>> [2016-02-23 13:26:11,531] INFO Client
>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client
>>> environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client
>>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,532] INFO Initiating client connection,
>>> connectString=localhost:2181 sessionTimeout=6000
>>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
>>> (org.I0Itec.zkclient.ZkClient)
>>> Debug is true storeKey true useTicketCache false useKeyTab true
>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
>>> refreshKrb5Config is false principal is
>>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>>> is false
>>> principal is
>>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>> Will use keytab
>>> Commit Succeeded
>>>
>>> [2016-02-23 13:26:11,734] INFO successfully logged in.
>>> (org.apache.zookeeper.Login)
>>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
>>> (org.apache.zookeeper.Login)
>>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
>>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
>>> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
>>> [2016-02-23 13:26:11,748] INFO Socket connection established to
>>> localhost/127.0.0.1:2181, initiating session
>>> (org.apache.zookeeper.ClientCnxn)
>>> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23
>>> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23
>>> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
>>> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:26:11,761] INFO Session establishment complete on server
>>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
>>> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
>>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
>>> (org.I0Itec.zkclient.ZkClient)
>>> [2016-02-23 13:26:11,773] ERROR An error:
>>> (java.security.PrivilegedActionException:
>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>> GSSException: No valid credentials provided (Mechanism level: Server not
>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>>> Client will go to AUTH_FAILED state.
>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
>>> member failed: javax.security.sasl.SaslException: An error:
>>> (java.security.PrivilegedActionException:
>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>> GSSException: No valid credentials provided (Mechanism level: Server not
>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
>>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
>>> (org.I0Itec.zkclient.ZkClient)
>>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
>>> (org.I0Itec.zkclient.ZkEventThread)
>>> . . .
>>>
>>> Any pointers?
>>>
>>> Cheers
>>> Oleg
>>
>
Re: Kerberized Kafka setup issues
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
Well, I am running on the same machine, so I say yes
Sent from my iPhone
> On Feb 23, 2016, at 18:05, Martin Gainty <mg...@hotmail.com> wrote:
>
> one more thing to check:
>
> specifically are the /etc/krb5.conf credentials the same you use to authenticate to ubuntu.oleg.com
>
> ?
> Martin
> __________________
>
>
>
>> Subject: Re: Kerberized Kafka setup issues
>> From: ozhurakousky@hortonworks.com
>> To: users@kafka.apache.org
>> Date: Tue, 23 Feb 2016 21:58:48 +0000
>>
>> Harsh
>>
>> I followed this blog (http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption) and got an environment via vagrant setup, no issues. I’ll poke around what the differences are and if find the issue will post.
>> Thanks for your help anyway.
>>
>> Cheers
>> Oleg
>> On Feb 23, 2016, at 4:06 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
>>
>> Yeah, I noticed the localhost as well, but I’ve changed it since to FQDN and it is still the same including 'sname is zookeeper/localhost@OLEG.COM<ma...@oleg.com>’
>>
>> Oleg
>>
>> On Feb 23, 2016, at 4:00 PM, Harsha <ka...@harsha.io>> wrote:
>>
>> whats your zookeeper.connect in server.properties looks like. Did you
>> use the hostname or localhost
>> -Harsha
>>
>> On Tue, Feb 23, 2016, at 12:01 PM, Oleg Zhurakousky wrote:
>> Still digging, but here is more info that may help
>>
>> 2016-02-23 14:59:24,240] INFO zookeeper state changed (SyncConnected)
>> (org.I0Itec.zkclient.ZkClient)
>> Found ticket for kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com> to go to
>> krbtgt/OLEG.COM@OLEG.COM<ma...@oleg.com> expiring on Wed Feb 24 00:59:24 EST 2016
>> Entered Krb5Context.initSecContext with state=STATE_NEW
>> Found ticket for kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com> to go to
>> krbtgt/OLEG.COM@OLEG.COM<ma...@oleg.com> expiring on Wed Feb 24 00:59:24 EST 2016
>> Service ticket not found in the subject
>> Credentials acquireServiceCreds: same realm
>> Using builtin default etypes for default_tgs_enctypes
>> default etypes for default_tgs_enctypes: 17 16 23.
>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
>> KrbKdcReq send: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88, timeout=30000, number of retries =3, #bytes=660
>> KDCCommunication: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88, timeout=30000,Attempt =1, #bytes=660
>> KrbKdcReq send: #bytes read=183
>> KdcAccessibility: remove ubuntu.oleg.com<http://ubuntu.oleg.com>
>> KDCRep: init() encoding tag is 126 req type is 13
>> KRBError:
>> cTime is Sat Aug 01 11:32:55 EDT 1998 901985575000
>> sTime is Tue Feb 23 14:59:24 EST 2016 1456257564000
>> suSec is 248635
>> error code is 7
>> error Message is Server not found in Kerberos database
>> cname is kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>> sname is zookeeper/localhost@OLEG.COM<ma...@oleg.com>
>> msgType is 30
>>
>> On Feb 23, 2016, at 2:46 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
>>
>> No joy. the same error
>>
>> KafkaServer {
>> com.sun.security.auth.module.Krb5LoginModule required
>> debug=true
>> useKeyTab=true
>> storeKey=true
>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
>> };
>> Client {
>> com.sun.security.auth.module.Krb5LoginModule required
>> debug=true
>> useKeyTab=true
>> serviceName=zookeeper
>> storeKey=true
>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
>> };
>> On Feb 23, 2016, at 2:41 PM, Harsha <ma...@harsha.io>> wrote:
>>
>> My bad it should be under Client section
>>
>> Client {
>> com.sun.security.auth.module.Krb5LoginModule required
>> debug=true
>> useKeyTab=true
>> storeKey=true
>> serviceName=zookeeper
>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
>> };
>>
>> -Harsha
>>
>> On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote:
>> can you try adding "serviceName=zookeeper" to KafkaServer section like
>> KafkaServer {
>> com.sun.security.auth.module.Krb5LoginModule required
>> debug=true
>> useKeyTab=true
>> storeKey=true
>> serviceName=zookeeper
>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
>> };
>>
>> On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
>> More info
>>
>> I am starting both services as myself ‘oleg’. Validated that both key tab
>> files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
>> and Kafka as ‘kafka’
>>
>> Oleg
>>
>> On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
>>
>> Harsha
>>
>> Thanks for following up. Here is is:
>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf
>> KafkaServer {
>> com.sun.security.auth.module.Krb5LoginModule required
>> debug=true
>> useKeyTab=true
>> storeKey=true
>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
>> };
>> Client {
>> com.sun.security.auth.module.Krb5LoginModule required
>> debug=true
>> useKeyTab=true
>> storeKey=true
>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
>> };
>>
>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf
>> Server {
>> com.sun.security.auth.module.Krb5LoginModule required
>> debug=true
>> useKeyTab=true
>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
>> storeKey=true
>> useTicketCache=false
>> principal="zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
>> };
>>
>> Cheers
>> Oleg
>>
>> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io>> wrote:
>>
>> Oleg,
>> Can you post your jaas configs. Its important that serviceName
>> must match the principal name with which zookeeper is running.
>> Whats the principal name zookeeper service is running with.
>> -Harsha
>>
>> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
>> Hey guys, first post here so bare with me
>>
>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
>> http://kafka.apache.org/documentation.html#security_sasl and i seem to be
>> very close, but not quite there yet.
>>
>> ZOOKEEPER
>> Starting Zookeeper seems to be OK (below is the relevant part of the log)
>> . . .
>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
>> (org.apache.zookeeper.server.ZooKeeperServer)
>> Debug is true storeKey true useTicketCache false useKeyTab true
>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
>> refreshKrb5Config is false principal is
>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>> is false
>> principal is
>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>> Will use keytab
>> Commit Succeeded
>>
>> [2016-02-23 13:22:40,541] INFO successfully logged in.
>> (org.apache.zookeeper.Login)
>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
>> (org.apache.zookeeper.server.NIOServerCnxnFactory)
>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
>> (org.apache.zookeeper.Login)
>> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23
>> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
>> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23
>> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
>> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new session
>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
>> (org.apache.zookeeper.server.persistence.FileTxnLog)
>> . . .
>>
>>
>> KAFKA
>> Starting Kafka server is not going well yet although I see that
>> interaction with Kerberos is successful (see relevant log below. the
>> error is at the bottom)
>> . . .
>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
>> (kafka.server.KafkaServer)
>> [2016-02-23 13:26:11,519] INFO JAAS File name:
>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
>> (org.I0Itec.zkclient.ZkClient)
>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
>> (org.I0Itec.zkclient.ZkEventThread)
>> [2016-02-23 13:26:11,527] INFO Client
>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
>> GMT (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
>> Corporation (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,527] INFO Client
>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,527] INFO Client
>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
>> . . . . .
>> [2016-02-23 13:26:11,531] INFO Client
>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client
>> environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client
>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,532] INFO Initiating client connection,
>> connectString=localhost:2181 sessionTimeout=6000
>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
>> (org.I0Itec.zkclient.ZkClient)
>> Debug is true storeKey true useTicketCache false useKeyTab true
>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
>> refreshKrb5Config is false principal is
>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>> is false
>> principal is
>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>> Will use keytab
>> Commit Succeeded
>>
>> [2016-02-23 13:26:11,734] INFO successfully logged in.
>> (org.apache.zookeeper.Login)
>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
>> (org.apache.zookeeper.Login)
>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
>> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
>> [2016-02-23 13:26:11,748] INFO Socket connection established to
>> localhost/127.0.0.1:2181, initiating session
>> (org.apache.zookeeper.ClientCnxn)
>> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23
>> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
>> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23
>> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
>> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
>> [2016-02-23 13:26:11,761] INFO Session establishment complete on server
>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
>> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
>> (org.I0Itec.zkclient.ZkClient)
>> [2016-02-23 13:26:11,773] ERROR An error:
>> (java.security.PrivilegedActionException:
>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>> GSSException: No valid credentials provided (Mechanism level: Server not
>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>> Client will go to AUTH_FAILED state.
>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
>> member failed: javax.security.sasl.SaslException: An error:
>> (java.security.PrivilegedActionException:
>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>> GSSException: No valid credentials provided (Mechanism level: Server not
>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
>> (org.I0Itec.zkclient.ZkClient)
>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
>> (org.I0Itec.zkclient.ZkEventThread)
>> . . .
>>
>> Any pointers?
>>
>> Cheers
>> Oleg
>
RE: Kerberized Kafka setup issues
Posted by Martin Gainty <mg...@hotmail.com>.
one more thing to check:
specifically are the /etc/krb5.conf credentials the same you use to authenticate to ubuntu.oleg.com
?
Martin
__________________
> Subject: Re: Kerberized Kafka setup issues
> From: ozhurakousky@hortonworks.com
> To: users@kafka.apache.org
> Date: Tue, 23 Feb 2016 21:58:48 +0000
>
> Harsh
>
> I followed this blog (http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption) and got an environment via vagrant setup, no issues. I’ll poke around what the differences are and if find the issue will post.
> Thanks for your help anyway.
>
> Cheers
> Oleg
> On Feb 23, 2016, at 4:06 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
>
> Yeah, I noticed the localhost as well, but I’ve changed it since to FQDN and it is still the same including 'sname is zookeeper/localhost@OLEG.COM<ma...@oleg.com>’
>
> Oleg
>
> On Feb 23, 2016, at 4:00 PM, Harsha <ka...@harsha.io>> wrote:
>
> whats your zookeeper.connect in server.properties looks like. Did you
> use the hostname or localhost
> -Harsha
>
> On Tue, Feb 23, 2016, at 12:01 PM, Oleg Zhurakousky wrote:
> Still digging, but here is more info that may help
>
> 2016-02-23 14:59:24,240] INFO zookeeper state changed (SyncConnected)
> (org.I0Itec.zkclient.ZkClient)
> Found ticket for kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com> to go to
> krbtgt/OLEG.COM@OLEG.COM<ma...@oleg.com> expiring on Wed Feb 24 00:59:24 EST 2016
> Entered Krb5Context.initSecContext with state=STATE_NEW
> Found ticket for kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com> to go to
> krbtgt/OLEG.COM@OLEG.COM<ma...@oleg.com> expiring on Wed Feb 24 00:59:24 EST 2016
> Service ticket not found in the subject
> Credentials acquireServiceCreds: same realm
> Using builtin default etypes for default_tgs_enctypes
> default etypes for default_tgs_enctypes: 17 16 23.
> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
> KrbKdcReq send: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88, timeout=30000, number of retries =3, #bytes=660
> KDCCommunication: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88, timeout=30000,Attempt =1, #bytes=660
> KrbKdcReq send: #bytes read=183
> KdcAccessibility: remove ubuntu.oleg.com<http://ubuntu.oleg.com>
> KDCRep: init() encoding tag is 126 req type is 13
> KRBError:
> cTime is Sat Aug 01 11:32:55 EDT 1998 901985575000
> sTime is Tue Feb 23 14:59:24 EST 2016 1456257564000
> suSec is 248635
> error code is 7
> error Message is Server not found in Kerberos database
> cname is kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> sname is zookeeper/localhost@OLEG.COM<ma...@oleg.com>
> msgType is 30
>
> On Feb 23, 2016, at 2:46 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
>
> No joy. the same error
>
> KafkaServer {
> com.sun.security.auth.module.Krb5LoginModule required
> debug=true
> useKeyTab=true
> storeKey=true
> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
> };
> Client {
> com.sun.security.auth.module.Krb5LoginModule required
> debug=true
> useKeyTab=true
> serviceName=zookeeper
> storeKey=true
> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
> };
> On Feb 23, 2016, at 2:41 PM, Harsha <ma...@harsha.io>> wrote:
>
> My bad it should be under Client section
>
> Client {
> com.sun.security.auth.module.Krb5LoginModule required
> debug=true
> useKeyTab=true
> storeKey=true
> serviceName=zookeeper
> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
> };
>
> -Harsha
>
> On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote:
> can you try adding "serviceName=zookeeper" to KafkaServer section like
> KafkaServer {
> com.sun.security.auth.module.Krb5LoginModule required
> debug=true
> useKeyTab=true
> storeKey=true
> serviceName=zookeeper
> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
> };
>
> On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
> More info
>
> I am starting both services as myself ‘oleg’. Validated that both key tab
> files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
> and Kafka as ‘kafka’
>
> Oleg
>
> On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
>
> Harsha
>
> Thanks for following up. Here is is:
> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf
> KafkaServer {
> com.sun.security.auth.module.Krb5LoginModule required
> debug=true
> useKeyTab=true
> storeKey=true
> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
> };
> Client {
> com.sun.security.auth.module.Krb5LoginModule required
> debug=true
> useKeyTab=true
> storeKey=true
> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
> };
>
> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf
> Server {
> com.sun.security.auth.module.Krb5LoginModule required
> debug=true
> useKeyTab=true
> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
> storeKey=true
> useTicketCache=false
> principal="zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
> };
>
> Cheers
> Oleg
>
> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io>> wrote:
>
> Oleg,
> Can you post your jaas configs. Its important that serviceName
> must match the principal name with which zookeeper is running.
> Whats the principal name zookeeper service is running with.
> -Harsha
>
> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
> Hey guys, first post here so bare with me
>
> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
> http://kafka.apache.org/documentation.html#security_sasl and i seem to be
> very close, but not quite there yet.
>
> ZOOKEEPER
> Starting Zookeeper seems to be OK (below is the relevant part of the log)
> . . .
> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
> (org.apache.zookeeper.server.ZooKeeperServer)
> Debug is true storeKey true useTicketCache false useKeyTab true
> doNotPrompt false ticketCache is null isInitiator true KeyTab is
> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
> refreshKrb5Config is false principal is
> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> tryFirstPass is false useFirstPass is false storePass is false clearPass
> is false
> principal is
> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> Will use keytab
> Commit Succeeded
>
> [2016-02-23 13:22:40,541] INFO successfully logged in.
> (org.apache.zookeeper.Login)
> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
> (org.apache.zookeeper.server.NIOServerCnxnFactory)
> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
> (org.apache.zookeeper.Login)
> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23
> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23
> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> [2016-02-23 13:23:09,025] INFO Client attempting to establish new session
> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
> (org.apache.zookeeper.server.persistence.FileTxnLog)
> . . .
>
>
> KAFKA
> Starting Kafka server is not going well yet although I see that
> interaction with Kerberos is successful (see relevant log below. the
> error is at the bottom)
> . . .
> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
> (kafka.server.KafkaServer)
> [2016-02-23 13:26:11,519] INFO JAAS File name:
> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
> (org.I0Itec.zkclient.ZkClient)
> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
> (org.I0Itec.zkclient.ZkEventThread)
> [2016-02-23 13:26:11,527] INFO Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> GMT (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
> Corporation (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,527] INFO Client
> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,527] INFO Client
> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
> . . . . .
> [2016-02-23 13:26:11,531] INFO Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client
> environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client
> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,532] INFO Initiating client connection,
> connectString=localhost:2181 sessionTimeout=6000
> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
> (org.I0Itec.zkclient.ZkClient)
> Debug is true storeKey true useTicketCache false useKeyTab true
> doNotPrompt false ticketCache is null isInitiator true KeyTab is
> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
> refreshKrb5Config is false principal is
> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> tryFirstPass is false useFirstPass is false storePass is false clearPass
> is false
> principal is
> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> Will use keytab
> Commit Succeeded
>
> [2016-02-23 13:26:11,734] INFO successfully logged in.
> (org.apache.zookeeper.Login)
> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
> (org.apache.zookeeper.Login)
> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
> (org.apache.zookeeper.client.ZooKeeperSaslClient)
> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
> [2016-02-23 13:26:11,748] INFO Socket connection established to
> localhost/127.0.0.1:2181, initiating session
> (org.apache.zookeeper.ClientCnxn)
> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23
> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23
> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
> [2016-02-23 13:26:11,761] INFO Session establishment complete on server
> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
> (org.I0Itec.zkclient.ZkClient)
> [2016-02-23 13:26:11,773] ERROR An error:
> (java.security.PrivilegedActionException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
> Client will go to AUTH_FAILED state.
> (org.apache.zookeeper.client.ZooKeeperSaslClient)
> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
> member failed: javax.security.sasl.SaslException: An error:
> (java.security.PrivilegedActionException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
> (org.I0Itec.zkclient.ZkClient)
> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
> (org.I0Itec.zkclient.ZkEventThread)
> . . .
>
> Any pointers?
>
> Cheers
> Oleg
>
>
>
>
>
>
>
>
>
>
>
Re: Kerberized Kafka setup issues
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
Harsh
I followed this blog (http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption) and got an environment via vagrant setup, no issues. I’ll poke around what the differences are and if find the issue will post.
Thanks for your help anyway.
Cheers
Oleg
On Feb 23, 2016, at 4:06 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
Yeah, I noticed the localhost as well, but I’ve changed it since to FQDN and it is still the same including 'sname is zookeeper/localhost@OLEG.COM<ma...@oleg.com>’
Oleg
On Feb 23, 2016, at 4:00 PM, Harsha <ka...@harsha.io>> wrote:
whats your zookeeper.connect in server.properties looks like. Did you
use the hostname or localhost
-Harsha
On Tue, Feb 23, 2016, at 12:01 PM, Oleg Zhurakousky wrote:
Still digging, but here is more info that may help
2016-02-23 14:59:24,240] INFO zookeeper state changed (SyncConnected)
(org.I0Itec.zkclient.ZkClient)
Found ticket for kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com> to go to
krbtgt/OLEG.COM@OLEG.COM<ma...@oleg.com> expiring on Wed Feb 24 00:59:24 EST 2016
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com> to go to
krbtgt/OLEG.COM@OLEG.COM<ma...@oleg.com> expiring on Wed Feb 24 00:59:24 EST 2016
Service ticket not found in the subject
Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 17 16 23.
CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
KrbKdcReq send: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88, timeout=30000, number of retries =3, #bytes=660
KDCCommunication: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88, timeout=30000,Attempt =1, #bytes=660
KrbKdcReq send: #bytes read=183
KdcAccessibility: remove ubuntu.oleg.com<http://ubuntu.oleg.com>
KDCRep: init() encoding tag is 126 req type is 13
KRBError:
cTime is Sat Aug 01 11:32:55 EDT 1998 901985575000
sTime is Tue Feb 23 14:59:24 EST 2016 1456257564000
suSec is 248635
error code is 7
error Message is Server not found in Kerberos database
cname is kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
sname is zookeeper/localhost@OLEG.COM<ma...@oleg.com>
msgType is 30
On Feb 23, 2016, at 2:46 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
No joy. the same error
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
storeKey=true
keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
serviceName=zookeeper
storeKey=true
keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
};
On Feb 23, 2016, at 2:41 PM, Harsha <ma...@harsha.io>> wrote:
My bad it should be under Client section
Client {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
storeKey=true
serviceName=zookeeper
keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
};
-Harsha
On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote:
can you try adding "serviceName=zookeeper" to KafkaServer section like
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
storeKey=true
serviceName=zookeeper
keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
};
On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
More info
I am starting both services as myself ‘oleg’. Validated that both key tab
files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
and Kafka as ‘kafka’
Oleg
On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
Harsha
Thanks for following up. Here is is:
oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
storeKey=true
keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
storeKey=true
keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
principal="kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
};
oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf
Server {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
storeKey=true
useTicketCache=false
principal="zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>";
};
Cheers
Oleg
On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io>> wrote:
Oleg,
Can you post your jaas configs. Its important that serviceName
must match the principal name with which zookeeper is running.
Whats the principal name zookeeper service is running with.
-Harsha
On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
Hey guys, first post here so bare with me
Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
http://kafka.apache.org/documentation.html#security_sasl and i seem to be
very close, but not quite there yet.
ZOOKEEPER
Starting Zookeeper seems to be OK (below is the relevant part of the log)
. . .
[2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
(org.apache.zookeeper.server.ZooKeeperServer)
Debug is true storeKey true useTicketCache false useKeyTab true
doNotPrompt false ticketCache is null isInitiator true KeyTab is
/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
refreshKrb5Config is false principal is
zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
tryFirstPass is false useFirstPass is false storePass is false clearPass
is false
principal is
zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
Will use keytab
Commit Succeeded
[2016-02-23 13:22:40,541] INFO successfully logged in.
(org.apache.zookeeper.Login)
[2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
(org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-02-23 13:22:40,544] INFO TGT refresh thread started.
(org.apache.zookeeper.Login)
[2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23
13:22:40 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23
23:22:40 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
21:47:35 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:23:09,012] INFO Accepted socket connection from
/127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-02-23 13:23:09,025] INFO Client attempting to establish new session
at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-02-23 13:23:09,026] INFO Creating new log file: log.57
(org.apache.zookeeper.server.persistence.FileTxnLog)
. . .
KAFKA
Starting Kafka server is not going well yet although I see that
interaction with Kerberos is successful (see relevant log below. the
error is at the bottom)
. . .
[2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
[2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
(kafka.server.KafkaServer)
[2016-02-23 13:26:11,519] INFO JAAS File name:
/home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
(org.I0Itec.zkclient.ZkClient)
[2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2016-02-23 13:26:11,527] INFO Client
environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
GMT (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
Corporation (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,527] INFO Client
environment:java.home=/usr/lib/jvm/java-8-oracle/jre
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,527] INFO Client
environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
. . . . .
[2016-02-23 13:26:11,531] INFO Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client
environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,531] INFO Client
environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,532] INFO Initiating client connection,
connectString=localhost:2181 sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
(org.apache.zookeeper.ZooKeeper)
[2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
(org.I0Itec.zkclient.ZkClient)
Debug is true storeKey true useTicketCache false useKeyTab true
doNotPrompt false ticketCache is null isInitiator true KeyTab is
/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
refreshKrb5Config is false principal is
kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
tryFirstPass is false useFirstPass is false storePass is false clearPass
is false
principal is
kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
Will use keytab
Commit Succeeded
[2016-02-23 13:26:11,734] INFO successfully logged in.
(org.apache.zookeeper.Login)
[2016-02-23 13:26:11,735] INFO TGT refresh thread started.
(org.apache.zookeeper.Login)
[2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
(org.apache.zookeeper.client.ZooKeeperSaslClient)
[2016-02-23 13:26:11,743] INFO Opening socket connection to server
localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2016-02-23 13:26:11,748] INFO Socket connection established to
localhost/127.0.0.1:2181, initiating session
(org.apache.zookeeper.ClientCnxn)
[2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23
13:26:11 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23
23:26:11 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
21:40:22 EST 2016 (org.apache.zookeeper.Login)
[2016-02-23 13:26:11,761] INFO Session establishment complete on server
localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
(org.I0Itec.zkclient.ZkClient)
[2016-02-23 13:26:11,773] ERROR An error:
(java.security.PrivilegedActionException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not
found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
Client will go to AUTH_FAILED state.
(org.apache.zookeeper.client.ZooKeeperSaslClient)
[2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
member failed: javax.security.sasl.SaslException: An error:
(java.security.PrivilegedActionException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not
found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
[2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
(org.I0Itec.zkclient.ZkClient)
[2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
. . .
Any pointers?
Cheers
Oleg
Re: Kerberized Kafka setup issues
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
Yeah, I noticed the localhost as well, but I’ve changed it since to FQDN and it is still the same including 'sname is zookeeper/localhost@OLEG.COM’
Oleg
> On Feb 23, 2016, at 4:00 PM, Harsha <ka...@harsha.io> wrote:
>
> whats your zookeeper.connect in server.properties looks like. Did you
> use the hostname or localhost
> -Harsha
>
> On Tue, Feb 23, 2016, at 12:01 PM, Oleg Zhurakousky wrote:
>> Still digging, but here is more info that may help
>>
>> 2016-02-23 14:59:24,240] INFO zookeeper state changed (SyncConnected)
>> (org.I0Itec.zkclient.ZkClient)
>> Found ticket for kafka/ubuntu.oleg.com@OLEG.COM to go to
>> krbtgt/OLEG.COM@OLEG.COM expiring on Wed Feb 24 00:59:24 EST 2016
>> Entered Krb5Context.initSecContext with state=STATE_NEW
>> Found ticket for kafka/ubuntu.oleg.com@OLEG.COM to go to
>> krbtgt/OLEG.COM@OLEG.COM expiring on Wed Feb 24 00:59:24 EST 2016
>> Service ticket not found in the subject
>>>>> Credentials acquireServiceCreds: same realm
>> Using builtin default etypes for default_tgs_enctypes
>> default etypes for default_tgs_enctypes: 17 16 23.
>>>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
>>>>> KrbKdcReq send: kdc=ubuntu.oleg.com UDP:88, timeout=30000, number of retries =3, #bytes=660
>>>>> KDCCommunication: kdc=ubuntu.oleg.com UDP:88, timeout=30000,Attempt =1, #bytes=660
>>>>> KrbKdcReq send: #bytes read=183
>>>>> KdcAccessibility: remove ubuntu.oleg.com
>>>>> KDCRep: init() encoding tag is 126 req type is 13
>>>>> KRBError:
>> cTime is Sat Aug 01 11:32:55 EDT 1998 901985575000
>> sTime is Tue Feb 23 14:59:24 EST 2016 1456257564000
>> suSec is 248635
>> error code is 7
>> error Message is Server not found in Kerberos database
>> cname is kafka/ubuntu.oleg.com@OLEG.COM
>> sname is zookeeper/localhost@OLEG.COM
>> msgType is 30
>>
>>> On Feb 23, 2016, at 2:46 PM, Oleg Zhurakousky <oz...@hortonworks.com> wrote:
>>>
>>> No joy. the same error
>>>
>>> KafkaServer {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> storeKey=true
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
>>> };
>>> Client {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> serviceName=zookeeper
>>> storeKey=true
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
>>> };
>>>> On Feb 23, 2016, at 2:41 PM, Harsha <ma...@harsha.io> wrote:
>>>>
>>>> My bad it should be under Client section
>>>>
>>>> Client {
>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>> debug=true
>>>> useKeyTab=true
>>>> storeKey=true
>>>> serviceName=zookeeper
>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
>>>> };
>>>>
>>>> -Harsha
>>>>
>>>> On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote:
>>>>> can you try adding "serviceName=zookeeper" to KafkaServer section like
>>>>> KafkaServer {
>>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>>> debug=true
>>>>> useKeyTab=true
>>>>> storeKey=true
>>>>> serviceName=zookeeper
>>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
>>>>> };
>>>>>
>>>>> On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
>>>>>> More info
>>>>>>
>>>>>> I am starting both services as myself ‘oleg’. Validated that both key tab
>>>>>> files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
>>>>>> and Kafka as ‘kafka’
>>>>>>
>>>>>> Oleg
>>>>>>
>>>>>>> On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky <oz...@hortonworks.com> wrote:
>>>>>>>
>>>>>>> Harsha
>>>>>>>
>>>>>>> Thanks for following up. Here is is:
>>>>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf
>>>>>>> KafkaServer {
>>>>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>>>>> debug=true
>>>>>>> useKeyTab=true
>>>>>>> storeKey=true
>>>>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>>>>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
>>>>>>> };
>>>>>>> Client {
>>>>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>>>>> debug=true
>>>>>>> useKeyTab=true
>>>>>>> storeKey=true
>>>>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>>>>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
>>>>>>> };
>>>>>>>
>>>>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf
>>>>>>> Server {
>>>>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>>>>> debug=true
>>>>>>> useKeyTab=true
>>>>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
>>>>>>> storeKey=true
>>>>>>> useTicketCache=false
>>>>>>> principal="zookeeper/ubuntu.oleg.com@OLEG.COM";
>>>>>>> };
>>>>>>>
>>>>>>> Cheers
>>>>>>> Oleg
>>>>>>>
>>>>>>>> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io> wrote:
>>>>>>>>
>>>>>>>> Oleg,
>>>>>>>> Can you post your jaas configs. Its important that serviceName
>>>>>>>> must match the principal name with which zookeeper is running.
>>>>>>>> Whats the principal name zookeeper service is running with.
>>>>>>>> -Harsha
>>>>>>>>
>>>>>>>> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
>>>>>>>>> Hey guys, first post here so bare with me
>>>>>>>>>
>>>>>>>>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
>>>>>>>>> http://kafka.apache.org/documentation.html#security_sasl and i seem to be
>>>>>>>>> very close, but not quite there yet.
>>>>>>>>>
>>>>>>>>> ZOOKEEPER
>>>>>>>>> Starting Zookeeper seems to be OK (below is the relevant part of the log)
>>>>>>>>> . . .
>>>>>>>>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
>>>>>>>>> (org.apache.zookeeper.server.ZooKeeperServer)
>>>>>>>>> Debug is true storeKey true useTicketCache false useKeyTab true
>>>>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>>>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
>>>>>>>>> refreshKrb5Config is false principal is
>>>>>>>>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>>>>>>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>>>>>>>>> is false
>>>>>>>>> principal is
>>>>>>>>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>>>>>>>> Will use keytab
>>>>>>>>> Commit Succeeded
>>>>>>>>>
>>>>>>>>> [2016-02-23 13:22:40,541] INFO successfully logged in.
>>>>>>>>> (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
>>>>>>>>> (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>>>>>>>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
>>>>>>>>> (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23
>>>>>>>>> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23
>>>>>>>>> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
>>>>>>>>> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
>>>>>>>>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>>>>>>>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new session
>>>>>>>>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
>>>>>>>>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
>>>>>>>>> (org.apache.zookeeper.server.persistence.FileTxnLog)
>>>>>>>>> . . .
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> KAFKA
>>>>>>>>> Starting Kafka server is not going well yet although I see that
>>>>>>>>> interaction with Kerberos is successful (see relevant log below. the
>>>>>>>>> error is at the bottom)
>>>>>>>>> . . .
>>>>>>>>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
>>>>>>>>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
>>>>>>>>> (kafka.server.KafkaServer)
>>>>>>>>> [2016-02-23 13:26:11,519] INFO JAAS File name:
>>>>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
>>>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>>>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
>>>>>>>>> (org.I0Itec.zkclient.ZkEventThread)
>>>>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>>>>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
>>>>>>>>> GMT (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
>>>>>>>>> Corporation (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>>>>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>>>>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
>>>>>>>>> . . . . .
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client
>>>>>>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client
>>>>>>>>> environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client
>>>>>>>>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,532] INFO Initiating client connection,
>>>>>>>>> connectString=localhost:2181 sessionTimeout=6000
>>>>>>>>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
>>>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>>>> Debug is true storeKey true useTicketCache false useKeyTab true
>>>>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>>>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
>>>>>>>>> refreshKrb5Config is false principal is
>>>>>>>>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>>>>>>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>>>>>>>>> is false
>>>>>>>>> principal is
>>>>>>>>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>>>>>>>> Will use keytab
>>>>>>>>> Commit Succeeded
>>>>>>>>>
>>>>>>>>> [2016-02-23 13:26:11,734] INFO successfully logged in.
>>>>>>>>> (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
>>>>>>>>> (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
>>>>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>>>>>>>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
>>>>>>>>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
>>>>>>>>> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
>>>>>>>>> [2016-02-23 13:26:11,748] INFO Socket connection established to
>>>>>>>>> localhost/127.0.0.1:2181, initiating session
>>>>>>>>> (org.apache.zookeeper.ClientCnxn)
>>>>>>>>> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23
>>>>>>>>> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23
>>>>>>>>> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
>>>>>>>>> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:26:11,761] INFO Session establishment complete on server
>>>>>>>>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
>>>>>>>>> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
>>>>>>>>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
>>>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>>>> [2016-02-23 13:26:11,773] ERROR An error:
>>>>>>>>> (java.security.PrivilegedActionException:
>>>>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>>>>>>>> GSSException: No valid credentials provided (Mechanism level: Server not
>>>>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>>>>>>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>>>>>>>>> Client will go to AUTH_FAILED state.
>>>>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>>>>>>>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
>>>>>>>>> member failed: javax.security.sasl.SaslException: An error:
>>>>>>>>> (java.security.PrivilegedActionException:
>>>>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>>>>>>>> GSSException: No valid credentials provided (Mechanism level: Server not
>>>>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>>>>>>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>>>>>>>>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
>>>>>>>>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
>>>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>>>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
>>>>>>>>> (org.I0Itec.zkclient.ZkEventThread)
>>>>>>>>> . . .
>>>>>>>>>
>>>>>>>>> Any pointers?
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>> Oleg
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>
>
Re: Kerberized Kafka setup issues
Posted by Harsha <ka...@harsha.io>.
whats your zookeeper.connect in server.properties looks like. Did you
use the hostname or localhost
-Harsha
On Tue, Feb 23, 2016, at 12:01 PM, Oleg Zhurakousky wrote:
> Still digging, but here is more info that may help
>
> 2016-02-23 14:59:24,240] INFO zookeeper state changed (SyncConnected)
> (org.I0Itec.zkclient.ZkClient)
> Found ticket for kafka/ubuntu.oleg.com@OLEG.COM to go to
> krbtgt/OLEG.COM@OLEG.COM expiring on Wed Feb 24 00:59:24 EST 2016
> Entered Krb5Context.initSecContext with state=STATE_NEW
> Found ticket for kafka/ubuntu.oleg.com@OLEG.COM to go to
> krbtgt/OLEG.COM@OLEG.COM expiring on Wed Feb 24 00:59:24 EST 2016
> Service ticket not found in the subject
> >>> Credentials acquireServiceCreds: same realm
> Using builtin default etypes for default_tgs_enctypes
> default etypes for default_tgs_enctypes: 17 16 23.
> >>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
> >>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
> >>> KrbKdcReq send: kdc=ubuntu.oleg.com UDP:88, timeout=30000, number of retries =3, #bytes=660
> >>> KDCCommunication: kdc=ubuntu.oleg.com UDP:88, timeout=30000,Attempt =1, #bytes=660
> >>> KrbKdcReq send: #bytes read=183
> >>> KdcAccessibility: remove ubuntu.oleg.com
> >>> KDCRep: init() encoding tag is 126 req type is 13
> >>>KRBError:
> cTime is Sat Aug 01 11:32:55 EDT 1998 901985575000
> sTime is Tue Feb 23 14:59:24 EST 2016 1456257564000
> suSec is 248635
> error code is 7
> error Message is Server not found in Kerberos database
> cname is kafka/ubuntu.oleg.com@OLEG.COM
> sname is zookeeper/localhost@OLEG.COM
> msgType is 30
>
> > On Feb 23, 2016, at 2:46 PM, Oleg Zhurakousky <oz...@hortonworks.com> wrote:
> >
> > No joy. the same error
> >
> > KafkaServer {
> > com.sun.security.auth.module.Krb5LoginModule required
> > debug=true
> > useKeyTab=true
> > storeKey=true
> > keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> > principal="kafka/ubuntu.oleg.com@OLEG.COM";
> > };
> > Client {
> > com.sun.security.auth.module.Krb5LoginModule required
> > debug=true
> > useKeyTab=true
> > serviceName=zookeeper
> > storeKey=true
> > keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> > principal="kafka/ubuntu.oleg.com@OLEG.COM";
> > };
> >> On Feb 23, 2016, at 2:41 PM, Harsha <ma...@harsha.io> wrote:
> >>
> >> My bad it should be under Client section
> >>
> >> Client {
> >> com.sun.security.auth.module.Krb5LoginModule required
> >> debug=true
> >> useKeyTab=true
> >> storeKey=true
> >> serviceName=zookeeper
> >> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >> principal="kafka/ubuntu.oleg.com@OLEG.COM";
> >> };
> >>
> >> -Harsha
> >>
> >> On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote:
> >>> can you try adding "serviceName=zookeeper" to KafkaServer section like
> >>> KafkaServer {
> >>> com.sun.security.auth.module.Krb5LoginModule required
> >>> debug=true
> >>> useKeyTab=true
> >>> storeKey=true
> >>> serviceName=zookeeper
> >>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
> >>> };
> >>>
> >>> On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
> >>>> More info
> >>>>
> >>>> I am starting both services as myself ‘oleg’. Validated that both key tab
> >>>> files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
> >>>> and Kafka as ‘kafka’
> >>>>
> >>>> Oleg
> >>>>
> >>>>> On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky <oz...@hortonworks.com> wrote:
> >>>>>
> >>>>> Harsha
> >>>>>
> >>>>> Thanks for following up. Here is is:
> >>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf
> >>>>> KafkaServer {
> >>>>> com.sun.security.auth.module.Krb5LoginModule required
> >>>>> debug=true
> >>>>> useKeyTab=true
> >>>>> storeKey=true
> >>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >>>>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
> >>>>> };
> >>>>> Client {
> >>>>> com.sun.security.auth.module.Krb5LoginModule required
> >>>>> debug=true
> >>>>> useKeyTab=true
> >>>>> storeKey=true
> >>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >>>>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
> >>>>> };
> >>>>>
> >>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf
> >>>>> Server {
> >>>>> com.sun.security.auth.module.Krb5LoginModule required
> >>>>> debug=true
> >>>>> useKeyTab=true
> >>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
> >>>>> storeKey=true
> >>>>> useTicketCache=false
> >>>>> principal="zookeeper/ubuntu.oleg.com@OLEG.COM";
> >>>>> };
> >>>>>
> >>>>> Cheers
> >>>>> Oleg
> >>>>>
> >>>>>> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io> wrote:
> >>>>>>
> >>>>>> Oleg,
> >>>>>> Can you post your jaas configs. Its important that serviceName
> >>>>>> must match the principal name with which zookeeper is running.
> >>>>>> Whats the principal name zookeeper service is running with.
> >>>>>> -Harsha
> >>>>>>
> >>>>>> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
> >>>>>>> Hey guys, first post here so bare with me
> >>>>>>>
> >>>>>>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
> >>>>>>> http://kafka.apache.org/documentation.html#security_sasl and i seem to be
> >>>>>>> very close, but not quite there yet.
> >>>>>>>
> >>>>>>> ZOOKEEPER
> >>>>>>> Starting Zookeeper seems to be OK (below is the relevant part of the log)
> >>>>>>> . . .
> >>>>>>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
> >>>>>>> (org.apache.zookeeper.server.ZooKeeperServer)
> >>>>>>> Debug is true storeKey true useTicketCache false useKeyTab true
> >>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
> >>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
> >>>>>>> refreshKrb5Config is false principal is
> >>>>>>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> >>>>>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
> >>>>>>> is false
> >>>>>>> principal is
> >>>>>>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> >>>>>>> Will use keytab
> >>>>>>> Commit Succeeded
> >>>>>>>
> >>>>>>> [2016-02-23 13:22:40,541] INFO successfully logged in.
> >>>>>>> (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
> >>>>>>> (org.apache.zookeeper.server.NIOServerCnxnFactory)
> >>>>>>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
> >>>>>>> (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23
> >>>>>>> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23
> >>>>>>> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
> >>>>>>> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
> >>>>>>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> >>>>>>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new session
> >>>>>>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
> >>>>>>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
> >>>>>>> (org.apache.zookeeper.server.persistence.FileTxnLog)
> >>>>>>> . . .
> >>>>>>>
> >>>>>>>
> >>>>>>> KAFKA
> >>>>>>> Starting Kafka server is not going well yet although I see that
> >>>>>>> interaction with Kerberos is successful (see relevant log below. the
> >>>>>>> error is at the bottom)
> >>>>>>> . . .
> >>>>>>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
> >>>>>>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
> >>>>>>> (kafka.server.KafkaServer)
> >>>>>>> [2016-02-23 13:26:11,519] INFO JAAS File name:
> >>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
> >>>>>>> (org.I0Itec.zkclient.ZkClient)
> >>>>>>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
> >>>>>>> (org.I0Itec.zkclient.ZkEventThread)
> >>>>>>> [2016-02-23 13:26:11,527] INFO Client
> >>>>>>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> >>>>>>> GMT (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
> >>>>>>> Corporation (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,527] INFO Client
> >>>>>>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,527] INFO Client
> >>>>>>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
> >>>>>>> . . . . .
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client
> >>>>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client
> >>>>>>> environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client
> >>>>>>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,532] INFO Initiating client connection,
> >>>>>>> connectString=localhost:2181 sessionTimeout=6000
> >>>>>>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
> >>>>>>> (org.I0Itec.zkclient.ZkClient)
> >>>>>>> Debug is true storeKey true useTicketCache false useKeyTab true
> >>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
> >>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
> >>>>>>> refreshKrb5Config is false principal is
> >>>>>>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> >>>>>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
> >>>>>>> is false
> >>>>>>> principal is
> >>>>>>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> >>>>>>> Will use keytab
> >>>>>>> Commit Succeeded
> >>>>>>>
> >>>>>>> [2016-02-23 13:26:11,734] INFO successfully logged in.
> >>>>>>> (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
> >>>>>>> (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
> >>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
> >>>>>>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
> >>>>>>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
> >>>>>>> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
> >>>>>>> [2016-02-23 13:26:11,748] INFO Socket connection established to
> >>>>>>> localhost/127.0.0.1:2181, initiating session
> >>>>>>> (org.apache.zookeeper.ClientCnxn)
> >>>>>>> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23
> >>>>>>> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23
> >>>>>>> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
> >>>>>>> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:26:11,761] INFO Session establishment complete on server
> >>>>>>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
> >>>>>>> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> >>>>>>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
> >>>>>>> (org.I0Itec.zkclient.ZkClient)
> >>>>>>> [2016-02-23 13:26:11,773] ERROR An error:
> >>>>>>> (java.security.PrivilegedActionException:
> >>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> >>>>>>> GSSException: No valid credentials provided (Mechanism level: Server not
> >>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
> >>>>>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
> >>>>>>> Client will go to AUTH_FAILED state.
> >>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
> >>>>>>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
> >>>>>>> member failed: javax.security.sasl.SaslException: An error:
> >>>>>>> (java.security.PrivilegedActionException:
> >>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> >>>>>>> GSSException: No valid credentials provided (Mechanism level: Server not
> >>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
> >>>>>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
> >>>>>>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
> >>>>>>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
> >>>>>>> (org.I0Itec.zkclient.ZkClient)
> >>>>>>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
> >>>>>>> (org.I0Itec.zkclient.ZkEventThread)
> >>>>>>> . . .
> >>>>>>>
> >>>>>>> Any pointers?
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>> Oleg
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> >
>
Re: Kerberized Kafka setup issues
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
Still digging, but here is more info that may help
2016-02-23 14:59:24,240] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
Found ticket for kafka/ubuntu.oleg.com@OLEG.COM to go to krbtgt/OLEG.COM@OLEG.COM expiring on Wed Feb 24 00:59:24 EST 2016
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for kafka/ubuntu.oleg.com@OLEG.COM to go to krbtgt/OLEG.COM@OLEG.COM expiring on Wed Feb 24 00:59:24 EST 2016
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 17 16 23.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
>>> KrbKdcReq send: kdc=ubuntu.oleg.com UDP:88, timeout=30000, number of retries =3, #bytes=660
>>> KDCCommunication: kdc=ubuntu.oleg.com UDP:88, timeout=30000,Attempt =1, #bytes=660
>>> KrbKdcReq send: #bytes read=183
>>> KdcAccessibility: remove ubuntu.oleg.com
>>> KDCRep: init() encoding tag is 126 req type is 13
>>>KRBError:
cTime is Sat Aug 01 11:32:55 EDT 1998 901985575000
sTime is Tue Feb 23 14:59:24 EST 2016 1456257564000
suSec is 248635
error code is 7
error Message is Server not found in Kerberos database
cname is kafka/ubuntu.oleg.com@OLEG.COM
sname is zookeeper/localhost@OLEG.COM
msgType is 30
> On Feb 23, 2016, at 2:46 PM, Oleg Zhurakousky <oz...@hortonworks.com> wrote:
>
> No joy. the same error
>
> KafkaServer {
> com.sun.security.auth.module.Krb5LoginModule required
> debug=true
> useKeyTab=true
> storeKey=true
> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> principal="kafka/ubuntu.oleg.com@OLEG.COM";
> };
> Client {
> com.sun.security.auth.module.Krb5LoginModule required
> debug=true
> useKeyTab=true
> serviceName=zookeeper
> storeKey=true
> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> principal="kafka/ubuntu.oleg.com@OLEG.COM";
> };
>> On Feb 23, 2016, at 2:41 PM, Harsha <ma...@harsha.io> wrote:
>>
>> My bad it should be under Client section
>>
>> Client {
>> com.sun.security.auth.module.Krb5LoginModule required
>> debug=true
>> useKeyTab=true
>> storeKey=true
>> serviceName=zookeeper
>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
>> };
>>
>> -Harsha
>>
>> On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote:
>>> can you try adding "serviceName=zookeeper" to KafkaServer section like
>>> KafkaServer {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> storeKey=true
>>> serviceName=zookeeper
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
>>> };
>>>
>>> On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
>>>> More info
>>>>
>>>> I am starting both services as myself ‘oleg’. Validated that both key tab
>>>> files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
>>>> and Kafka as ‘kafka’
>>>>
>>>> Oleg
>>>>
>>>>> On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky <oz...@hortonworks.com> wrote:
>>>>>
>>>>> Harsha
>>>>>
>>>>> Thanks for following up. Here is is:
>>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf
>>>>> KafkaServer {
>>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>>> debug=true
>>>>> useKeyTab=true
>>>>> storeKey=true
>>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
>>>>> };
>>>>> Client {
>>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>>> debug=true
>>>>> useKeyTab=true
>>>>> storeKey=true
>>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
>>>>> };
>>>>>
>>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf
>>>>> Server {
>>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>>> debug=true
>>>>> useKeyTab=true
>>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
>>>>> storeKey=true
>>>>> useTicketCache=false
>>>>> principal="zookeeper/ubuntu.oleg.com@OLEG.COM";
>>>>> };
>>>>>
>>>>> Cheers
>>>>> Oleg
>>>>>
>>>>>> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io> wrote:
>>>>>>
>>>>>> Oleg,
>>>>>> Can you post your jaas configs. Its important that serviceName
>>>>>> must match the principal name with which zookeeper is running.
>>>>>> Whats the principal name zookeeper service is running with.
>>>>>> -Harsha
>>>>>>
>>>>>> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
>>>>>>> Hey guys, first post here so bare with me
>>>>>>>
>>>>>>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
>>>>>>> http://kafka.apache.org/documentation.html#security_sasl and i seem to be
>>>>>>> very close, but not quite there yet.
>>>>>>>
>>>>>>> ZOOKEEPER
>>>>>>> Starting Zookeeper seems to be OK (below is the relevant part of the log)
>>>>>>> . . .
>>>>>>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
>>>>>>> (org.apache.zookeeper.server.ZooKeeperServer)
>>>>>>> Debug is true storeKey true useTicketCache false useKeyTab true
>>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
>>>>>>> refreshKrb5Config is false principal is
>>>>>>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>>>>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>>>>>>> is false
>>>>>>> principal is
>>>>>>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>>>>>> Will use keytab
>>>>>>> Commit Succeeded
>>>>>>>
>>>>>>> [2016-02-23 13:22:40,541] INFO successfully logged in.
>>>>>>> (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
>>>>>>> (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>>>>>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
>>>>>>> (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23
>>>>>>> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23
>>>>>>> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
>>>>>>> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
>>>>>>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>>>>>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new session
>>>>>>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
>>>>>>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
>>>>>>> (org.apache.zookeeper.server.persistence.FileTxnLog)
>>>>>>> . . .
>>>>>>>
>>>>>>>
>>>>>>> KAFKA
>>>>>>> Starting Kafka server is not going well yet although I see that
>>>>>>> interaction with Kerberos is successful (see relevant log below. the
>>>>>>> error is at the bottom)
>>>>>>> . . .
>>>>>>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
>>>>>>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
>>>>>>> (kafka.server.KafkaServer)
>>>>>>> [2016-02-23 13:26:11,519] INFO JAAS File name:
>>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
>>>>>>> (org.I0Itec.zkclient.ZkEventThread)
>>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
>>>>>>> GMT (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
>>>>>>> Corporation (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
>>>>>>> . . . . .
>>>>>>> [2016-02-23 13:26:11,531] INFO Client
>>>>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client
>>>>>>> environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client
>>>>>>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,532] INFO Initiating client connection,
>>>>>>> connectString=localhost:2181 sessionTimeout=6000
>>>>>>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>> Debug is true storeKey true useTicketCache false useKeyTab true
>>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
>>>>>>> refreshKrb5Config is false principal is
>>>>>>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>>>>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>>>>>>> is false
>>>>>>> principal is
>>>>>>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>>>>>> Will use keytab
>>>>>>> Commit Succeeded
>>>>>>>
>>>>>>> [2016-02-23 13:26:11,734] INFO successfully logged in.
>>>>>>> (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
>>>>>>> (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
>>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>>>>>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
>>>>>>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
>>>>>>> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
>>>>>>> [2016-02-23 13:26:11,748] INFO Socket connection established to
>>>>>>> localhost/127.0.0.1:2181, initiating session
>>>>>>> (org.apache.zookeeper.ClientCnxn)
>>>>>>> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23
>>>>>>> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23
>>>>>>> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
>>>>>>> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:26:11,761] INFO Session establishment complete on server
>>>>>>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
>>>>>>> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
>>>>>>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>> [2016-02-23 13:26:11,773] ERROR An error:
>>>>>>> (java.security.PrivilegedActionException:
>>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>>>>>> GSSException: No valid credentials provided (Mechanism level: Server not
>>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>>>>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>>>>>>> Client will go to AUTH_FAILED state.
>>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>>>>>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
>>>>>>> member failed: javax.security.sasl.SaslException: An error:
>>>>>>> (java.security.PrivilegedActionException:
>>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>>>>>> GSSException: No valid credentials provided (Mechanism level: Server not
>>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>>>>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>>>>>>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
>>>>>>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
>>>>>>> (org.I0Itec.zkclient.ZkEventThread)
>>>>>>> . . .
>>>>>>>
>>>>>>> Any pointers?
>>>>>>>
>>>>>>> Cheers
>>>>>>> Oleg
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>
>
Re: Kerberized Kafka setup issues
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
No joy. the same error
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
storeKey=true
keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
principal="kafka/ubuntu.oleg.com@OLEG.COM";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
serviceName=zookeeper
storeKey=true
keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
principal="kafka/ubuntu.oleg.com@OLEG.COM";
};
> On Feb 23, 2016, at 2:41 PM, Harsha <ma...@harsha.io> wrote:
>
> My bad it should be under Client section
>
> Client {
> com.sun.security.auth.module.Krb5LoginModule required
> debug=true
> useKeyTab=true
> storeKey=true
> serviceName=zookeeper
> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> principal="kafka/ubuntu.oleg.com@OLEG.COM";
> };
>
> -Harsha
>
> On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote:
>> can you try adding "serviceName=zookeeper" to KafkaServer section like
>> KafkaServer {
>> com.sun.security.auth.module.Krb5LoginModule required
>> debug=true
>> useKeyTab=true
>> storeKey=true
>> serviceName=zookeeper
>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
>> };
>>
>> On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
>>> More info
>>>
>>> I am starting both services as myself ‘oleg’. Validated that both key tab
>>> files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
>>> and Kafka as ‘kafka’
>>>
>>> Oleg
>>>
>>>> On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky <oz...@hortonworks.com> wrote:
>>>>
>>>> Harsha
>>>>
>>>> Thanks for following up. Here is is:
>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf
>>>> KafkaServer {
>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>> debug=true
>>>> useKeyTab=true
>>>> storeKey=true
>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
>>>> };
>>>> Client {
>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>> debug=true
>>>> useKeyTab=true
>>>> storeKey=true
>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>> principal="kafka/ubuntu.oleg.com@OLEG.COM";
>>>> };
>>>>
>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf
>>>> Server {
>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>> debug=true
>>>> useKeyTab=true
>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
>>>> storeKey=true
>>>> useTicketCache=false
>>>> principal="zookeeper/ubuntu.oleg.com@OLEG.COM";
>>>> };
>>>>
>>>> Cheers
>>>> Oleg
>>>>
>>>>> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io> wrote:
>>>>>
>>>>> Oleg,
>>>>> Can you post your jaas configs. Its important that serviceName
>>>>> must match the principal name with which zookeeper is running.
>>>>> Whats the principal name zookeeper service is running with.
>>>>> -Harsha
>>>>>
>>>>> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
>>>>>> Hey guys, first post here so bare with me
>>>>>>
>>>>>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
>>>>>> http://kafka.apache.org/documentation.html#security_sasl and i seem to be
>>>>>> very close, but not quite there yet.
>>>>>>
>>>>>> ZOOKEEPER
>>>>>> Starting Zookeeper seems to be OK (below is the relevant part of the log)
>>>>>> . . .
>>>>>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
>>>>>> (org.apache.zookeeper.server.ZooKeeperServer)
>>>>>> Debug is true storeKey true useTicketCache false useKeyTab true
>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
>>>>>> refreshKrb5Config is false principal is
>>>>>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>>>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>>>>>> is false
>>>>>> principal is
>>>>>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>>>>> Will use keytab
>>>>>> Commit Succeeded
>>>>>>
>>>>>> [2016-02-23 13:22:40,541] INFO successfully logged in.
>>>>>> (org.apache.zookeeper.Login)
>>>>>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
>>>>>> (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>>>>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
>>>>>> (org.apache.zookeeper.Login)
>>>>>> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23
>>>>>> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
>>>>>> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23
>>>>>> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
>>>>>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
>>>>>> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
>>>>>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
>>>>>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>>>>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new session
>>>>>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
>>>>>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
>>>>>> (org.apache.zookeeper.server.persistence.FileTxnLog)
>>>>>> . . .
>>>>>>
>>>>>>
>>>>>> KAFKA
>>>>>> Starting Kafka server is not going well yet although I see that
>>>>>> interaction with Kerberos is successful (see relevant log below. the
>>>>>> error is at the bottom)
>>>>>> . . .
>>>>>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
>>>>>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
>>>>>> (kafka.server.KafkaServer)
>>>>>> [2016-02-23 13:26:11,519] INFO JAAS File name:
>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
>>>>>> (org.I0Itec.zkclient.ZkEventThread)
>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
>>>>>> GMT (org.apache.zookeeper.ZooKeeper)
>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
>>>>>> Corporation (org.apache.zookeeper.ZooKeeper)
>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
>>>>>> . . . . .
>>>>>> [2016-02-23 13:26:11,531] INFO Client
>>>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>> [2016-02-23 13:26:11,531] INFO Client
>>>>>> environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>> [2016-02-23 13:26:11,531] INFO Client
>>>>>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>> [2016-02-23 13:26:11,532] INFO Initiating client connection,
>>>>>> connectString=localhost:2181 sessionTimeout=6000
>>>>>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>> Debug is true storeKey true useTicketCache false useKeyTab true
>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
>>>>>> refreshKrb5Config is false principal is
>>>>>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>>>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>>>>>> is false
>>>>>> principal is
>>>>>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>>>>> Will use keytab
>>>>>> Commit Succeeded
>>>>>>
>>>>>> [2016-02-23 13:26:11,734] INFO successfully logged in.
>>>>>> (org.apache.zookeeper.Login)
>>>>>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
>>>>>> (org.apache.zookeeper.Login)
>>>>>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>>>>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
>>>>>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
>>>>>> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
>>>>>> [2016-02-23 13:26:11,748] INFO Socket connection established to
>>>>>> localhost/127.0.0.1:2181, initiating session
>>>>>> (org.apache.zookeeper.ClientCnxn)
>>>>>> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23
>>>>>> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
>>>>>> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23
>>>>>> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
>>>>>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
>>>>>> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
>>>>>> [2016-02-23 13:26:11,761] INFO Session establishment complete on server
>>>>>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
>>>>>> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
>>>>>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>> [2016-02-23 13:26:11,773] ERROR An error:
>>>>>> (java.security.PrivilegedActionException:
>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>>>>> GSSException: No valid credentials provided (Mechanism level: Server not
>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>>>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>>>>>> Client will go to AUTH_FAILED state.
>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>>>>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
>>>>>> member failed: javax.security.sasl.SaslException: An error:
>>>>>> (java.security.PrivilegedActionException:
>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>>>>> GSSException: No valid credentials provided (Mechanism level: Server not
>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>>>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>>>>>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
>>>>>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
>>>>>> (org.I0Itec.zkclient.ZkEventThread)
>>>>>> . . .
>>>>>>
>>>>>> Any pointers?
>>>>>>
>>>>>> Cheers
>>>>>> Oleg
>>>>>>
>>>>>
>>>>
>>>>
>>>
>
Re: Kerberized Kafka setup issues
Posted by Harsha <ma...@harsha.io>.
My bad it should be under Client section
Client {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
storeKey=true
serviceName=zookeeper
keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
principal="kafka/ubuntu.oleg.com@OLEG.COM";
};
-Harsha
On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote:
> can you try adding "serviceName=zookeeper" to KafkaServer section like
> KafkaServer {
> com.sun.security.auth.module.Krb5LoginModule required
> debug=true
> useKeyTab=true
> storeKey=true
> serviceName=zookeeper
> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> principal="kafka/ubuntu.oleg.com@OLEG.COM";
> };
>
> On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
> > More info
> >
> > I am starting both services as myself ‘oleg’. Validated that both key tab
> > files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
> > and Kafka as ‘kafka’
> >
> > Oleg
> >
> > > On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky <oz...@hortonworks.com> wrote:
> > >
> > > Harsha
> > >
> > > Thanks for following up. Here is is:
> > > oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf
> > > KafkaServer {
> > > com.sun.security.auth.module.Krb5LoginModule required
> > > debug=true
> > > useKeyTab=true
> > > storeKey=true
> > > keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> > > principal="kafka/ubuntu.oleg.com@OLEG.COM";
> > > };
> > > Client {
> > > com.sun.security.auth.module.Krb5LoginModule required
> > > debug=true
> > > useKeyTab=true
> > > storeKey=true
> > > keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> > > principal="kafka/ubuntu.oleg.com@OLEG.COM";
> > > };
> > >
> > > oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf
> > > Server {
> > > com.sun.security.auth.module.Krb5LoginModule required
> > > debug=true
> > > useKeyTab=true
> > > keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
> > > storeKey=true
> > > useTicketCache=false
> > > principal="zookeeper/ubuntu.oleg.com@OLEG.COM";
> > > };
> > >
> > > Cheers
> > > Oleg
> > >
> > >> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io> wrote:
> > >>
> > >> Oleg,
> > >> Can you post your jaas configs. Its important that serviceName
> > >> must match the principal name with which zookeeper is running.
> > >> Whats the principal name zookeeper service is running with.
> > >> -Harsha
> > >>
> > >> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
> > >>> Hey guys, first post here so bare with me
> > >>>
> > >>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
> > >>> http://kafka.apache.org/documentation.html#security_sasl and i seem to be
> > >>> very close, but not quite there yet.
> > >>>
> > >>> ZOOKEEPER
> > >>> Starting Zookeeper seems to be OK (below is the relevant part of the log)
> > >>> . . .
> > >>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
> > >>> (org.apache.zookeeper.server.ZooKeeperServer)
> > >>> Debug is true storeKey true useTicketCache false useKeyTab true
> > >>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
> > >>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
> > >>> refreshKrb5Config is false principal is
> > >>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> > >>> tryFirstPass is false useFirstPass is false storePass is false clearPass
> > >>> is false
> > >>> principal is
> > >>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> > >>> Will use keytab
> > >>> Commit Succeeded
> > >>>
> > >>> [2016-02-23 13:22:40,541] INFO successfully logged in.
> > >>> (org.apache.zookeeper.Login)
> > >>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
> > >>> (org.apache.zookeeper.server.NIOServerCnxnFactory)
> > >>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
> > >>> (org.apache.zookeeper.Login)
> > >>> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23
> > >>> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
> > >>> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23
> > >>> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
> > >>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
> > >>> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
> > >>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
> > >>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> > >>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new session
> > >>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
> > >>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
> > >>> (org.apache.zookeeper.server.persistence.FileTxnLog)
> > >>> . . .
> > >>>
> > >>>
> > >>> KAFKA
> > >>> Starting Kafka server is not going well yet although I see that
> > >>> interaction with Kerberos is successful (see relevant log below. the
> > >>> error is at the bottom)
> > >>> . . .
> > >>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
> > >>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
> > >>> (kafka.server.KafkaServer)
> > >>> [2016-02-23 13:26:11,519] INFO JAAS File name:
> > >>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
> > >>> (org.I0Itec.zkclient.ZkClient)
> > >>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
> > >>> (org.I0Itec.zkclient.ZkEventThread)
> > >>> [2016-02-23 13:26:11,527] INFO Client
> > >>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> > >>> GMT (org.apache.zookeeper.ZooKeeper)
> > >>> [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
> > >>> (org.apache.zookeeper.ZooKeeper)
> > >>> [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
> > >>> (org.apache.zookeeper.ZooKeeper)
> > >>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
> > >>> Corporation (org.apache.zookeeper.ZooKeeper)
> > >>> [2016-02-23 13:26:11,527] INFO Client
> > >>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
> > >>> (org.apache.zookeeper.ZooKeeper)
> > >>> [2016-02-23 13:26:11,527] INFO Client
> > >>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
> > >>> . . . . .
> > >>> [2016-02-23 13:26:11,531] INFO Client
> > >>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > >>> (org.apache.zookeeper.ZooKeeper)
> > >>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
> > >>> (org.apache.zookeeper.ZooKeeper)
> > >>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
> > >>> (org.apache.zookeeper.ZooKeeper)
> > >>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
> > >>> (org.apache.zookeeper.ZooKeeper)
> > >>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
> > >>> (org.apache.zookeeper.ZooKeeper)
> > >>> [2016-02-23 13:26:11,531] INFO Client
> > >>> environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
> > >>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
> > >>> (org.apache.zookeeper.ZooKeeper)
> > >>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
> > >>> (org.apache.zookeeper.ZooKeeper)
> > >>> [2016-02-23 13:26:11,531] INFO Client
> > >>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
> > >>> (org.apache.zookeeper.ZooKeeper)
> > >>> [2016-02-23 13:26:11,532] INFO Initiating client connection,
> > >>> connectString=localhost:2181 sessionTimeout=6000
> > >>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
> > >>> (org.apache.zookeeper.ZooKeeper)
> > >>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
> > >>> (org.I0Itec.zkclient.ZkClient)
> > >>> Debug is true storeKey true useTicketCache false useKeyTab true
> > >>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
> > >>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
> > >>> refreshKrb5Config is false principal is
> > >>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> > >>> tryFirstPass is false useFirstPass is false storePass is false clearPass
> > >>> is false
> > >>> principal is
> > >>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> > >>> Will use keytab
> > >>> Commit Succeeded
> > >>>
> > >>> [2016-02-23 13:26:11,734] INFO successfully logged in.
> > >>> (org.apache.zookeeper.Login)
> > >>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
> > >>> (org.apache.zookeeper.Login)
> > >>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
> > >>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
> > >>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
> > >>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
> > >>> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
> > >>> [2016-02-23 13:26:11,748] INFO Socket connection established to
> > >>> localhost/127.0.0.1:2181, initiating session
> > >>> (org.apache.zookeeper.ClientCnxn)
> > >>> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23
> > >>> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
> > >>> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23
> > >>> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
> > >>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
> > >>> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
> > >>> [2016-02-23 13:26:11,761] INFO Session establishment complete on server
> > >>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
> > >>> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> > >>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
> > >>> (org.I0Itec.zkclient.ZkClient)
> > >>> [2016-02-23 13:26:11,773] ERROR An error:
> > >>> (java.security.PrivilegedActionException:
> > >>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> > >>> GSSException: No valid credentials provided (Mechanism level: Server not
> > >>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
> > >>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
> > >>> Client will go to AUTH_FAILED state.
> > >>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
> > >>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
> > >>> member failed: javax.security.sasl.SaslException: An error:
> > >>> (java.security.PrivilegedActionException:
> > >>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> > >>> GSSException: No valid credentials provided (Mechanism level: Server not
> > >>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
> > >>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
> > >>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
> > >>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
> > >>> (org.I0Itec.zkclient.ZkClient)
> > >>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
> > >>> (org.I0Itec.zkclient.ZkEventThread)
> > >>> . . .
> > >>>
> > >>> Any pointers?
> > >>>
> > >>> Cheers
> > >>> Oleg
> > >>>
> > >>
> > >
> > >
> >
Re: Kerberized Kafka setup issues
Posted by Harsha <ka...@harsha.io>.
can you try adding "serviceName=zookeeper" to KafkaServer section like
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
storeKey=true
serviceName=zookeeper
keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
principal="kafka/ubuntu.oleg.com@OLEG.COM";
};
On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
> More info
>
> I am starting both services as myself ‘oleg’. Validated that both key tab
> files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
> and Kafka as ‘kafka’
>
> Oleg
>
> > On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky <oz...@hortonworks.com> wrote:
> >
> > Harsha
> >
> > Thanks for following up. Here is is:
> > oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf
> > KafkaServer {
> > com.sun.security.auth.module.Krb5LoginModule required
> > debug=true
> > useKeyTab=true
> > storeKey=true
> > keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> > principal="kafka/ubuntu.oleg.com@OLEG.COM";
> > };
> > Client {
> > com.sun.security.auth.module.Krb5LoginModule required
> > debug=true
> > useKeyTab=true
> > storeKey=true
> > keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> > principal="kafka/ubuntu.oleg.com@OLEG.COM";
> > };
> >
> > oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf
> > Server {
> > com.sun.security.auth.module.Krb5LoginModule required
> > debug=true
> > useKeyTab=true
> > keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
> > storeKey=true
> > useTicketCache=false
> > principal="zookeeper/ubuntu.oleg.com@OLEG.COM";
> > };
> >
> > Cheers
> > Oleg
> >
> >> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io> wrote:
> >>
> >> Oleg,
> >> Can you post your jaas configs. Its important that serviceName
> >> must match the principal name with which zookeeper is running.
> >> Whats the principal name zookeeper service is running with.
> >> -Harsha
> >>
> >> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
> >>> Hey guys, first post here so bare with me
> >>>
> >>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
> >>> http://kafka.apache.org/documentation.html#security_sasl and i seem to be
> >>> very close, but not quite there yet.
> >>>
> >>> ZOOKEEPER
> >>> Starting Zookeeper seems to be OK (below is the relevant part of the log)
> >>> . . .
> >>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
> >>> (org.apache.zookeeper.server.ZooKeeperServer)
> >>> Debug is true storeKey true useTicketCache false useKeyTab true
> >>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
> >>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
> >>> refreshKrb5Config is false principal is
> >>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> >>> tryFirstPass is false useFirstPass is false storePass is false clearPass
> >>> is false
> >>> principal is
> >>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> >>> Will use keytab
> >>> Commit Succeeded
> >>>
> >>> [2016-02-23 13:22:40,541] INFO successfully logged in.
> >>> (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
> >>> (org.apache.zookeeper.server.NIOServerCnxnFactory)
> >>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
> >>> (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23
> >>> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23
> >>> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
> >>> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
> >>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> >>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new session
> >>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
> >>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
> >>> (org.apache.zookeeper.server.persistence.FileTxnLog)
> >>> . . .
> >>>
> >>>
> >>> KAFKA
> >>> Starting Kafka server is not going well yet although I see that
> >>> interaction with Kerberos is successful (see relevant log below. the
> >>> error is at the bottom)
> >>> . . .
> >>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
> >>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
> >>> (kafka.server.KafkaServer)
> >>> [2016-02-23 13:26:11,519] INFO JAAS File name:
> >>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
> >>> (org.I0Itec.zkclient.ZkClient)
> >>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
> >>> (org.I0Itec.zkclient.ZkEventThread)
> >>> [2016-02-23 13:26:11,527] INFO Client
> >>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> >>> GMT (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
> >>> Corporation (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,527] INFO Client
> >>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,527] INFO Client
> >>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
> >>> . . . . .
> >>> [2016-02-23 13:26:11,531] INFO Client
> >>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client
> >>> environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,531] INFO Client
> >>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,532] INFO Initiating client connection,
> >>> connectString=localhost:2181 sessionTimeout=6000
> >>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
> >>> (org.apache.zookeeper.ZooKeeper)
> >>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
> >>> (org.I0Itec.zkclient.ZkClient)
> >>> Debug is true storeKey true useTicketCache false useKeyTab true
> >>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
> >>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
> >>> refreshKrb5Config is false principal is
> >>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> >>> tryFirstPass is false useFirstPass is false storePass is false clearPass
> >>> is false
> >>> principal is
> >>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> >>> Will use keytab
> >>> Commit Succeeded
> >>>
> >>> [2016-02-23 13:26:11,734] INFO successfully logged in.
> >>> (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
> >>> (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
> >>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
> >>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
> >>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
> >>> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
> >>> [2016-02-23 13:26:11,748] INFO Socket connection established to
> >>> localhost/127.0.0.1:2181, initiating session
> >>> (org.apache.zookeeper.ClientCnxn)
> >>> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23
> >>> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23
> >>> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
> >>> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
> >>> [2016-02-23 13:26:11,761] INFO Session establishment complete on server
> >>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
> >>> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> >>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
> >>> (org.I0Itec.zkclient.ZkClient)
> >>> [2016-02-23 13:26:11,773] ERROR An error:
> >>> (java.security.PrivilegedActionException:
> >>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> >>> GSSException: No valid credentials provided (Mechanism level: Server not
> >>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
> >>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
> >>> Client will go to AUTH_FAILED state.
> >>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
> >>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
> >>> member failed: javax.security.sasl.SaslException: An error:
> >>> (java.security.PrivilegedActionException:
> >>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> >>> GSSException: No valid credentials provided (Mechanism level: Server not
> >>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
> >>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
> >>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
> >>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
> >>> (org.I0Itec.zkclient.ZkClient)
> >>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
> >>> (org.I0Itec.zkclient.ZkEventThread)
> >>> . . .
> >>>
> >>> Any pointers?
> >>>
> >>> Cheers
> >>> Oleg
> >>>
> >>
> >
> >
>
Re: Kerberized Kafka setup issues
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
More info
I am starting both services as myself ‘oleg’. Validated that both key tab files are readable. o I am assuming Zookeeper is started as ‘zookeeper’ and Kafka as ‘kafka’
Oleg
> On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky <oz...@hortonworks.com> wrote:
>
> Harsha
>
> Thanks for following up. Here is is:
> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf
> KafkaServer {
> com.sun.security.auth.module.Krb5LoginModule required
> debug=true
> useKeyTab=true
> storeKey=true
> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> principal="kafka/ubuntu.oleg.com@OLEG.COM";
> };
> Client {
> com.sun.security.auth.module.Krb5LoginModule required
> debug=true
> useKeyTab=true
> storeKey=true
> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> principal="kafka/ubuntu.oleg.com@OLEG.COM";
> };
>
> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf
> Server {
> com.sun.security.auth.module.Krb5LoginModule required
> debug=true
> useKeyTab=true
> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
> storeKey=true
> useTicketCache=false
> principal="zookeeper/ubuntu.oleg.com@OLEG.COM";
> };
>
> Cheers
> Oleg
>
>> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io> wrote:
>>
>> Oleg,
>> Can you post your jaas configs. Its important that serviceName
>> must match the principal name with which zookeeper is running.
>> Whats the principal name zookeeper service is running with.
>> -Harsha
>>
>> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
>>> Hey guys, first post here so bare with me
>>>
>>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
>>> http://kafka.apache.org/documentation.html#security_sasl and i seem to be
>>> very close, but not quite there yet.
>>>
>>> ZOOKEEPER
>>> Starting Zookeeper seems to be OK (below is the relevant part of the log)
>>> . . .
>>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
>>> (org.apache.zookeeper.server.ZooKeeperServer)
>>> Debug is true storeKey true useTicketCache false useKeyTab true
>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
>>> refreshKrb5Config is false principal is
>>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>>> is false
>>> principal is
>>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>> Will use keytab
>>> Commit Succeeded
>>>
>>> [2016-02-23 13:22:40,541] INFO successfully logged in.
>>> (org.apache.zookeeper.Login)
>>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
>>> (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
>>> (org.apache.zookeeper.Login)
>>> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23
>>> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23
>>> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
>>> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
>>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new session
>>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
>>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
>>> (org.apache.zookeeper.server.persistence.FileTxnLog)
>>> . . .
>>>
>>>
>>> KAFKA
>>> Starting Kafka server is not going well yet although I see that
>>> interaction with Kerberos is successful (see relevant log below. the
>>> error is at the bottom)
>>> . . .
>>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
>>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
>>> (kafka.server.KafkaServer)
>>> [2016-02-23 13:26:11,519] INFO JAAS File name:
>>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
>>> (org.I0Itec.zkclient.ZkClient)
>>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
>>> (org.I0Itec.zkclient.ZkEventThread)
>>> [2016-02-23 13:26:11,527] INFO Client
>>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
>>> GMT (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
>>> Corporation (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,527] INFO Client
>>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,527] INFO Client
>>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
>>> . . . . .
>>> [2016-02-23 13:26:11,531] INFO Client
>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client
>>> environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client
>>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,532] INFO Initiating client connection,
>>> connectString=localhost:2181 sessionTimeout=6000
>>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
>>> (org.I0Itec.zkclient.ZkClient)
>>> Debug is true storeKey true useTicketCache false useKeyTab true
>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
>>> refreshKrb5Config is false principal is
>>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>>> is false
>>> principal is
>>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>>> Will use keytab
>>> Commit Succeeded
>>>
>>> [2016-02-23 13:26:11,734] INFO successfully logged in.
>>> (org.apache.zookeeper.Login)
>>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
>>> (org.apache.zookeeper.Login)
>>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
>>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
>>> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
>>> [2016-02-23 13:26:11,748] INFO Socket connection established to
>>> localhost/127.0.0.1:2181, initiating session
>>> (org.apache.zookeeper.ClientCnxn)
>>> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23
>>> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23
>>> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
>>> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:26:11,761] INFO Session establishment complete on server
>>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
>>> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
>>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
>>> (org.I0Itec.zkclient.ZkClient)
>>> [2016-02-23 13:26:11,773] ERROR An error:
>>> (java.security.PrivilegedActionException:
>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>> GSSException: No valid credentials provided (Mechanism level: Server not
>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>>> Client will go to AUTH_FAILED state.
>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
>>> member failed: javax.security.sasl.SaslException: An error:
>>> (java.security.PrivilegedActionException:
>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>> GSSException: No valid credentials provided (Mechanism level: Server not
>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
>>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
>>> (org.I0Itec.zkclient.ZkClient)
>>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
>>> (org.I0Itec.zkclient.ZkEventThread)
>>> . . .
>>>
>>> Any pointers?
>>>
>>> Cheers
>>> Oleg
>>>
>>
>
>
Re: Kerberized Kafka setup issues
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
Harsha
Thanks for following up. Here is is:
oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
storeKey=true
keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
principal="kafka/ubuntu.oleg.com@OLEG.COM";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
storeKey=true
keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
principal="kafka/ubuntu.oleg.com@OLEG.COM";
};
oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf
Server {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
storeKey=true
useTicketCache=false
principal="zookeeper/ubuntu.oleg.com@OLEG.COM";
};
Cheers
Oleg
> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io> wrote:
>
> Oleg,
> Can you post your jaas configs. Its important that serviceName
> must match the principal name with which zookeeper is running.
> Whats the principal name zookeeper service is running with.
> -Harsha
>
> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
>> Hey guys, first post here so bare with me
>>
>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
>> http://kafka.apache.org/documentation.html#security_sasl and i seem to be
>> very close, but not quite there yet.
>>
>> ZOOKEEPER
>> Starting Zookeeper seems to be OK (below is the relevant part of the log)
>> . . .
>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
>> (org.apache.zookeeper.server.ZooKeeperServer)
>> Debug is true storeKey true useTicketCache false useKeyTab true
>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
>> refreshKrb5Config is false principal is
>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>> is false
>> principal is
>> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>> Will use keytab
>> Commit Succeeded
>>
>> [2016-02-23 13:22:40,541] INFO successfully logged in.
>> (org.apache.zookeeper.Login)
>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
>> (org.apache.zookeeper.server.NIOServerCnxnFactory)
>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
>> (org.apache.zookeeper.Login)
>> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23
>> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
>> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23
>> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
>> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new session
>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
>> (org.apache.zookeeper.server.persistence.FileTxnLog)
>> . . .
>>
>>
>> KAFKA
>> Starting Kafka server is not going well yet although I see that
>> interaction with Kerberos is successful (see relevant log below. the
>> error is at the bottom)
>> . . .
>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
>> (kafka.server.KafkaServer)
>> [2016-02-23 13:26:11,519] INFO JAAS File name:
>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
>> (org.I0Itec.zkclient.ZkClient)
>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
>> (org.I0Itec.zkclient.ZkEventThread)
>> [2016-02-23 13:26:11,527] INFO Client
>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
>> GMT (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
>> Corporation (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,527] INFO Client
>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,527] INFO Client
>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
>> . . . . .
>> [2016-02-23 13:26:11,531] INFO Client
>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client
>> environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,531] INFO Client
>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,532] INFO Initiating client connection,
>> connectString=localhost:2181 sessionTimeout=6000
>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
>> (org.apache.zookeeper.ZooKeeper)
>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
>> (org.I0Itec.zkclient.ZkClient)
>> Debug is true storeKey true useTicketCache false useKeyTab true
>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
>> refreshKrb5Config is false principal is
>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>> is false
>> principal is
>> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
>> Will use keytab
>> Commit Succeeded
>>
>> [2016-02-23 13:26:11,734] INFO successfully logged in.
>> (org.apache.zookeeper.Login)
>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
>> (org.apache.zookeeper.Login)
>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
>> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
>> [2016-02-23 13:26:11,748] INFO Socket connection established to
>> localhost/127.0.0.1:2181, initiating session
>> (org.apache.zookeeper.ClientCnxn)
>> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23
>> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
>> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23
>> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
>> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
>> [2016-02-23 13:26:11,761] INFO Session establishment complete on server
>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
>> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
>> (org.I0Itec.zkclient.ZkClient)
>> [2016-02-23 13:26:11,773] ERROR An error:
>> (java.security.PrivilegedActionException:
>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>> GSSException: No valid credentials provided (Mechanism level: Server not
>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>> Client will go to AUTH_FAILED state.
>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
>> member failed: javax.security.sasl.SaslException: An error:
>> (java.security.PrivilegedActionException:
>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>> GSSException: No valid credentials provided (Mechanism level: Server not
>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
>> (org.I0Itec.zkclient.ZkClient)
>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
>> (org.I0Itec.zkclient.ZkEventThread)
>> . . .
>>
>> Any pointers?
>>
>> Cheers
>> Oleg
>>
>
Re: Kerberized Kafka setup issues
Posted by Harsha <ka...@harsha.io>.
Oleg,
Can you post your jaas configs. Its important that serviceName
must match the principal name with which zookeeper is running.
Whats the principal name zookeeper service is running with.
-Harsha
On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
> Hey guys, first post here so bare with me
>
> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
> http://kafka.apache.org/documentation.html#security_sasl and i seem to be
> very close, but not quite there yet.
>
> ZOOKEEPER
> Starting Zookeeper seems to be OK (below is the relevant part of the log)
> . . .
> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
> (org.apache.zookeeper.server.ZooKeeperServer)
> Debug is true storeKey true useTicketCache false useKeyTab true
> doNotPrompt false ticketCache is null isInitiator true KeyTab is
> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
> refreshKrb5Config is false principal is
> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> tryFirstPass is false useFirstPass is false storePass is false clearPass
> is false
> principal is
> zookeeper/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> Will use keytab
> Commit Succeeded
>
> [2016-02-23 13:22:40,541] INFO successfully logged in.
> (org.apache.zookeeper.Login)
> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
> (org.apache.zookeeper.server.NIOServerCnxnFactory)
> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
> (org.apache.zookeeper.Login)
> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23
> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23
> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> [2016-02-23 13:23:09,025] INFO Client attempting to establish new session
> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
> (org.apache.zookeeper.server.persistence.FileTxnLog)
> . . .
>
>
> KAFKA
> Starting Kafka server is not going well yet although I see that
> interaction with Kerberos is successful (see relevant log below. the
> error is at the bottom)
> . . .
> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
> (kafka.server.KafkaServer)
> [2016-02-23 13:26:11,519] INFO JAAS File name:
> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
> (org.I0Itec.zkclient.ZkClient)
> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
> (org.I0Itec.zkclient.ZkEventThread)
> [2016-02-23 13:26:11,527] INFO Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> GMT (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
> Corporation (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,527] INFO Client
> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,527] INFO Client
> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
> . . . . .
> [2016-02-23 13:26:11,531] INFO Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client
> environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,531] INFO Client
> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,532] INFO Initiating client connection,
> connectString=localhost:2181 sessionTimeout=6000
> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
> (org.apache.zookeeper.ZooKeeper)
> [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
> (org.I0Itec.zkclient.ZkClient)
> Debug is true storeKey true useTicketCache false useKeyTab true
> doNotPrompt false ticketCache is null isInitiator true KeyTab is
> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
> refreshKrb5Config is false principal is
> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> tryFirstPass is false useFirstPass is false storePass is false clearPass
> is false
> principal is
> kafka/ubuntu.oleg.com@OLEG.COM<ma...@oleg.com>
> Will use keytab
> Commit Succeeded
>
> [2016-02-23 13:26:11,734] INFO successfully logged in.
> (org.apache.zookeeper.Login)
> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
> (org.apache.zookeeper.Login)
> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
> (org.apache.zookeeper.client.ZooKeeperSaslClient)
> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
> [2016-02-23 13:26:11,748] INFO Socket connection established to
> localhost/127.0.0.1:2181, initiating session
> (org.apache.zookeeper.ClientCnxn)
> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23
> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23
> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
> [2016-02-23 13:26:11,761] INFO Session establishment complete on server
> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
> (org.I0Itec.zkclient.ZkClient)
> [2016-02-23 13:26:11,773] ERROR An error:
> (java.security.PrivilegedActionException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
> Client will go to AUTH_FAILED state.
> (org.apache.zookeeper.client.ZooKeeperSaslClient)
> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
> member failed: javax.security.sasl.SaslException: An error:
> (java.security.PrivilegedActionException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
> (org.I0Itec.zkclient.ZkClient)
> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
> (org.I0Itec.zkclient.ZkEventThread)
> . . .
>
> Any pointers?
>
> Cheers
> Oleg
>