SASL authenticate error when tring to connect zookeeper

[HE Ye <Ye...@alcatel-lucent.com>]

Hi expert,

Now I  am verifying connection between broker and zookeeper using SASL mechanisms and the zookeeper always claims: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)

From Kerberos log, I see zookeeper sent AS_REQ, broker sent AS_REQ and TGS_REQ. Kerberos server answered them successfully. 

Then through Wireshark, I see broker sent SMPP message to zookeeper, then failure happened. Confused by that. It is supposed for broker to send KRB_AP_REQ message to zookeeper then, right?

Already confused for a week! Very appreciate if any hint on this issue!

Here is the log:

zookeeper:
[2016-11-15 08:34:21,319] INFO Established session 0x1586868cc200000 with negotiated timeout 6000 for client /10.160.32.153:33610 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-11-15 08:34:21,320] WARN Client failed to SASL authenticate: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)] (org.apache.zookeeper.server.ZooKeeperServer)
[2016-11-15 08:34:21,320] WARN Closing client connection due to SASL authentication failure. (org.apache.zookeeper.server.ZooKeeperServer)
[2016-11-15 08:34:21,320] INFO Closed socket connection for client /10.160.32.153:33610 which had sessionid 0x1586868cc200000 (org.apache.zookeeper.server.NIOServerCnxn)


kafka broker:
[2016-11-15 08:34:21,316] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2016-11-15 08:34:21,321] INFO Unable to read additional data from server sessionid 0x1586868cc200000, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-11-15 08:34:21,421] INFO zookeeper state changed (Disconnected) (org.I0Itec.zkclient.ZkClient)

Kerberos:
Nov 15 08:34:08 YeKerberosServer-0-0-1 krb5kdc[10896](info): AS_REQ (1 etypes {17}) 10.160.32.153: ISSUE: authtime 1479220448, etypes {rep=17 tkt=17 ses=17}, zookeeper/kafka.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Nov 15 08:34:19 YeKerberosServer-0-0-1 krb5kdc[10896](info): AS_REQ (1 etypes {17}) 10.160.32.153: NEEDED_PREAUTH: kafka/kafka.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required
Nov 15 08:34:19 YeKerberosServer-0-0-1 krb5kdc[10896](info): AS_REQ (1 etypes {17}) 10.160.32.153: ISSUE: authtime 1479220459, etypes {rep=17 tkt=17 ses=17}, kafka/kafka.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Nov 15 08:34:19 YeKerberosServer-0-0-1 krb5kdc[10896](info): TGS_REQ (1 etypes {17}) 10.160.32.153: ISSUE: authtime 1479220459, etypes {rep=17 tkt=17 ses=17}, kafka/kafka.example.com@EXAMPLE.COM for zookeeper/10.160.32.153@EXAMPLE.COM

kafka_server_ jaas.conf
    KafkaServer {
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        useTicketCache=false
        serviceName="kafka"
        keyTab="/u/ainet/kafka_2.11-0.10.0.0/config/kafka.keytab"
        principal="kafka/kafka.example.com@EXAMPLE.COM";
    };

    // Zookeeper client authentication
    Client {
       com.sun.security.auth.module.Krb5LoginModule required
       //useTicketCache=true;
       useKeyTab=true
       storeKey=true
       useTicketCache=false
       serviceName="zookeeper"
       doNotPrompt=true
       refreshKrb5Config=true
       isInitiator=true
       keyTab="/u/ainet/kafka_2.11-0.10.0.0/config/kafka.keytab"
       principal="kafka/kafka.example.com@EXAMPLE.COM";
    };

zookeeper_server_jaas.conf
    Server {
       com.sun.security.auth.module.Krb5LoginModule required
       useKeyTab=true
       storeKey=true
       useTicketCache=false
       keyTab="/u/ainet/kafka_2.11-0.10.0.0/config/zookeeper.keytab"
       principal="zookeeper/kafka.example.com@EXAMPLE.COM";
    };

krb5.conf:
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 dns_lookup_realm = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 rdns = false
 default_realm = EXAMPLE.COM
 default_ccache_name = KEYRING:persistent:%{uid}
default_tkt_enctypes = aes128-cts
  default_tgs_enctypes = aes128-cts
 permitted_enctypes = aes128-cts
 debug=true

[realms]
 EXAMPLE.COM = {
  kdc = kbserver.example.com
  admin_server = kbserver.example.com
 }

[domain_realm]
 .example.com = EXAMPLE.COM
 example.com = EXAMPLE.COM

[appdefaults]
validate=false


Thanks,
Ye