You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Alex (JIRA)" <ji...@apache.org> on 2016/05/12 07:18:12 UTC
[jira] [Closed] (KAFKA-3668) Unable to authenticate Kafka broker to
secured Zookeeper
[ https://issues.apache.org/jira/browse/KAFKA-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex closed KAFKA-3668.
-----------------------
> Unable to authenticate Kafka broker to secured Zookeeper
> --------------------------------------------------------
>
> Key: KAFKA-3668
> URL: https://issues.apache.org/jira/browse/KAFKA-3668
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.9.0.0, 0.9.0.1
> Environment: Red Hat Enterprise Linux Server release 7.0 (Maipo)
> Java 1.8.0_66-b17
> Kafka 0.9.0.0 and 0.9.0.1
> Reporter: Alex
> Fix For: 0.9.0.0, 0.9.0.1
>
>
> Hello,
> we are running into trouble when trying to connect Kafka broker to secured Zookeeper, Kerberos protected.
> Configuration is as simple as possible: 1 Zookeeper, 1 Kafka broker and Kerberos. All running on local machine.
> Zookeeper successfully starts and receives TGT from Kerberos AS_REQ. Then Kafka broker obtains TGT from AS_REQ, but it is unable to get TGS from TGS_REQ because <unknown server> as krb5kdc.log shows:
> krb5kdc.log
> ...
> May 06 17:41:42 SBT-IPO-204.ca.sbrf.ru krb5kdc[1580](info): AS_REQ (4 etypes {18 17 16 23}) 10.116.93.88: ISSUE: authtime 1462545702, etypes {rep=18 tkt=18 ses=18}, zookeeper/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU for krbtgt/CA.SBRF.RU@CA.SBRF.RU
> May 06 17:44:24 SBT-IPO-204.ca.sbrf.ru krb5kdc[1580](info): AS_REQ (4 etypes {18 17 16 23}) 10.116.93.88: ISSUE: authtime 1462545864, etypes {rep=18 tkt=18 ses=18}, kafka/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU for krbtgt/CA.SBRF.RU@CA.SBRF.RU
> May 06 17:44:24 SBT-IPO-204.ca.sbrf.ru krb5kdc[1580](info): TGS_REQ (4 etypes {18 17 16 23}) 10.116.93.88: LOOKING_UP_SERVER: authtime 0, kafka/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU for <unknown server>, Server not found in Kerberos database
> What is the possible reason of this problem?
> KAFKA CONFIG:
>
> zookeeper.properties
> dataDir=/tmp/zookeeper
> clientPort=2181
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> jaasLoginRenew=3600000
>
> server.properties
> broker.id=0
> log.dirs=/tmp/kafka-logs
> listeners=SASL_PLAINTEXT://10.116.93.88:9092
> security.inter.broker.protocol=SASL_PLAINTEXT
> zookeeper.connect=10.116.93.88:2181
> sasl.kerberos.service.name=kafka
> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> zookeeper.set.acl=true
> #allow.everyone.if.no.acl.found=true
> #sasl.enabled.mechanisms=GSSAPI
> #sasl.mechanism.inter.broker.protocol=GSSAPI
> JVM params:
> Kafka:
> -Djava.security.krb5.conf=/etc/krb5.conf
> -Djava.security.auth.login.config=config/kafka-broker-jaas.conf
> Zookeeper:
> -Djava.security.krb5.conf=/etc/krb5.conf
> -Djava.security.auth.login.config=config/zookeeper.conf
>
> JAAS files:
> kafka-broker-jaas.conf:
> KafkaServer {
> com.sun.security.auth.module.Krb5LoginModule required
> useKeyTab=true
> storeKey=true
> keyTab="/etc/security/keytabs/kafka.keytab"
> debug=true
> useTicketCache=false
> principal="kafka/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU";
> };
> Client {
> com.sun.security.auth.module.Krb5LoginModule required
> useKeyTab=true
> storeKey=true
> keyTab="/etc/security/keytabs/kafka.keytab"
> debug=true
> useTicketCache=false
> principal="kafka/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU";
> };
> zookeeper-jaas.conf
> Server {
> com.sun.security.auth.module.Krb5LoginModule required
> useKeyTab=true
> keyTab="/etc/security/keytabs/zookeeper.keytab"
> storeKey=true
> useTicketCache=false
> debug=true
> principal="zookeeper/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU";
> };
> KERBEROS 5 CONFIG:
> krb5.conf
> [logging]
> default = FILE:/var/log/krb5libs.log
> kdc = FILE:/var/log/krb5kdc.log
> admin_server = FILE:/var/log/kadmind.log
> [libdefaults]
> dns_lookup_realm = false
> ticket_lifetime = 24h
> renew_lifetime = 7d
> forwardable = true
> rdns = false
> default_realm = CA.SBRF.RU
> default_ccache_name = KEYRING:persistent:%{uid}
> [realms]
> CA.SBRF.RU = {
> kdc = SBT-IPO-204.ca.sbrf.ru
> admin_server = SBT-IPO-204.ca.sbrf.ru
> }
> [domain_realm]
> .ca.sbrf.ru = CA.SBRF.RU
> ca.sbrf.ru = CA.SBRF.RU
> kdc.conf
> [kdcdefaults]
> kdc_ports = 88
> kdc_tcp_ports = 88
> [realms]
> CA.SBRF.RU = {
> #master_key_type = aes256-cts
> acl_file = /var/kerberos/krb5kdc/kadm5.acl
> dict_file = /usr/share/dict/words
> admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
> supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
> }
> kadm.conf
> */admin@CA.SBRF.RU *
> LOGS:
> Zookeeper: bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
> ...
> [2016-05-06 17:41:42,750] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-05-06 17:41:42,750] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
> Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /etc/security/keytabs/zookeeper.keytab refreshKrb5Config is false principal is zookeeper/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU tryFirstPass is false useFirstPass is false storePass is false clearPass is false
> principal is zookeeper/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU
> Will use keytab
> Commit Succeeded
> [2016-05-06 17:41:43,137] INFO successfully logged in. (org.apache.zookeeper.Login)
> [2016-05-06 17:41:43,143] INFO TGT refresh thread started. (org.apache.zookeeper.Login)
> [2016-05-06 17:41:43,150] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> [2016-05-06 17:41:43,169] INFO TGT valid starting at: Fri May 06 17:41:42 MSK 2016 (org.apache.zookeeper.Login)
> [2016-05-06 17:41:43,170] INFO TGT expires: Sat May 07 17:41:42 MSK 2016 (org.apache.zookeeper.Login)
> [2016-05-06 17:41:43,170] INFO TGT refresh sleeping until: Sat May 07 14:04:31 MSK 2016 (org.apache.zookeeper.Login)
>
> ...Here Kafka starts...
> [2016-05-06 17:44:24,933] INFO Accepted socket connection from /10.116.93.88:58825 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> [2016-05-06 17:44:24,952] ERROR Zookeeper Server failed to create a SaslServer to interact with a client during session initiation: javax.security.sasl.SaslException: Failure to initialize security context [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)] (org.apache.zookeeper.server.ZooKeeperSaslServer)
> javax.security.sasl.SaslException: Failure to initialize security context [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)]
> at com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(Unknown Source)
> at com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(Unknown Source)
> at javax.security.sasl.Sasl.createSaslServer(Unknown Source)
> at org.apache.zookeeper.server.ZooKeeperSaslServer$1.run(ZooKeeperSaslServer.java:118)
> at org.apache.zookeeper.server.ZooKeeperSaslServer$1.run(ZooKeeperSaslServer.java:114)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at org.apache.zookeeper.server.ZooKeeperSaslServer.createSaslServer(ZooKeeperSaslServer.java:114)
> at org.apache.zookeeper.server.ZooKeeperSaslServer.<init>(ZooKeeperSaslServer.java:48)
> at org.apache.zookeeper.server.NIOServerCnxn.<init>(NIOServerCnxn.java:100)
> at org.apache.zookeeper.server.NIOServerCnxnFactory.createConnection(NIOServerCnxnFactory.java:161)
> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:202)
> at java.lang.Thread.run(Unknown Source)
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)
> at sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Unknown Source)
> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Unknown Source)
> at sun.security.jgss.GSSManagerImpl.getCredentialElement(Unknown Source)
> at sun.security.jgss.GSSCredentialImpl.add(Unknown Source)
> at sun.security.jgss.GSSCredentialImpl.<init>(Unknown Source)
> at sun.security.jgss.GSSManagerImpl.createCredential(Unknown Source)
> ... 13 more
> [2016-05-06 17:44:24,961] INFO Client attempting to establish new session at /10.116.93.88:58825 (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-05-06 17:44:24,963] INFO Creating new log file: log.53 (org.apache.zookeeper.server.persistence.FileTxnLog)
> [2016-05-06 17:44:24,972] INFO Established session 0x154868461350000 with negotiated timeout 6000 for client /10.116.93.88:58825 (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-05-06 17:44:28,997] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
> EndOfStreamException: Unable to read additional data from client sessionid 0x154868461350000, likely client has closed socket
> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Unknown Source)
> [2016-05-06 17:44:29,001] INFO Closed socket connection for client /10.116.93.88:58825 which had sessionid 0x154868461350000 (org.apache.zookeeper.server.NIOServerCnxn)
> [2016-05-06 17:44:33,001] INFO Expiring session 0x154868461350000, timeout of 6000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-05-06 17:44:33,002] INFO Processed session termination for sessionid: 0x154868461350000 (org.apache.zookeeper.server.PrepRequestProcessor)
> Kafka: bin/kafka-server-start.sh -daemon config/server.properties
> ...
> [2016-05-06 17:44:24,353] INFO starting (kafka.server.KafkaServer)
> [2016-05-06 17:44:24,360] INFO Connecting to zookeeper on 10.116.93.88:2181 (kafka.server.KafkaServer)
> [2016-05-06 17:44:30,428] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
> at kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
> at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
> at kafka.Kafka$.main(Kafka.scala:67)
> at kafka.Kafka.main(Kafka.scala)
> [2016-05-06 17:44:30,431] INFO shutting down (kafka.server.KafkaServer)
> [2016-05-06 17:44:30,438] INFO shut down completed (kafka.server.KafkaServer)
> [2016-05-06 17:44:30,439] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
> at kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
> at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
> at kafka.Kafka$.main(Kafka.scala:67)
> at kafka.Kafka.main(Kafka.scala)
> [2016-05-06 17:44:30,442] INFO shutting down (kafka.server.KafkaServer)
>
>
> UPDATE:
> This is not actually a Kafka issue.
> The problem was at specifying the wrong FQDN (Fully Qualified Domain Name) at DNS.
> Kafka box has two DNS records:
> - with uppercase
> - with lowercase
> Kafka requests user with lowercase FQDN.
> Example:
> SBT-IPO-204.ca.sbrf.ru
> should be
> sbt-ipo-204.ca.sbrf.ru in JAAS file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)