You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Alex (JIRA)" <ji...@apache.org> on 2016/05/06 15:14:12 UTC

[jira] [Created] (KAFKA-3668) Unable to authenticate Kafka broker to secured Zookeeper

Alex created KAFKA-3668:
---------------------------

             Summary: Unable to authenticate Kafka broker to secured Zookeeper
                 Key: KAFKA-3668
                 URL: https://issues.apache.org/jira/browse/KAFKA-3668
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 0.9.0.1, 0.9.0.0
         Environment: Red Hat Enterprise Linux Server release 7.0 (Maipo)
Java 1.8.0_66-b17
Kafka 0.9.0.0 and 0.9.0.1
            Reporter: Alex
             Fix For: 0.9.0.1, 0.9.0.0


Hello,

we are running into trouble when trying to connect Kafka broker to secured Zookeeper, Kerberos protected. 
Configuration is as simple as possible: 1 Zookeeper, 1 Kafka broker and Kerberos. All running on local machine.

Zookeeper successfully starts and receives TGT from Kerberos AS_REQ. Then Kafka broker obtains TGT from AS_REQ, but it is unable to get TGS from TGS_REQ because <unknown server> as krb5kdc.log shows:
krb5kdc.log
	...
	May 06 17:41:42 SBT-IPO-204.ca.sbrf.ru krb5kdc[1580](info): AS_REQ (4 etypes {18 17 16 23}) 10.116.93.88: ISSUE: authtime 1462545702, etypes {rep=18 tkt=18 ses=18}, zookeeper/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU for krbtgt/CA.SBRF.RU@CA.SBRF.RU
	May 06 17:44:24 SBT-IPO-204.ca.sbrf.ru krb5kdc[1580](info): AS_REQ (4 etypes {18 17 16 23}) 10.116.93.88: ISSUE: authtime 1462545864, etypes {rep=18 tkt=18 ses=18}, kafka/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU for krbtgt/CA.SBRF.RU@CA.SBRF.RU
	May 06 17:44:24 SBT-IPO-204.ca.sbrf.ru krb5kdc[1580](info): TGS_REQ (4 etypes {18 17 16 23}) 10.116.93.88: LOOKING_UP_SERVER: authtime 0,  kafka/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU for <unknown server>, Server not found in Kerberos database

What is the possible reason of this problem? 


KAFKA CONFIG:
	
	zookeeper.properties
		dataDir=/tmp/zookeeper
		clientPort=2181
		authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
		jaasLoginRenew=3600000
		
	server.properties
		broker.id=0
		log.dirs=/tmp/kafka-logs
		listeners=SASL_PLAINTEXT://10.116.93.88:9092
		security.inter.broker.protocol=SASL_PLAINTEXT
		zookeeper.connect=10.116.93.88:2181
		sasl.kerberos.service.name=kafka
		authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
		zookeeper.set.acl=true
		#allow.everyone.if.no.acl.found=true
		#sasl.enabled.mechanisms=GSSAPI
		#sasl.mechanism.inter.broker.protocol=GSSAPI


	JVM params:

		Kafka:
			-Djava.security.krb5.conf=/etc/krb5.conf 
			-Djava.security.auth.login.config=config/kafka-broker-jaas.conf

		Zookeeper:
			-Djava.security.krb5.conf=/etc/krb5.conf 
			-Djava.security.auth.login.config=config/zookeeper.conf
			

	JAAS files:

		kafka-broker-jaas.conf:
			KafkaServer {
				com.sun.security.auth.module.Krb5LoginModule required
				useKeyTab=true
				storeKey=true
				keyTab="/etc/security/keytabs/kafka.keytab"
				debug=true
				useTicketCache=false
				principal="kafka/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU";
			};
			Client {
				com.sun.security.auth.module.Krb5LoginModule required
				useKeyTab=true
				storeKey=true
				keyTab="/etc/security/keytabs/kafka.keytab"
				debug=true
				useTicketCache=false
				principal="kafka/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU";
			};

		zookeeper-jaas.conf
			Server {
				com.sun.security.auth.module.Krb5LoginModule required
				useKeyTab=true
				keyTab="/etc/security/keytabs/zookeeper.keytab"
				storeKey=true
				useTicketCache=false
				debug=true
				principal="zookeeper/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU";
			};


KERBEROS 5 CONFIG:

	krb5.conf
		[logging]
		 default = FILE:/var/log/krb5libs.log
		 kdc = FILE:/var/log/krb5kdc.log
		 admin_server = FILE:/var/log/kadmind.log

		[libdefaults]
		 dns_lookup_realm = false
		 ticket_lifetime = 24h
		 renew_lifetime = 7d
		 forwardable = true
		 rdns = false
		 default_realm = CA.SBRF.RU
		 default_ccache_name = KEYRING:persistent:%{uid}

		[realms]
		CA.SBRF.RU = {
		  kdc = SBT-IPO-204.ca.sbrf.ru
		  admin_server = SBT-IPO-204.ca.sbrf.ru
		 }

		[domain_realm]
		 .ca.sbrf.ru = CA.SBRF.RU
		 ca.sbrf.ru = CA.SBRF.RU

	kdc.conf
		[kdcdefaults]
		  kdc_ports = 88
		  kdc_tcp_ports = 88

		[realms]
		  CA.SBRF.RU = {
		  #master_key_type = aes256-cts
		  acl_file = /var/kerberos/krb5kdc/kadm5.acl
		  dict_file = /usr/share/dict/words
		  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
		  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
		 }		

	kadm.conf
		*/admin@CA.SBRF.RU      *




LOGS:

	Zookeeper: bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

		...
		[2016-05-06 17:41:42,750] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
		[2016-05-06 17:41:42,750] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
		Debug is  true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /etc/security/keytabs/zookeeper.keytab refreshKrb5Config is false principal is zookeeper/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU tryFirstPass is false useFirstPass is false storePass is false clearPass is false
		principal is zookeeper/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU
		Will use keytab
		Commit Succeeded 

		[2016-05-06 17:41:43,137] INFO successfully logged in. (org.apache.zookeeper.Login)
		[2016-05-06 17:41:43,143] INFO TGT refresh thread started. (org.apache.zookeeper.Login)
		[2016-05-06 17:41:43,150] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
		[2016-05-06 17:41:43,169] INFO TGT valid starting at:        Fri May 06 17:41:42 MSK 2016 (org.apache.zookeeper.Login)
		[2016-05-06 17:41:43,170] INFO TGT expires:                  Sat May 07 17:41:42 MSK 2016 (org.apache.zookeeper.Login)
		[2016-05-06 17:41:43,170] INFO TGT refresh sleeping until: Sat May 07 14:04:31 MSK 2016 (org.apache.zookeeper.Login)
		
		...Here Kafka starts...

		[2016-05-06 17:44:24,933] INFO Accepted socket connection from /10.116.93.88:58825 (org.apache.zookeeper.server.NIOServerCnxnFactory)
		[2016-05-06 17:44:24,952] ERROR Zookeeper Server failed to create a SaslServer to interact with a client during session initiation: javax.security.sasl.SaslException: Failure to initialize security context [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)] (org.apache.zookeeper.server.ZooKeeperSaslServer)
		javax.security.sasl.SaslException: Failure to initialize security context [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)]
			at com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(Unknown Source)
			at com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(Unknown Source)
			at javax.security.sasl.Sasl.createSaslServer(Unknown Source)
			at org.apache.zookeeper.server.ZooKeeperSaslServer$1.run(ZooKeeperSaslServer.java:118)
			at org.apache.zookeeper.server.ZooKeeperSaslServer$1.run(ZooKeeperSaslServer.java:114)
			at java.security.AccessController.doPrivileged(Native Method)
			at javax.security.auth.Subject.doAs(Unknown Source)
			at org.apache.zookeeper.server.ZooKeeperSaslServer.createSaslServer(ZooKeeperSaslServer.java:114)
			at org.apache.zookeeper.server.ZooKeeperSaslServer.<init>(ZooKeeperSaslServer.java:48)
			at org.apache.zookeeper.server.NIOServerCnxn.<init>(NIOServerCnxn.java:100)
			at org.apache.zookeeper.server.NIOServerCnxnFactory.createConnection(NIOServerCnxnFactory.java:161)
			at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:202)
			at java.lang.Thread.run(Unknown Source)
		Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)
			at sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Unknown Source)
			at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Unknown Source)
			at sun.security.jgss.GSSManagerImpl.getCredentialElement(Unknown Source)
			at sun.security.jgss.GSSCredentialImpl.add(Unknown Source)
			at sun.security.jgss.GSSCredentialImpl.<init>(Unknown Source)
			at sun.security.jgss.GSSManagerImpl.createCredential(Unknown Source)
			... 13 more
		[2016-05-06 17:44:24,961] INFO Client attempting to establish new session at /10.116.93.88:58825 (org.apache.zookeeper.server.ZooKeeperServer)
		[2016-05-06 17:44:24,963] INFO Creating new log file: log.53 (org.apache.zookeeper.server.persistence.FileTxnLog)
		[2016-05-06 17:44:24,972] INFO Established session 0x154868461350000 with negotiated timeout 6000 for client /10.116.93.88:58825 (org.apache.zookeeper.server.ZooKeeperServer)
		[2016-05-06 17:44:28,997] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
		EndOfStreamException: Unable to read additional data from client sessionid 0x154868461350000, likely client has closed socket
			at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
			at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
			at java.lang.Thread.run(Unknown Source)
		[2016-05-06 17:44:29,001] INFO Closed socket connection for client /10.116.93.88:58825 which had sessionid 0x154868461350000 (org.apache.zookeeper.server.NIOServerCnxn)
		[2016-05-06 17:44:33,001] INFO Expiring session 0x154868461350000, timeout of 6000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
		[2016-05-06 17:44:33,002] INFO Processed session termination for sessionid: 0x154868461350000 (org.apache.zookeeper.server.PrepRequestProcessor)


	Kafka: bin/kafka-server-start.sh -daemon config/server.properties

		...
		[2016-05-06 17:44:24,353] INFO starting (kafka.server.KafkaServer)
		[2016-05-06 17:44:24,360] INFO Connecting to zookeeper on 10.116.93.88:2181 (kafka.server.KafkaServer)
		[2016-05-06 17:44:30,428] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
		org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
			at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
			at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
			at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
			at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
			at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
			at kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
			at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
			at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
			at kafka.Kafka$.main(Kafka.scala:67)
			at kafka.Kafka.main(Kafka.scala)
		[2016-05-06 17:44:30,431] INFO shutting down (kafka.server.KafkaServer)
		[2016-05-06 17:44:30,438] INFO shut down completed (kafka.server.KafkaServer)
		[2016-05-06 17:44:30,439] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
		org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
			at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
			at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
			at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
			at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
			at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
			at kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
			at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
			at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
			at kafka.Kafka$.main(Kafka.scala:67)
			at kafka.Kafka.main(Kafka.scala)
		[2016-05-06 17:44:30,442] INFO shutting down (kafka.server.KafkaServer)

		
		








--
This message was sent by Atlassian JIRA
(v6.3.4#6332)