You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Ivanov, Evgeny" <Ev...@vtbcapital.ru> on 2022/06/06 09:27:22 UTC
SASL authentication between Kafka and Zookeeper
Hi all,
could you please advise correct configuration settings (jaas and config files) to enable SASL authentication between Kafka and Zookeeper ?
Here is the error I get:
[2022-06-06 10:54:30,348] ERROR SASL authentication failed using login context 'Client'. (org.apache.zookeeper.client.ZooKeeperSaslClient)
javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member: the quorum member's saslToken is null.
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:310)
at org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:270)
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:936)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:98)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)
[2022-06-06 10:54:30,352] DEBUG [ZooKeeperClient Kafka server] Received event: WatchedEvent state:AuthFailed type:None path:null (kafka.zookeeper.ZooKeeperClient)
[2022-06-06 10:54:30,353] ERROR [ZooKeeperClient Kafka server] Auth failed, initialized=true connectionState=AUTH_FAILED (kafka.zookeeper.ZooKeeperClient)
[2022-06-06 10:54:30,365] DEBUG Scheduling task auth-failed with initial delay 1000 ms and period -1 ms. (kafka.utils.KafkaScheduler)
[2022-06-06 10:54:30,372] INFO EventThread shut down for session: 0x100ec2b518b0000 (org.apache.zookeeper.ClientCnxn)
[2022-06-06 10:54:30,395] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /consumers
at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:566)
at kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1729)
at kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1627)
at kafka.zk.KafkaZkClient.$anonfun$createTopLevelPaths$1(KafkaZkClient.scala:1619)
at kafka.zk.KafkaZkClient.$anonfun$createTopLevelPaths$1$adapted(KafkaZkClient.scala:1619)
at scala.collection.immutable.List.foreach(List.scala:333)
at kafka.zk.KafkaZkClient.createTopLevelPaths(KafkaZkClient.scala:1619)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:492)
at kafka.server.KafkaServer.startup(KafkaServer.scala:201)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
[2022-06-06 10:54:30,396] INFO shutting down (kafka.server.KafkaServer)
[2022-06-06 10:54:30,399] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
And here is our configuration:
server.properties
broker.id=432
listeners = PLAINTEXT://smsk02ap432u:9094,SASL_SSL://smsk02ap432u:9095
sasl.enabled.mechanisms=PLAIN
ssl.keystore.location=/app/kafka/certs/server/server.keystore.jks
ssl.keystore.password=Moscow123
authorizer.class.name=kafka.security.authorizer.AclAuthorizer
zookeeper.connect=smsk02ap432u:2181
zookeeper.connection.timeout.ms=18000
kafka_server_jaas.conf
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret"
user_admin="admin-secret"
user_alice="alice-secret"
user_john="John1"
user_tom="Tom1";
};
Client {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret"
user_admin="admin-secret";
};
zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/app/zookeeper/data
clientPort=2181
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000
zookeeper_jaas.conf
Server {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret"
user_admin="admin-secret";
};
Best regards,
Evgeny
Business Application Support
VTB Capital
Telephone.: +7 (495) 960 9999 (ext.264423)
Mobile: +7 (916) 091-8939
________________________________
This email message (and any attachments) is confidential and may be privileged or otherwise protected from disclosure by applicable law. If you are not the intended recipient or have received this in error please notify the system manager, postmaster@vtbcapital.ru and remove this message and any attachments from your system. Any unauthorized dissemination, copying or other use of this message and/or any attachments is strictly prohibited and may constitute a breach of civil or criminal law.
JSC VTB Capital may monitor email traffic data and also the content of email.
RE: SASL authentication between Kafka and Zookeeper
Posted by "Ivanov, Evgeny" <Ev...@vtbcapital.ru>.
Hi,
here you are
/app/zookeeper/apache-zookeeper-3.8.0-bin
/app/kafka/kafka_2.13-3.1.0
Best regards,
Evgeny
Business Application Support
VTB Capital
Telephone.: +7 (495) 960 9999 (ext.264423)
Mobile: +7 (916) 091-8939
-----Original Message-----
From: Francois Papon <fr...@openobject.fr>
Sent: 6 июня 2022 г. 14:44
To: users@kafka.apache.org
Subject: Re: SASL authentication between Kafka and Zookeeper
Hi,
Wich version of Kafka/Zookeeper are you using?
On 06/06/2022 12:11, Ivanov, Evgeny wrote:
> Hi François,
>
>
>
> yes, I did (for both Client and Server), but no luck:
>
>
>
> [2022-06-06 13:06:11,348] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
>
> [2022-06-06 13:06:11,348] ERROR Exception while trying to create SASL client. (org.apache.zookeeper.client.ZooKeeperSaslClient)
>
> java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
>
> at org.apache.zookeeper.util.SecurityUtils.createSaslClient(SecurityUtils.java:72)
>
> at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:251)
>
> at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:139)
>
> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1161)
>
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1210)
>
> [2022-06-06 13:06:11,351] INFO Opening socket connection to server smsk02ap432u/10.78.197.18:2181. (org.apache.zookeeper.ClientCnxn)
>
> [2022-06-06 13:06:11,353] INFO SASL config status: Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
>
> [2022-06-06 13:06:11,361] INFO Socket connection established, initiating session, client: /10.78.197.18:35924, server: smsk02ap432u/10.78.197.18:2181 (org.apache.zookeeper.ClientCnxn)
>
> [2022-06-06 13:06:11,405] INFO Session establishment complete on server smsk02ap432u/10.78.197.18:2181, session id = 0x100eca49b4d0000, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
>
> [2022-06-06 13:06:11,409] ERROR SASL authentication with Zookeeper Quorum member failed. (org.apache.zookeeper.ClientCnxn)
>
> javax.security.sasl.SaslException: saslClient failed to initialize properly: it's null.
>
> at org.apache.zookeeper.client.ZooKeeperSaslClient.initialize(ZooKeeperSaslClient.java:399)
>
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1220)
>
> [2022-06-06 13:06:11,410] DEBUG [ZooKeeperClient Kafka server] Received event: WatchedEvent state:SyncConnected type:None path:null (kafka.zookeeper.ZooKeeperClient)
>
> [2022-06-06 13:06:11,411] DEBUG [ZooKeeperClient Kafka server] Received event: WatchedEvent state:null type:None path:null (kafka.zookeeper.ZooKeeperClient)
>
> [2022-06-06 13:06:11,411] INFO EventThread shut down for session: 0x100eca49b4d0000 (org.apache.zookeeper.ClientCnxn)
>
> [2022-06-06 13:06:11,412] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
>
> [2022-06-06 13:06:11,413] DEBUG Shutting down task scheduler. (kafka.utils.KafkaScheduler)
>
> [2022-06-06 13:06:11,415] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
>
> [2022-06-06 13:06:11,421] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
>
> kafka.zookeeper.ZooKeeperClientAuthFailedException: Auth failed either before or while waiting for connection
>
> at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:260)
>
> at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:108)
>
> at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1981)
>
> at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:491)
>
> at kafka.server.KafkaServer.startup(KafkaServer.scala:201)
>
> at kafka.Kafka$.main(Kafka.scala:109)
>
> at kafka.Kafka.main(Kafka.scala)
>
>
>
> Best regards,
>
> Evgeny
>
>
>
> Business Application Support
>
> VTB Capital
>
> Telephone.: +7 (495) 960 9999 (ext.264423)
>
> Mobile: +7 (916) 091-8939
>
>
>
>
>
> -----Original Message-----
> From: fpapon <fp...@apache.org>
> Sent: 6 июня 2022 г. 13:08
> To: users@kafka.apache.org
> Subject: Re: SASL authentication between Kafka and Zookeeper
>
>
>
> Hi,
>
>
>
> Did you tried to use?
>
>
>
> org.apache.zookeeper.server.auth.DigestLoginModule instead of org.apache.kafka.common.security.plain.PlainLoginModule
>
>
>
> Regards,
>
>
>
> François
>
>
>
> On 06/06/2022 11:27, Ivanov, Evgeny wrote:
>
>> org.apache.kafka.common.security.plain.PlainLoginModule required
>
>
> --
>
> --
>
> François
>
>
>
> ________________________________
>
> This email message (and any attachments) is confidential and may be privileged or otherwise protected from disclosure by applicable law. If you are not the intended recipient or have received this in error please notify the system manager, postmaster@vtbcapital.ru and remove this message and any attachments from your system. Any unauthorized dissemination, copying or other use of this message and/or any attachments is strictly prohibited and may constitute a breach of civil or criminal law.
> JSC VTB Capital may monitor email traffic data and also the content of email.
________________________________
This email message (and any attachments) is confidential and may be privileged or otherwise protected from disclosure by applicable law. If you are not the intended recipient or have received this in error please notify the system manager, postmaster@vtbcapital.ru and remove this message and any attachments from your system. Any unauthorized dissemination, copying or other use of this message and/or any attachments is strictly prohibited and may constitute a breach of civil or criminal law.
JSC VTB Capital may monitor email traffic data and also the content of email.
Re: SASL authentication between Kafka and Zookeeper
Posted by Francois Papon <fr...@openobject.fr>.
Hi,
Wich version of Kafka/Zookeeper are you using?
On 06/06/2022 12:11, Ivanov, Evgeny wrote:
> Hi François,
>
>
>
> yes, I did (for both Client and Server), but no luck:
>
>
>
> [2022-06-06 13:06:11,348] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
>
> [2022-06-06 13:06:11,348] ERROR Exception while trying to create SASL client. (org.apache.zookeeper.client.ZooKeeperSaslClient)
>
> java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
>
> at org.apache.zookeeper.util.SecurityUtils.createSaslClient(SecurityUtils.java:72)
>
> at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:251)
>
> at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:139)
>
> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1161)
>
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1210)
>
> [2022-06-06 13:06:11,351] INFO Opening socket connection to server smsk02ap432u/10.78.197.18:2181. (org.apache.zookeeper.ClientCnxn)
>
> [2022-06-06 13:06:11,353] INFO SASL config status: Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
>
> [2022-06-06 13:06:11,361] INFO Socket connection established, initiating session, client: /10.78.197.18:35924, server: smsk02ap432u/10.78.197.18:2181 (org.apache.zookeeper.ClientCnxn)
>
> [2022-06-06 13:06:11,405] INFO Session establishment complete on server smsk02ap432u/10.78.197.18:2181, session id = 0x100eca49b4d0000, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
>
> [2022-06-06 13:06:11,409] ERROR SASL authentication with Zookeeper Quorum member failed. (org.apache.zookeeper.ClientCnxn)
>
> javax.security.sasl.SaslException: saslClient failed to initialize properly: it's null.
>
> at org.apache.zookeeper.client.ZooKeeperSaslClient.initialize(ZooKeeperSaslClient.java:399)
>
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1220)
>
> [2022-06-06 13:06:11,410] DEBUG [ZooKeeperClient Kafka server] Received event: WatchedEvent state:SyncConnected type:None path:null (kafka.zookeeper.ZooKeeperClient)
>
> [2022-06-06 13:06:11,411] DEBUG [ZooKeeperClient Kafka server] Received event: WatchedEvent state:null type:None path:null (kafka.zookeeper.ZooKeeperClient)
>
> [2022-06-06 13:06:11,411] INFO EventThread shut down for session: 0x100eca49b4d0000 (org.apache.zookeeper.ClientCnxn)
>
> [2022-06-06 13:06:11,412] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
>
> [2022-06-06 13:06:11,413] DEBUG Shutting down task scheduler. (kafka.utils.KafkaScheduler)
>
> [2022-06-06 13:06:11,415] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
>
> [2022-06-06 13:06:11,421] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
>
> kafka.zookeeper.ZooKeeperClientAuthFailedException: Auth failed either before or while waiting for connection
>
> at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:260)
>
> at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:108)
>
> at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1981)
>
> at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:491)
>
> at kafka.server.KafkaServer.startup(KafkaServer.scala:201)
>
> at kafka.Kafka$.main(Kafka.scala:109)
>
> at kafka.Kafka.main(Kafka.scala)
>
>
>
> Best regards,
>
> Evgeny
>
>
>
> Business Application Support
>
> VTB Capital
>
> Telephone.: +7 (495) 960 9999 (ext.264423)
>
> Mobile: +7 (916) 091-8939
>
>
>
>
>
> -----Original Message-----
> From: fpapon <fp...@apache.org>
> Sent: 6 июня 2022 г. 13:08
> To: users@kafka.apache.org
> Subject: Re: SASL authentication between Kafka and Zookeeper
>
>
>
> Hi,
>
>
>
> Did you tried to use?
>
>
>
> org.apache.zookeeper.server.auth.DigestLoginModule instead of org.apache.kafka.common.security.plain.PlainLoginModule
>
>
>
> Regards,
>
>
>
> François
>
>
>
> On 06/06/2022 11:27, Ivanov, Evgeny wrote:
>
>> org.apache.kafka.common.security.plain.PlainLoginModule required
>
>
> --
>
> --
>
> François
>
>
>
> ________________________________
>
> This email message (and any attachments) is confidential and may be privileged or otherwise protected from disclosure by applicable law. If you are not the intended recipient or have received this in error please notify the system manager, postmaster@vtbcapital.ru and remove this message and any attachments from your system. Any unauthorized dissemination, copying or other use of this message and/or any attachments is strictly prohibited and may constitute a breach of civil or criminal law.
> JSC VTB Capital may monitor email traffic data and also the content of email.
RE: SASL authentication between Kafka and Zookeeper
Posted by "Ivanov, Evgeny" <Ev...@vtbcapital.ru>.
Hi François,
yes, I did (for both Client and Server), but no luck:
[2022-06-06 13:06:11,348] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2022-06-06 13:06:11,348] ERROR Exception while trying to create SASL client. (org.apache.zookeeper.client.ZooKeeperSaslClient)
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
at org.apache.zookeeper.util.SecurityUtils.createSaslClient(SecurityUtils.java:72)
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:251)
at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:139)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1161)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1210)
[2022-06-06 13:06:11,351] INFO Opening socket connection to server smsk02ap432u/10.78.197.18:2181. (org.apache.zookeeper.ClientCnxn)
[2022-06-06 13:06:11,353] INFO SASL config status: Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2022-06-06 13:06:11,361] INFO Socket connection established, initiating session, client: /10.78.197.18:35924, server: smsk02ap432u/10.78.197.18:2181 (org.apache.zookeeper.ClientCnxn)
[2022-06-06 13:06:11,405] INFO Session establishment complete on server smsk02ap432u/10.78.197.18:2181, session id = 0x100eca49b4d0000, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
[2022-06-06 13:06:11,409] ERROR SASL authentication with Zookeeper Quorum member failed. (org.apache.zookeeper.ClientCnxn)
javax.security.sasl.SaslException: saslClient failed to initialize properly: it's null.
at org.apache.zookeeper.client.ZooKeeperSaslClient.initialize(ZooKeeperSaslClient.java:399)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1220)
[2022-06-06 13:06:11,410] DEBUG [ZooKeeperClient Kafka server] Received event: WatchedEvent state:SyncConnected type:None path:null (kafka.zookeeper.ZooKeeperClient)
[2022-06-06 13:06:11,411] DEBUG [ZooKeeperClient Kafka server] Received event: WatchedEvent state:null type:None path:null (kafka.zookeeper.ZooKeeperClient)
[2022-06-06 13:06:11,411] INFO EventThread shut down for session: 0x100eca49b4d0000 (org.apache.zookeeper.ClientCnxn)
[2022-06-06 13:06:11,412] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2022-06-06 13:06:11,413] DEBUG Shutting down task scheduler. (kafka.utils.KafkaScheduler)
[2022-06-06 13:06:11,415] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2022-06-06 13:06:11,421] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientAuthFailedException: Auth failed either before or while waiting for connection
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:260)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:108)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1981)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:491)
at kafka.server.KafkaServer.startup(KafkaServer.scala:201)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
Best regards,
Evgeny
Business Application Support
VTB Capital
Telephone.: +7 (495) 960 9999 (ext.264423)
Mobile: +7 (916) 091-8939
-----Original Message-----
From: fpapon <fp...@apache.org>
Sent: 6 июня 2022 г. 13:08
To: users@kafka.apache.org
Subject: Re: SASL authentication between Kafka and Zookeeper
Hi,
Did you tried to use?
org.apache.zookeeper.server.auth.DigestLoginModule instead of org.apache.kafka.common.security.plain.PlainLoginModule
Regards,
François
On 06/06/2022 11:27, Ivanov, Evgeny wrote:
> org.apache.kafka.common.security.plain.PlainLoginModule required
--
--
François
________________________________
This email message (and any attachments) is confidential and may be privileged or otherwise protected from disclosure by applicable law. If you are not the intended recipient or have received this in error please notify the system manager, postmaster@vtbcapital.ru and remove this message and any attachments from your system. Any unauthorized dissemination, copying or other use of this message and/or any attachments is strictly prohibited and may constitute a breach of civil or criminal law.
JSC VTB Capital may monitor email traffic data and also the content of email.
Re: SASL authentication between Kafka and Zookeeper
Posted by fpapon <fp...@apache.org>.
Hi,
Did you tried to use?
org.apache.zookeeper.server.auth.DigestLoginModule instead of
org.apache.kafka.common.security.plain.PlainLoginModule
Regards,
François
On 06/06/2022 11:27, Ivanov, Evgeny wrote:
> org.apache.kafka.common.security.plain.PlainLoginModule required
--
--
François