You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by arjun s v <ar...@gmail.com> on 2023/11/07 11:08:19 UTC
Digest-MD5 authentication between kafka and zookeeper
Hi team,
I'm trying to configure *Digest-MD5* authentication between kafka and
zookeeper.
Also I need to set ACL with digest scheme and credentials.
I don't want to enable SASL.
I tried to follow this
<https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication>
doc
from zookeeper,
- If I configured a jaas file, I have to set zookeeper.sasl.client=true(if
not kafka throws error from JaasUtils) which enables sasl authentication.
- If I set zookeeper.set.acl=true, I'm forced to configure TLS.
I couldn't find any doc by kafka to enable Digest-MD5 authentication.
I cannot configure kerberos or TLS, just a Digest-MD5 is sufficient for my
usecase.
Please let me know if there are any docs to enable Digest-MD5 auth between
kafka and zookeeper.
Regards,
Arjun S V
Re: Digest-MD5 authentication between kafka and zookeeper
Posted by arjun s v <ar...@gmail.com>.
"Digest-MD5 is SASL authentication, so not sure what you mean here."
If I set zookeeper.sasl.client=true, zookeeper expects a "saslToken" and
throws the following error,
"SASL authentication failed using login context 'Client' with exception:
{}" "javax.security.sasl.SaslException: Error in authenticating with a
Zookeeper Quorum member: the quorum member's saslToken is null.
at
org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
at
org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
at
org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
"Hmm, that config shouldn't have anything to do with TLS. You can set ACL's
with or without TLS encryption. Were you getting an error?"
"Fatal error during KafkaServer startup. Prepare to shutdown"
"java.lang.SecurityException: zookeeper.set.acl is true, but ZooKeeper
client TLS configuration identifying at least
kafka.server.KafkaConfig$@7b22ec89.ZkSslClientEnableProp,
kafka.server.KafkaConfig$@7b22ec89.ZkClientCnxnSocketProp, and
kafka.server.KafkaConfig$@7b22ec89.ZkSslKeyStoreLocationProp was not
present and the verification of the JAAS login file failed
[java.security.auth.login.config=./../config/kafka_server_jaas.conf,
zookeeper.sasl.client=false, zookeeper.sasl.clientconfig=default:Client]
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:445)
at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
"This was the 2nd result in a google search:
https://docs.confluent.io/platform/current/security/zk-security.html"
FYKI, I've googled, asked chat gpt, surfed over many zookeeper and kafka
docs and blog,
I remember trying the doc you suggested here about 10 days back in the
initial days of this task!
About the doc you suggested,
I cannot configure SSL as I already mentioned, If I skip ssl config part
from your suggested doc and tried Digest-MD5, I come up "saslToken missing"
exception which I mentioned above!
I don't really understand what saslToken is and how to make it get
generated for Digest auth!
Please assist!
On Thu, Nov 9, 2023 at 7:15 PM Alex Craig <al...@gmail.com> wrote:
> " I couldn't find any doc by kafka to enable Digest-MD5 authentication."
> This was the 2nd result in a google search:
> https://docs.confluent.io/platform/current/security/zk-security.html
>
> " I don't want to enable SASL."
> Digest-MD5 is SASL authentication, so not sure what you mean here.
>
> " If I set zookeeper.set.acl=true, I'm forced to configure TLS."
> Hmm, that config shouldn't have anything to do with TLS. You can set ACL's
> with or without TLS encryption. Were you getting an error?
>
> On Wed, Nov 8, 2023 at 11:35 PM arjun s v <ar...@gmail.com> wrote:
>
> > Team,
> >
> > Please consider this as high priority, we need to enable authentication
> > ASAP. Please assist.
> > On Tue, Nov 7, 2023 at 4:38 PM arjun s v <ar...@gmail.com> wrote:
> >
> > > Hi team,
> > >
> > > I'm trying to configure *Digest-MD5* authentication between kafka and
> > > zookeeper.
> > > Also I need to set ACL with digest scheme and credentials.
> > > I don't want to enable SASL.
> > > I tried to follow this
> > > <
> >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
> >
> > doc
> > > from zookeeper,
> > >
> > > - If I configured a jaas file, I have to set
> > zookeeper.sasl.client=true(if
> > > not kafka throws error from JaasUtils) which enables sasl
> > authentication.
> > > - If I set zookeeper.set.acl=true, I'm forced to configure TLS.
> > >
> > > I couldn't find any doc by kafka to enable Digest-MD5 authentication.
> > > I cannot configure kerberos or TLS, just a Digest-MD5 is sufficient for
> > my
> > > usecase.
> > > Please let me know if there are any docs to enable Digest-MD5 auth
> > between
> > > kafka and zookeeper.
> > >
> > > Regards,
> > > Arjun S V
> > >
> >
>
Re: Digest-MD5 authentication between kafka and zookeeper
Posted by arjun s v <ar...@gmail.com>.
Made a small mistake, in zk_server_jaas.conf
provided,
> Server{
> org.apache.zookeeper.server.auth.DigestLoginModule required
> username="super"
> password="adminsecret";
};
Instead of,
> Server{
> org.apache.zookeeper.server.auth.DigestLoginModule required
> user_super="adminsecret";
};
Now after making this change, I can make the kafka nodes as world-readable
and modifiable only by brokers (as mentioned in kafka doc)
Thanks and regards
Arjun S V
On Thu, Nov 23, 2023 at 10:57 AM arjun s v <ar...@gmail.com> wrote:
> Hi Alex Brekken,
>
> Sorry for the delayed response, I tried your fix,
> At first I got
>
>> Fatal error during KafkaServer startup. Prepare to shutdown"
>> "org.apache.kafka.common.KafkaException: Exception while determining if
>> ZooKeeper is secure
>> [java.security.auth.login.config=./../config/kafka_server_jaas.conf,
>> zookeeper.sasl.client=false, zookeeper.sasl.clientconfig=default:Client]
>> at
>> org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:75)
>> at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:441)
>> at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
>> at kafka.Kafka$.main(Kafka.scala:109)
>> at kafka.Kafka.main(Kafka.scala)
>
>
> Then I set zookeeper.sasl.client=true
>
>> - 10.91.21.142 arjun-8481 - - - 23
>> org.apache.zookeeper.client.ZooKeeperSaslClient respondToServer SEVERE
>> "23-11-2023 10:49:21:770" - "SASL authentication failed using login context
>> 'Client' with exception: {}" "javax.security.sasl.SaslException: Error in
>> authenticating with a Zookeeper Quorum member: the quorum member's
>> saslToken is null.
>> at
>> org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
>> at
>> org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
>> at
>> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103)
>> at
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
>> - 10.91.21.142 arjun-8481 - - - 23
>> org.apache.zookeeper.ClientCnxn$SendThread run INFO "23-11-2023
>> 10:49:21:771" - "Unable to read additional data from server sessionid
>> 0x100147c3ccb0000, likely server has closed socket, closing socket
>> connection and attempting reconnect" - - - - - - 1700716761771 - - - - - -
>> - - logger_name=org.apache.zookeeper.ClientCnxn
>> - 10.91.21.142 arjun-8481 - - - 24 kafka.utils.Logging error SEVERE
>> "23-11-2023 10:49:21:771" - "[ZooKeeperClient Kafka server] Auth failed." -
>> - - - - - 1700716761771 - - - - - - - - logtype=application
>> logger_name=kafka.zookeeper.ZooKeeperClient
>> - 10.91.21.142 arjun-8481 - - - 24
>> org.apache.zookeeper.ClientCnxn$EventThread run INFO "23-11-2023
>> 10:49:21:773" - "EventThread shut down for session: 0x100147c3ccb0000" - -
>> - - - - 1700716761773 - - - - - - - - logtype=application
>> thread_name=main-EventThread logger_name=org.apache.zookeeper.ClientCnxn
>> - 10.91.21.142 arjun-8481 - - - 1 kafka.utils.Logging fatal SEVERE
>> "23-11-2023 10:49:21:887" - "Fatal error during KafkaServer startup.
>> Prepare to shutdown"
>> "org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode
>> = AuthFailed for /kafka/1/kafka/1
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
>> at
>> kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:583)
>> at
>> kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1729)
>> at
>> kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1627)
>> at
>> kafka.server.KafkaServer.$anonfun$initZkClient$2(KafkaServer.scala:451)
>> at
>> kafka.server.KafkaServer.$anonfun$initZkClient$2$adapted(KafkaServer.scala:448)
>> at scala.Option.foreach(Option.scala:437)
>> at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:448)
>> at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
>> at kafka.Kafka$.main(Kafka.scala:109)
>> at kafka.Kafka.main(Kafka.scala)
>
>
> Please advice.
>
> On Mon, Nov 13, 2023 at 4:11 AM Alex Brekken <br...@gmail.com> wrote:
>
>> I see a couple of things that look wrong. First, remove this line from
>> your
>> ZK config:
>>
>> zookeeper.authProvider.1=org.apache.zookeeper.server.auth.DigestAuthenticationProvider.
>> And replace it with this:
>>
>> authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider.
>>
>> Additionally, I think you need to add these lines to the ZK config if you
>> want ZK to ZK authentication:
>> quorum.auth.enableSasl=true
>> quorum.auth.learnerRequireSasl=true
>> quorum.auth.serverRequireSasl=true
>>
>> The rest looks OK.
>>
>> On Sat, Nov 11, 2023 at 9:12 PM arjun s v <ar...@gmail.com> wrote:
>>
>> > A small correction, I'm not trying to enable mTLS, just a simple
>> > authentication(Digest or Plain) is enough,
>> > Sharing the jaas files and config files,
>> > kafka_server_jaas.conf
>> >
>> > > admin.KafkaServer{
>> > >
>> > > org.apache.kafka.common.security.plain.PlainLoginModule required
>> > >
>> > > username="USERNAME"
>> > >
>> > > password="PASSWORD";
>> > >
>> > > };
>> > >
>> > > KafkaServer{
>> > >
>> > > org.apache.kafka.common.security.plain.PlainLoginModule required
>> > >
>> > > username="USERNAME"
>> > >
>> > > password="PASSWORD";
>> > >
>> > > };
>> > >
>> > > Client{
>> > >
>> > > org.apache.zookeeper.server.auth.DigestLoginModule required
>> > >
>> > > username="super"
>> > >
>> > > password="adminsecret";
>> > >
>> > > };
>> > >
>> > server properties
>> >
>> > > zookeeper.sasl.client=false
>> >
>> >
>> java.security.auth.login.config=$base_dir/../config/kafka_server_jaas.conf
>> >
>> > zookeeper.connection.timeout.ms=6000
>> >
>> > zookeeper.sync.time.ms=2000
>> >
>> > zookeeper.set.acl=true
>> >
>> >
>> > zk_server_jaas.conf
>> >
>> > > QuorumServer{
>> > > org.apache.zookeeper.server.auth.DigestLoginModule required
>> > > user_test="test";
>> > > };
>> > > QuorumLearner{
>> > > org.apache.zookeeper.server.auth.DigestLoginModule required
>> > > username="test"
>> > > password="test";
>> > > };
>> > > Server{
>> > > org.apache.zookeeper.server.auth.DigestLoginModule required
>> > > username="super"
>> > > password="adminsecret";
>> >
>> > };
>> >
>> >
>> > zoo.cfg
>> >
>> > >
>> > >
>> >
>> zookeeper.authProvider.1=org.apache.zookeeper.server.auth.DigestAuthenticationProvider
>> >
>> >
>> > env
>> >
>> > > SERVER_JVMFLAGS="$SERVER_JVMFLAGS
>> > > -Djava.security.auth.login.config=$ZOOCFGDIR/zk_server_jaas.conf"
>> > >
>> > On Fri, Nov 10, 2023 at 7:03 PM Alex Brekken <br...@gmail.com> wrote:
>> >
>> > > Ok, so you're trying to enable both SASL authentication (digest) and
>> TLS,
>> > > using mTLS for Zookeeper? I'm just trying to understand the bigger
>> > > picture. The error you're getting regarding the Sasl token sounds
>> like
>> > > either the jaas config on the Kafka broker side is wrong/missing, or
>> the
>> > > jaas config on the ZK side is wrong/missing. (you need both - in this
>> > case
>> > > the broker is the "client" and ZK is the "server"). Are you able to
>> share
>> > > the jaas config you're using for both Kafka and ZK? Without seeing
>> that
>> > > it's tough to know. Also, to make troubleshooting easier you might
>> want
>> > to
>> > > leave TLS out of it for now and get SASL working first. (or
>> vice-versa)
>> > >
>> > >
>> > >
>> > > On Thu, Nov 9, 2023 at 11:26 PM arjun s v <ar...@gmail.com>
>> wrote:
>> > >
>> > > > "Digest-MD5 is SASL authentication, so not sure what you mean here."
>> > > > If I set zookeeper.sasl.client=true, zookeeper expects a "saslToken"
>> > and
>> > > > throws the following error,
>> > > >
>> > > > "SASL authentication failed using login context 'Client' with
>> > exception:
>> > > > {}" "javax.security.sasl.SaslException: Error in authenticating
>> with a
>> > > > Zookeeper Quorum member: the quorum member's saslToken is null.
>> > > >
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
>> > > >
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
>> > > >
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103)
>> > > >
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365)
>> > > >
>> > > > at
>> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
>> > > >
>> > > >
>> > > > "Hmm, that config shouldn't have anything to do with TLS. You can
>> set
>> > > ACL's
>> > > >
>> > > > with or without TLS encryption. Were you getting an error?"
>> > > >
>> > > >
>> > > > "Fatal error during KafkaServer startup. Prepare to shutdown"
>> > > > "java.lang.SecurityException: zookeeper.set.acl is true, but
>> ZooKeeper
>> > > > client TLS configuration identifying at least
>> > > > kafka.server.KafkaConfig$@7b22ec89.ZkSslClientEnableProp,
>> > > > kafka.server.KafkaConfig$@7b22ec89.ZkClientCnxnSocketProp, and
>> > > > kafka.server.KafkaConfig$@7b22ec89.ZkSslKeyStoreLocationProp was not
>> > > > present and the verification of the JAAS login file failed
>> > > > [java.security.auth.login.config=./../config/kafka_server_jaas.conf,
>> > > > zookeeper.sasl.client=false,
>> > zookeeper.sasl.clientconfig=default:Client]
>> > > >
>> > > > at
>> kafka.server.KafkaServer.initZkClient(KafkaServer.scala:445)
>> > > >
>> > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
>> > > >
>> > > > at kafka.Kafka$.main(Kafka.scala:109)
>> > > >
>> > > > at kafka.Kafka.main(Kafka.scala)
>> > > >
>> > > >
>> > > > "This was the 2nd result in a google search:
>> > > >
>> https://docs.confluent.io/platform/current/security/zk-security.html"
>> > > >
>> > > > FYKI, I've googled, asked chat gpt, surfed over many zookeeper and
>> > kafka
>> > > > docs and blog,
>> > > > I remember trying the doc you suggested here about 10 days back in
>> the
>> > > > initial days of this task!
>> > > > About the doc you suggested,
>> > > > I cannot configure SSL as I already mentioned, If I skip ssl config
>> > part
>> > > > from your suggested doc and tried Digest-MD5, I come up "saslToken
>> > > missing"
>> > > > exception which I mentioned above!
>> > > > I don't really understand what saslToken is and how to make it get
>> > > > generated for Digest auth!
>> > > > Please assist!
>> > > >
>> > > > On Thu, Nov 9, 2023 at 7:15 PM Alex Craig <al...@gmail.com>
>> > wrote:
>> > > >
>> > > > > " I couldn't find any doc by kafka to enable Digest-MD5
>> > > authentication."
>> > > > > This was the 2nd result in a google search:
>> > > > >
>> https://docs.confluent.io/platform/current/security/zk-security.html
>> > > > >
>> > > > > " I don't want to enable SASL."
>> > > > > Digest-MD5 is SASL authentication, so not sure what you mean here.
>> > > > >
>> > > > > " If I set zookeeper.set.acl=true, I'm forced to configure TLS."
>> > > > > Hmm, that config shouldn't have anything to do with TLS. You can
>> set
>> > > > ACL's
>> > > > > with or without TLS encryption. Were you getting an error?
>> > > > >
>> > > > > On Wed, Nov 8, 2023 at 11:35 PM arjun s v <ar...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > Team,
>> > > > > >
>> > > > > > Please consider this as high priority, we need to enable
>> > > authentication
>> > > > > > ASAP. Please assist.
>> > > > > > On Tue, Nov 7, 2023 at 4:38 PM arjun s v <arjun.cse97@gmail.com
>> >
>> > > > wrote:
>> > > > > >
>> > > > > > > Hi team,
>> > > > > > >
>> > > > > > > I'm trying to configure *Digest-MD5* authentication between
>> kafka
>> > > and
>> > > > > > > zookeeper.
>> > > > > > > Also I need to set ACL with digest scheme and credentials.
>> > > > > > > I don't want to enable SASL.
>> > > > > > > I tried to follow this
>> > > > > > > <
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
>> > > > > >
>> > > > > > doc
>> > > > > > > from zookeeper,
>> > > > > > >
>> > > > > > > - If I configured a jaas file, I have to set
>> > > > > > zookeeper.sasl.client=true(if
>> > > > > > > not kafka throws error from JaasUtils) which enables sasl
>> > > > > > authentication.
>> > > > > > > - If I set zookeeper.set.acl=true, I'm forced to configure
>> > TLS.
>> > > > > > >
>> > > > > > > I couldn't find any doc by kafka to enable Digest-MD5
>> > > authentication.
>> > > > > > > I cannot configure kerberos or TLS, just a Digest-MD5 is
>> > sufficient
>> > > > for
>> > > > > > my
>> > > > > > > usecase.
>> > > > > > > Please let me know if there are any docs to enable Digest-MD5
>> > auth
>> > > > > > between
>> > > > > > > kafka and zookeeper.
>> > > > > > >
>> > > > > > > Regards,
>> > > > > > > Arjun S V
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: Digest-MD5 authentication between kafka and zookeeper
Posted by arjun s v <ar...@gmail.com>.
Hi Alex Brekken,
Sorry for the delayed response, I tried your fix,
At first I got
> Fatal error during KafkaServer startup. Prepare to shutdown"
> "org.apache.kafka.common.KafkaException: Exception while determining if
> ZooKeeper is secure
> [java.security.auth.login.config=./../config/kafka_server_jaas.conf,
> zookeeper.sasl.client=false, zookeeper.sasl.clientconfig=default:Client]
> at
> org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:75)
> at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:441)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
> at kafka.Kafka$.main(Kafka.scala:109)
> at kafka.Kafka.main(Kafka.scala)
Then I set zookeeper.sasl.client=true
> - 10.91.21.142 arjun-8481 - - - 23
> org.apache.zookeeper.client.ZooKeeperSaslClient respondToServer SEVERE
> "23-11-2023 10:49:21:770" - "SASL authentication failed using login context
> 'Client' with exception: {}" "javax.security.sasl.SaslException: Error in
> authenticating with a Zookeeper Quorum member: the quorum member's
> saslToken is null.
> at
> org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
> at
> org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
> - 10.91.21.142 arjun-8481 - - - 23
> org.apache.zookeeper.ClientCnxn$SendThread run INFO "23-11-2023
> 10:49:21:771" - "Unable to read additional data from server sessionid
> 0x100147c3ccb0000, likely server has closed socket, closing socket
> connection and attempting reconnect" - - - - - - 1700716761771 - - - - - -
> - - logger_name=org.apache.zookeeper.ClientCnxn
> - 10.91.21.142 arjun-8481 - - - 24 kafka.utils.Logging error SEVERE
> "23-11-2023 10:49:21:771" - "[ZooKeeperClient Kafka server] Auth failed." -
> - - - - - 1700716761771 - - - - - - - - logtype=application
> logger_name=kafka.zookeeper.ZooKeeperClient
> - 10.91.21.142 arjun-8481 - - - 24
> org.apache.zookeeper.ClientCnxn$EventThread run INFO "23-11-2023
> 10:49:21:773" - "EventThread shut down for session: 0x100147c3ccb0000" - -
> - - - - 1700716761773 - - - - - - - - logtype=application
> thread_name=main-EventThread logger_name=org.apache.zookeeper.ClientCnxn
> - 10.91.21.142 arjun-8481 - - - 1 kafka.utils.Logging fatal SEVERE
> "23-11-2023 10:49:21:887" - "Fatal error during KafkaServer startup.
> Prepare to shutdown"
> "org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode
> = AuthFailed for /kafka/1/kafka/1
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
> at
> kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:583)
> at
> kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1729)
> at
> kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1627)
> at
> kafka.server.KafkaServer.$anonfun$initZkClient$2(KafkaServer.scala:451)
> at
> kafka.server.KafkaServer.$anonfun$initZkClient$2$adapted(KafkaServer.scala:448)
> at scala.Option.foreach(Option.scala:437)
> at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:448)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
> at kafka.Kafka$.main(Kafka.scala:109)
> at kafka.Kafka.main(Kafka.scala)
Please advice.
On Mon, Nov 13, 2023 at 4:11 AM Alex Brekken <br...@gmail.com> wrote:
> I see a couple of things that look wrong. First, remove this line from your
> ZK config:
>
> zookeeper.authProvider.1=org.apache.zookeeper.server.auth.DigestAuthenticationProvider.
> And replace it with this:
>
> authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider.
>
> Additionally, I think you need to add these lines to the ZK config if you
> want ZK to ZK authentication:
> quorum.auth.enableSasl=true
> quorum.auth.learnerRequireSasl=true
> quorum.auth.serverRequireSasl=true
>
> The rest looks OK.
>
> On Sat, Nov 11, 2023 at 9:12 PM arjun s v <ar...@gmail.com> wrote:
>
> > A small correction, I'm not trying to enable mTLS, just a simple
> > authentication(Digest or Plain) is enough,
> > Sharing the jaas files and config files,
> > kafka_server_jaas.conf
> >
> > > admin.KafkaServer{
> > >
> > > org.apache.kafka.common.security.plain.PlainLoginModule required
> > >
> > > username="USERNAME"
> > >
> > > password="PASSWORD";
> > >
> > > };
> > >
> > > KafkaServer{
> > >
> > > org.apache.kafka.common.security.plain.PlainLoginModule required
> > >
> > > username="USERNAME"
> > >
> > > password="PASSWORD";
> > >
> > > };
> > >
> > > Client{
> > >
> > > org.apache.zookeeper.server.auth.DigestLoginModule required
> > >
> > > username="super"
> > >
> > > password="adminsecret";
> > >
> > > };
> > >
> > server properties
> >
> > > zookeeper.sasl.client=false
> >
> >
> java.security.auth.login.config=$base_dir/../config/kafka_server_jaas.conf
> >
> > zookeeper.connection.timeout.ms=6000
> >
> > zookeeper.sync.time.ms=2000
> >
> > zookeeper.set.acl=true
> >
> >
> > zk_server_jaas.conf
> >
> > > QuorumServer{
> > > org.apache.zookeeper.server.auth.DigestLoginModule required
> > > user_test="test";
> > > };
> > > QuorumLearner{
> > > org.apache.zookeeper.server.auth.DigestLoginModule required
> > > username="test"
> > > password="test";
> > > };
> > > Server{
> > > org.apache.zookeeper.server.auth.DigestLoginModule required
> > > username="super"
> > > password="adminsecret";
> >
> > };
> >
> >
> > zoo.cfg
> >
> > >
> > >
> >
> zookeeper.authProvider.1=org.apache.zookeeper.server.auth.DigestAuthenticationProvider
> >
> >
> > env
> >
> > > SERVER_JVMFLAGS="$SERVER_JVMFLAGS
> > > -Djava.security.auth.login.config=$ZOOCFGDIR/zk_server_jaas.conf"
> > >
> > On Fri, Nov 10, 2023 at 7:03 PM Alex Brekken <br...@gmail.com> wrote:
> >
> > > Ok, so you're trying to enable both SASL authentication (digest) and
> TLS,
> > > using mTLS for Zookeeper? I'm just trying to understand the bigger
> > > picture. The error you're getting regarding the Sasl token sounds like
> > > either the jaas config on the Kafka broker side is wrong/missing, or
> the
> > > jaas config on the ZK side is wrong/missing. (you need both - in this
> > case
> > > the broker is the "client" and ZK is the "server"). Are you able to
> share
> > > the jaas config you're using for both Kafka and ZK? Without seeing
> that
> > > it's tough to know. Also, to make troubleshooting easier you might
> want
> > to
> > > leave TLS out of it for now and get SASL working first. (or
> vice-versa)
> > >
> > >
> > >
> > > On Thu, Nov 9, 2023 at 11:26 PM arjun s v <ar...@gmail.com>
> wrote:
> > >
> > > > "Digest-MD5 is SASL authentication, so not sure what you mean here."
> > > > If I set zookeeper.sasl.client=true, zookeeper expects a "saslToken"
> > and
> > > > throws the following error,
> > > >
> > > > "SASL authentication failed using login context 'Client' with
> > exception:
> > > > {}" "javax.security.sasl.SaslException: Error in authenticating with
> a
> > > > Zookeeper Quorum member: the quorum member's saslToken is null.
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365)
> > > >
> > > > at
> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
> > > >
> > > >
> > > > "Hmm, that config shouldn't have anything to do with TLS. You can set
> > > ACL's
> > > >
> > > > with or without TLS encryption. Were you getting an error?"
> > > >
> > > >
> > > > "Fatal error during KafkaServer startup. Prepare to shutdown"
> > > > "java.lang.SecurityException: zookeeper.set.acl is true, but
> ZooKeeper
> > > > client TLS configuration identifying at least
> > > > kafka.server.KafkaConfig$@7b22ec89.ZkSslClientEnableProp,
> > > > kafka.server.KafkaConfig$@7b22ec89.ZkClientCnxnSocketProp, and
> > > > kafka.server.KafkaConfig$@7b22ec89.ZkSslKeyStoreLocationProp was not
> > > > present and the verification of the JAAS login file failed
> > > > [java.security.auth.login.config=./../config/kafka_server_jaas.conf,
> > > > zookeeper.sasl.client=false,
> > zookeeper.sasl.clientconfig=default:Client]
> > > >
> > > > at
> kafka.server.KafkaServer.initZkClient(KafkaServer.scala:445)
> > > >
> > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
> > > >
> > > > at kafka.Kafka$.main(Kafka.scala:109)
> > > >
> > > > at kafka.Kafka.main(Kafka.scala)
> > > >
> > > >
> > > > "This was the 2nd result in a google search:
> > > > https://docs.confluent.io/platform/current/security/zk-security.html
> "
> > > >
> > > > FYKI, I've googled, asked chat gpt, surfed over many zookeeper and
> > kafka
> > > > docs and blog,
> > > > I remember trying the doc you suggested here about 10 days back in
> the
> > > > initial days of this task!
> > > > About the doc you suggested,
> > > > I cannot configure SSL as I already mentioned, If I skip ssl config
> > part
> > > > from your suggested doc and tried Digest-MD5, I come up "saslToken
> > > missing"
> > > > exception which I mentioned above!
> > > > I don't really understand what saslToken is and how to make it get
> > > > generated for Digest auth!
> > > > Please assist!
> > > >
> > > > On Thu, Nov 9, 2023 at 7:15 PM Alex Craig <al...@gmail.com>
> > wrote:
> > > >
> > > > > " I couldn't find any doc by kafka to enable Digest-MD5
> > > authentication."
> > > > > This was the 2nd result in a google search:
> > > > >
> https://docs.confluent.io/platform/current/security/zk-security.html
> > > > >
> > > > > " I don't want to enable SASL."
> > > > > Digest-MD5 is SASL authentication, so not sure what you mean here.
> > > > >
> > > > > " If I set zookeeper.set.acl=true, I'm forced to configure TLS."
> > > > > Hmm, that config shouldn't have anything to do with TLS. You can
> set
> > > > ACL's
> > > > > with or without TLS encryption. Were you getting an error?
> > > > >
> > > > > On Wed, Nov 8, 2023 at 11:35 PM arjun s v <ar...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Team,
> > > > > >
> > > > > > Please consider this as high priority, we need to enable
> > > authentication
> > > > > > ASAP. Please assist.
> > > > > > On Tue, Nov 7, 2023 at 4:38 PM arjun s v <ar...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi team,
> > > > > > >
> > > > > > > I'm trying to configure *Digest-MD5* authentication between
> kafka
> > > and
> > > > > > > zookeeper.
> > > > > > > Also I need to set ACL with digest scheme and credentials.
> > > > > > > I don't want to enable SASL.
> > > > > > > I tried to follow this
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
> > > > > >
> > > > > > doc
> > > > > > > from zookeeper,
> > > > > > >
> > > > > > > - If I configured a jaas file, I have to set
> > > > > > zookeeper.sasl.client=true(if
> > > > > > > not kafka throws error from JaasUtils) which enables sasl
> > > > > > authentication.
> > > > > > > - If I set zookeeper.set.acl=true, I'm forced to configure
> > TLS.
> > > > > > >
> > > > > > > I couldn't find any doc by kafka to enable Digest-MD5
> > > authentication.
> > > > > > > I cannot configure kerberos or TLS, just a Digest-MD5 is
> > sufficient
> > > > for
> > > > > > my
> > > > > > > usecase.
> > > > > > > Please let me know if there are any docs to enable Digest-MD5
> > auth
> > > > > > between
> > > > > > > kafka and zookeeper.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Arjun S V
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Digest-MD5 authentication between kafka and zookeeper
Posted by Alex Brekken <br...@gmail.com>.
I see a couple of things that look wrong. First, remove this line from your
ZK config:
zookeeper.authProvider.1=org.apache.zookeeper.server.auth.DigestAuthenticationProvider.
And replace it with this:
authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider.
Additionally, I think you need to add these lines to the ZK config if you
want ZK to ZK authentication:
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=true
quorum.auth.serverRequireSasl=true
The rest looks OK.
On Sat, Nov 11, 2023 at 9:12 PM arjun s v <ar...@gmail.com> wrote:
> A small correction, I'm not trying to enable mTLS, just a simple
> authentication(Digest or Plain) is enough,
> Sharing the jaas files and config files,
> kafka_server_jaas.conf
>
> > admin.KafkaServer{
> >
> > org.apache.kafka.common.security.plain.PlainLoginModule required
> >
> > username="USERNAME"
> >
> > password="PASSWORD";
> >
> > };
> >
> > KafkaServer{
> >
> > org.apache.kafka.common.security.plain.PlainLoginModule required
> >
> > username="USERNAME"
> >
> > password="PASSWORD";
> >
> > };
> >
> > Client{
> >
> > org.apache.zookeeper.server.auth.DigestLoginModule required
> >
> > username="super"
> >
> > password="adminsecret";
> >
> > };
> >
> server properties
>
> > zookeeper.sasl.client=false
>
> java.security.auth.login.config=$base_dir/../config/kafka_server_jaas.conf
>
> zookeeper.connection.timeout.ms=6000
>
> zookeeper.sync.time.ms=2000
>
> zookeeper.set.acl=true
>
>
> zk_server_jaas.conf
>
> > QuorumServer{
> > org.apache.zookeeper.server.auth.DigestLoginModule required
> > user_test="test";
> > };
> > QuorumLearner{
> > org.apache.zookeeper.server.auth.DigestLoginModule required
> > username="test"
> > password="test";
> > };
> > Server{
> > org.apache.zookeeper.server.auth.DigestLoginModule required
> > username="super"
> > password="adminsecret";
>
> };
>
>
> zoo.cfg
>
> >
> >
> zookeeper.authProvider.1=org.apache.zookeeper.server.auth.DigestAuthenticationProvider
>
>
> env
>
> > SERVER_JVMFLAGS="$SERVER_JVMFLAGS
> > -Djava.security.auth.login.config=$ZOOCFGDIR/zk_server_jaas.conf"
> >
> On Fri, Nov 10, 2023 at 7:03 PM Alex Brekken <br...@gmail.com> wrote:
>
> > Ok, so you're trying to enable both SASL authentication (digest) and TLS,
> > using mTLS for Zookeeper? I'm just trying to understand the bigger
> > picture. The error you're getting regarding the Sasl token sounds like
> > either the jaas config on the Kafka broker side is wrong/missing, or the
> > jaas config on the ZK side is wrong/missing. (you need both - in this
> case
> > the broker is the "client" and ZK is the "server"). Are you able to share
> > the jaas config you're using for both Kafka and ZK? Without seeing that
> > it's tough to know. Also, to make troubleshooting easier you might want
> to
> > leave TLS out of it for now and get SASL working first. (or vice-versa)
> >
> >
> >
> > On Thu, Nov 9, 2023 at 11:26 PM arjun s v <ar...@gmail.com> wrote:
> >
> > > "Digest-MD5 is SASL authentication, so not sure what you mean here."
> > > If I set zookeeper.sasl.client=true, zookeeper expects a "saslToken"
> and
> > > throws the following error,
> > >
> > > "SASL authentication failed using login context 'Client' with
> exception:
> > > {}" "javax.security.sasl.SaslException: Error in authenticating with a
> > > Zookeeper Quorum member: the quorum member's saslToken is null.
> > >
> > > at
> > >
> > >
> >
> org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
> > >
> > > at
> > >
> > >
> >
> org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
> > >
> > > at
> > >
> > >
> >
> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
> > >
> > > at
> > >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103)
> > >
> > > at
> > >
> > >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365)
> > >
> > > at
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
> > >
> > >
> > > "Hmm, that config shouldn't have anything to do with TLS. You can set
> > ACL's
> > >
> > > with or without TLS encryption. Were you getting an error?"
> > >
> > >
> > > "Fatal error during KafkaServer startup. Prepare to shutdown"
> > > "java.lang.SecurityException: zookeeper.set.acl is true, but ZooKeeper
> > > client TLS configuration identifying at least
> > > kafka.server.KafkaConfig$@7b22ec89.ZkSslClientEnableProp,
> > > kafka.server.KafkaConfig$@7b22ec89.ZkClientCnxnSocketProp, and
> > > kafka.server.KafkaConfig$@7b22ec89.ZkSslKeyStoreLocationProp was not
> > > present and the verification of the JAAS login file failed
> > > [java.security.auth.login.config=./../config/kafka_server_jaas.conf,
> > > zookeeper.sasl.client=false,
> zookeeper.sasl.clientconfig=default:Client]
> > >
> > > at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:445)
> > >
> > > at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
> > >
> > > at kafka.Kafka$.main(Kafka.scala:109)
> > >
> > > at kafka.Kafka.main(Kafka.scala)
> > >
> > >
> > > "This was the 2nd result in a google search:
> > > https://docs.confluent.io/platform/current/security/zk-security.html"
> > >
> > > FYKI, I've googled, asked chat gpt, surfed over many zookeeper and
> kafka
> > > docs and blog,
> > > I remember trying the doc you suggested here about 10 days back in the
> > > initial days of this task!
> > > About the doc you suggested,
> > > I cannot configure SSL as I already mentioned, If I skip ssl config
> part
> > > from your suggested doc and tried Digest-MD5, I come up "saslToken
> > missing"
> > > exception which I mentioned above!
> > > I don't really understand what saslToken is and how to make it get
> > > generated for Digest auth!
> > > Please assist!
> > >
> > > On Thu, Nov 9, 2023 at 7:15 PM Alex Craig <al...@gmail.com>
> wrote:
> > >
> > > > " I couldn't find any doc by kafka to enable Digest-MD5
> > authentication."
> > > > This was the 2nd result in a google search:
> > > > https://docs.confluent.io/platform/current/security/zk-security.html
> > > >
> > > > " I don't want to enable SASL."
> > > > Digest-MD5 is SASL authentication, so not sure what you mean here.
> > > >
> > > > " If I set zookeeper.set.acl=true, I'm forced to configure TLS."
> > > > Hmm, that config shouldn't have anything to do with TLS. You can set
> > > ACL's
> > > > with or without TLS encryption. Were you getting an error?
> > > >
> > > > On Wed, Nov 8, 2023 at 11:35 PM arjun s v <ar...@gmail.com>
> > wrote:
> > > >
> > > > > Team,
> > > > >
> > > > > Please consider this as high priority, we need to enable
> > authentication
> > > > > ASAP. Please assist.
> > > > > On Tue, Nov 7, 2023 at 4:38 PM arjun s v <ar...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi team,
> > > > > >
> > > > > > I'm trying to configure *Digest-MD5* authentication between kafka
> > and
> > > > > > zookeeper.
> > > > > > Also I need to set ACL with digest scheme and credentials.
> > > > > > I don't want to enable SASL.
> > > > > > I tried to follow this
> > > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
> > > > >
> > > > > doc
> > > > > > from zookeeper,
> > > > > >
> > > > > > - If I configured a jaas file, I have to set
> > > > > zookeeper.sasl.client=true(if
> > > > > > not kafka throws error from JaasUtils) which enables sasl
> > > > > authentication.
> > > > > > - If I set zookeeper.set.acl=true, I'm forced to configure
> TLS.
> > > > > >
> > > > > > I couldn't find any doc by kafka to enable Digest-MD5
> > authentication.
> > > > > > I cannot configure kerberos or TLS, just a Digest-MD5 is
> sufficient
> > > for
> > > > > my
> > > > > > usecase.
> > > > > > Please let me know if there are any docs to enable Digest-MD5
> auth
> > > > > between
> > > > > > kafka and zookeeper.
> > > > > >
> > > > > > Regards,
> > > > > > Arjun S V
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Digest-MD5 authentication between kafka and zookeeper
Posted by arjun s v <ar...@gmail.com>.
A small correction, I'm not trying to enable mTLS, just a simple
authentication(Digest or Plain) is enough,
Sharing the jaas files and config files,
kafka_server_jaas.conf
> admin.KafkaServer{
>
> org.apache.kafka.common.security.plain.PlainLoginModule required
>
> username="USERNAME"
>
> password="PASSWORD";
>
> };
>
> KafkaServer{
>
> org.apache.kafka.common.security.plain.PlainLoginModule required
>
> username="USERNAME"
>
> password="PASSWORD";
>
> };
>
> Client{
>
> org.apache.zookeeper.server.auth.DigestLoginModule required
>
> username="super"
>
> password="adminsecret";
>
> };
>
server properties
> zookeeper.sasl.client=false
java.security.auth.login.config=$base_dir/../config/kafka_server_jaas.conf
zookeeper.connection.timeout.ms=6000
zookeeper.sync.time.ms=2000
zookeeper.set.acl=true
zk_server_jaas.conf
> QuorumServer{
> org.apache.zookeeper.server.auth.DigestLoginModule required
> user_test="test";
> };
> QuorumLearner{
> org.apache.zookeeper.server.auth.DigestLoginModule required
> username="test"
> password="test";
> };
> Server{
> org.apache.zookeeper.server.auth.DigestLoginModule required
> username="super"
> password="adminsecret";
};
zoo.cfg
>
> zookeeper.authProvider.1=org.apache.zookeeper.server.auth.DigestAuthenticationProvider
env
> SERVER_JVMFLAGS="$SERVER_JVMFLAGS
> -Djava.security.auth.login.config=$ZOOCFGDIR/zk_server_jaas.conf"
>
On Fri, Nov 10, 2023 at 7:03 PM Alex Brekken <br...@gmail.com> wrote:
> Ok, so you're trying to enable both SASL authentication (digest) and TLS,
> using mTLS for Zookeeper? I'm just trying to understand the bigger
> picture. The error you're getting regarding the Sasl token sounds like
> either the jaas config on the Kafka broker side is wrong/missing, or the
> jaas config on the ZK side is wrong/missing. (you need both - in this case
> the broker is the "client" and ZK is the "server"). Are you able to share
> the jaas config you're using for both Kafka and ZK? Without seeing that
> it's tough to know. Also, to make troubleshooting easier you might want to
> leave TLS out of it for now and get SASL working first. (or vice-versa)
>
>
>
> On Thu, Nov 9, 2023 at 11:26 PM arjun s v <ar...@gmail.com> wrote:
>
> > "Digest-MD5 is SASL authentication, so not sure what you mean here."
> > If I set zookeeper.sasl.client=true, zookeeper expects a "saslToken" and
> > throws the following error,
> >
> > "SASL authentication failed using login context 'Client' with exception:
> > {}" "javax.security.sasl.SaslException: Error in authenticating with a
> > Zookeeper Quorum member: the quorum member's saslToken is null.
> >
> > at
> >
> >
> org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
> >
> > at
> >
> >
> org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
> >
> > at
> >
> >
> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
> >
> > at
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103)
> >
> > at
> >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365)
> >
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
> >
> >
> > "Hmm, that config shouldn't have anything to do with TLS. You can set
> ACL's
> >
> > with or without TLS encryption. Were you getting an error?"
> >
> >
> > "Fatal error during KafkaServer startup. Prepare to shutdown"
> > "java.lang.SecurityException: zookeeper.set.acl is true, but ZooKeeper
> > client TLS configuration identifying at least
> > kafka.server.KafkaConfig$@7b22ec89.ZkSslClientEnableProp,
> > kafka.server.KafkaConfig$@7b22ec89.ZkClientCnxnSocketProp, and
> > kafka.server.KafkaConfig$@7b22ec89.ZkSslKeyStoreLocationProp was not
> > present and the verification of the JAAS login file failed
> > [java.security.auth.login.config=./../config/kafka_server_jaas.conf,
> > zookeeper.sasl.client=false, zookeeper.sasl.clientconfig=default:Client]
> >
> > at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:445)
> >
> > at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
> >
> > at kafka.Kafka$.main(Kafka.scala:109)
> >
> > at kafka.Kafka.main(Kafka.scala)
> >
> >
> > "This was the 2nd result in a google search:
> > https://docs.confluent.io/platform/current/security/zk-security.html"
> >
> > FYKI, I've googled, asked chat gpt, surfed over many zookeeper and kafka
> > docs and blog,
> > I remember trying the doc you suggested here about 10 days back in the
> > initial days of this task!
> > About the doc you suggested,
> > I cannot configure SSL as I already mentioned, If I skip ssl config part
> > from your suggested doc and tried Digest-MD5, I come up "saslToken
> missing"
> > exception which I mentioned above!
> > I don't really understand what saslToken is and how to make it get
> > generated for Digest auth!
> > Please assist!
> >
> > On Thu, Nov 9, 2023 at 7:15 PM Alex Craig <al...@gmail.com> wrote:
> >
> > > " I couldn't find any doc by kafka to enable Digest-MD5
> authentication."
> > > This was the 2nd result in a google search:
> > > https://docs.confluent.io/platform/current/security/zk-security.html
> > >
> > > " I don't want to enable SASL."
> > > Digest-MD5 is SASL authentication, so not sure what you mean here.
> > >
> > > " If I set zookeeper.set.acl=true, I'm forced to configure TLS."
> > > Hmm, that config shouldn't have anything to do with TLS. You can set
> > ACL's
> > > with or without TLS encryption. Were you getting an error?
> > >
> > > On Wed, Nov 8, 2023 at 11:35 PM arjun s v <ar...@gmail.com>
> wrote:
> > >
> > > > Team,
> > > >
> > > > Please consider this as high priority, we need to enable
> authentication
> > > > ASAP. Please assist.
> > > > On Tue, Nov 7, 2023 at 4:38 PM arjun s v <ar...@gmail.com>
> > wrote:
> > > >
> > > > > Hi team,
> > > > >
> > > > > I'm trying to configure *Digest-MD5* authentication between kafka
> and
> > > > > zookeeper.
> > > > > Also I need to set ACL with digest scheme and credentials.
> > > > > I don't want to enable SASL.
> > > > > I tried to follow this
> > > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
> > > >
> > > > doc
> > > > > from zookeeper,
> > > > >
> > > > > - If I configured a jaas file, I have to set
> > > > zookeeper.sasl.client=true(if
> > > > > not kafka throws error from JaasUtils) which enables sasl
> > > > authentication.
> > > > > - If I set zookeeper.set.acl=true, I'm forced to configure TLS.
> > > > >
> > > > > I couldn't find any doc by kafka to enable Digest-MD5
> authentication.
> > > > > I cannot configure kerberos or TLS, just a Digest-MD5 is sufficient
> > for
> > > > my
> > > > > usecase.
> > > > > Please let me know if there are any docs to enable Digest-MD5 auth
> > > > between
> > > > > kafka and zookeeper.
> > > > >
> > > > > Regards,
> > > > > Arjun S V
> > > > >
> > > >
> > >
> >
>
Re: Digest-MD5 authentication between kafka and zookeeper
Posted by Alex Brekken <br...@gmail.com>.
Ok, so you're trying to enable both SASL authentication (digest) and TLS,
using mTLS for Zookeeper? I'm just trying to understand the bigger
picture. The error you're getting regarding the Sasl token sounds like
either the jaas config on the Kafka broker side is wrong/missing, or the
jaas config on the ZK side is wrong/missing. (you need both - in this case
the broker is the "client" and ZK is the "server"). Are you able to share
the jaas config you're using for both Kafka and ZK? Without seeing that
it's tough to know. Also, to make troubleshooting easier you might want to
leave TLS out of it for now and get SASL working first. (or vice-versa)
On Thu, Nov 9, 2023 at 11:26 PM arjun s v <ar...@gmail.com> wrote:
> "Digest-MD5 is SASL authentication, so not sure what you mean here."
> If I set zookeeper.sasl.client=true, zookeeper expects a "saslToken" and
> throws the following error,
>
> "SASL authentication failed using login context 'Client' with exception:
> {}" "javax.security.sasl.SaslException: Error in authenticating with a
> Zookeeper Quorum member: the quorum member's saslToken is null.
>
> at
>
> org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
>
> at
>
> org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
>
> at
>
> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
>
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103)
>
> at
>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
>
>
> "Hmm, that config shouldn't have anything to do with TLS. You can set ACL's
>
> with or without TLS encryption. Were you getting an error?"
>
>
> "Fatal error during KafkaServer startup. Prepare to shutdown"
> "java.lang.SecurityException: zookeeper.set.acl is true, but ZooKeeper
> client TLS configuration identifying at least
> kafka.server.KafkaConfig$@7b22ec89.ZkSslClientEnableProp,
> kafka.server.KafkaConfig$@7b22ec89.ZkClientCnxnSocketProp, and
> kafka.server.KafkaConfig$@7b22ec89.ZkSslKeyStoreLocationProp was not
> present and the verification of the JAAS login file failed
> [java.security.auth.login.config=./../config/kafka_server_jaas.conf,
> zookeeper.sasl.client=false, zookeeper.sasl.clientconfig=default:Client]
>
> at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:445)
>
> at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
>
> at kafka.Kafka$.main(Kafka.scala:109)
>
> at kafka.Kafka.main(Kafka.scala)
>
>
> "This was the 2nd result in a google search:
> https://docs.confluent.io/platform/current/security/zk-security.html"
>
> FYKI, I've googled, asked chat gpt, surfed over many zookeeper and kafka
> docs and blog,
> I remember trying the doc you suggested here about 10 days back in the
> initial days of this task!
> About the doc you suggested,
> I cannot configure SSL as I already mentioned, If I skip ssl config part
> from your suggested doc and tried Digest-MD5, I come up "saslToken missing"
> exception which I mentioned above!
> I don't really understand what saslToken is and how to make it get
> generated for Digest auth!
> Please assist!
>
> On Thu, Nov 9, 2023 at 7:15 PM Alex Craig <al...@gmail.com> wrote:
>
> > " I couldn't find any doc by kafka to enable Digest-MD5 authentication."
> > This was the 2nd result in a google search:
> > https://docs.confluent.io/platform/current/security/zk-security.html
> >
> > " I don't want to enable SASL."
> > Digest-MD5 is SASL authentication, so not sure what you mean here.
> >
> > " If I set zookeeper.set.acl=true, I'm forced to configure TLS."
> > Hmm, that config shouldn't have anything to do with TLS. You can set
> ACL's
> > with or without TLS encryption. Were you getting an error?
> >
> > On Wed, Nov 8, 2023 at 11:35 PM arjun s v <ar...@gmail.com> wrote:
> >
> > > Team,
> > >
> > > Please consider this as high priority, we need to enable authentication
> > > ASAP. Please assist.
> > > On Tue, Nov 7, 2023 at 4:38 PM arjun s v <ar...@gmail.com>
> wrote:
> > >
> > > > Hi team,
> > > >
> > > > I'm trying to configure *Digest-MD5* authentication between kafka and
> > > > zookeeper.
> > > > Also I need to set ACL with digest scheme and credentials.
> > > > I don't want to enable SASL.
> > > > I tried to follow this
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
> > >
> > > doc
> > > > from zookeeper,
> > > >
> > > > - If I configured a jaas file, I have to set
> > > zookeeper.sasl.client=true(if
> > > > not kafka throws error from JaasUtils) which enables sasl
> > > authentication.
> > > > - If I set zookeeper.set.acl=true, I'm forced to configure TLS.
> > > >
> > > > I couldn't find any doc by kafka to enable Digest-MD5 authentication.
> > > > I cannot configure kerberos or TLS, just a Digest-MD5 is sufficient
> for
> > > my
> > > > usecase.
> > > > Please let me know if there are any docs to enable Digest-MD5 auth
> > > between
> > > > kafka and zookeeper.
> > > >
> > > > Regards,
> > > > Arjun S V
> > > >
> > >
> >
>
Re: Digest-MD5 authentication between kafka and zookeeper
Posted by arjun s v <ar...@gmail.com>.
"Digest-MD5 is SASL authentication, so not sure what you mean here."
If I set zookeeper.sasl.client=true, zookeeper expects a "saslToken" and
throws the following error,
"SASL authentication failed using login context 'Client' with exception:
{}" "javax.security.sasl.SaslException: Error in authenticating with a
Zookeeper Quorum member: the quorum member's saslToken is null.
at
org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
at
org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
at
org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
"Hmm, that config shouldn't have anything to do with TLS. You can set ACL's
with or without TLS encryption. Were you getting an error?"
"Fatal error during KafkaServer startup. Prepare to shutdown"
"java.lang.SecurityException: zookeeper.set.acl is true, but ZooKeeper
client TLS configuration identifying at least
kafka.server.KafkaConfig$@7b22ec89.ZkSslClientEnableProp,
kafka.server.KafkaConfig$@7b22ec89.ZkClientCnxnSocketProp, and
kafka.server.KafkaConfig$@7b22ec89.ZkSslKeyStoreLocationProp was not
present and the verification of the JAAS login file failed
[java.security.auth.login.config=./../config/kafka_server_jaas.conf,
zookeeper.sasl.client=false, zookeeper.sasl.clientconfig=default:Client]
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:445)
at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
"This was the 2nd result in a google search:
https://docs.confluent.io/platform/current/security/zk-security.html"
FYKI, I've googled, asked chat gpt, surfed over many zookeeper and kafka
docs and blog,
I remember trying the doc you suggested here about 10 days back in the
initial days of this task!
About the doc you suggested,
I cannot configure SSL as I already mentioned, If I skip ssl config part
from your suggested doc and tried Digest-MD5, I come up "saslToken missing"
exception which I mentioned above!
I don't really understand what saslToken is and how to make it get
generated for Digest auth!
Please assist!
On Thu, Nov 9, 2023 at 7:15 PM Alex Craig <al...@gmail.com> wrote:
> " I couldn't find any doc by kafka to enable Digest-MD5 authentication."
> This was the 2nd result in a google search:
> https://docs.confluent.io/platform/current/security/zk-security.html
>
> " I don't want to enable SASL."
> Digest-MD5 is SASL authentication, so not sure what you mean here.
>
> " If I set zookeeper.set.acl=true, I'm forced to configure TLS."
> Hmm, that config shouldn't have anything to do with TLS. You can set ACL's
> with or without TLS encryption. Were you getting an error?
>
> On Wed, Nov 8, 2023 at 11:35 PM arjun s v <ar...@gmail.com> wrote:
>
> > Team,
> >
> > Please consider this as high priority, we need to enable authentication
> > ASAP. Please assist.
> > On Tue, Nov 7, 2023 at 4:38 PM arjun s v <ar...@gmail.com> wrote:
> >
> > > Hi team,
> > >
> > > I'm trying to configure *Digest-MD5* authentication between kafka and
> > > zookeeper.
> > > Also I need to set ACL with digest scheme and credentials.
> > > I don't want to enable SASL.
> > > I tried to follow this
> > > <
> >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
> >
> > doc
> > > from zookeeper,
> > >
> > > - If I configured a jaas file, I have to set
> > zookeeper.sasl.client=true(if
> > > not kafka throws error from JaasUtils) which enables sasl
> > authentication.
> > > - If I set zookeeper.set.acl=true, I'm forced to configure TLS.
> > >
> > > I couldn't find any doc by kafka to enable Digest-MD5 authentication.
> > > I cannot configure kerberos or TLS, just a Digest-MD5 is sufficient for
> > my
> > > usecase.
> > > Please let me know if there are any docs to enable Digest-MD5 auth
> > between
> > > kafka and zookeeper.
> > >
> > > Regards,
> > > Arjun S V
> > >
> >
>
Re: Digest-MD5 authentication between kafka and zookeeper
Posted by Alex Craig <al...@gmail.com>.
" I couldn't find any doc by kafka to enable Digest-MD5 authentication."
This was the 2nd result in a google search:
https://docs.confluent.io/platform/current/security/zk-security.html
" I don't want to enable SASL."
Digest-MD5 is SASL authentication, so not sure what you mean here.
" If I set zookeeper.set.acl=true, I'm forced to configure TLS."
Hmm, that config shouldn't have anything to do with TLS. You can set ACL's
with or without TLS encryption. Were you getting an error?
On Wed, Nov 8, 2023 at 11:35 PM arjun s v <ar...@gmail.com> wrote:
> Team,
>
> Please consider this as high priority, we need to enable authentication
> ASAP. Please assist.
> On Tue, Nov 7, 2023 at 4:38 PM arjun s v <ar...@gmail.com> wrote:
>
> > Hi team,
> >
> > I'm trying to configure *Digest-MD5* authentication between kafka and
> > zookeeper.
> > Also I need to set ACL with digest scheme and credentials.
> > I don't want to enable SASL.
> > I tried to follow this
> > <
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication>
> doc
> > from zookeeper,
> >
> > - If I configured a jaas file, I have to set
> zookeeper.sasl.client=true(if
> > not kafka throws error from JaasUtils) which enables sasl
> authentication.
> > - If I set zookeeper.set.acl=true, I'm forced to configure TLS.
> >
> > I couldn't find any doc by kafka to enable Digest-MD5 authentication.
> > I cannot configure kerberos or TLS, just a Digest-MD5 is sufficient for
> my
> > usecase.
> > Please let me know if there are any docs to enable Digest-MD5 auth
> between
> > kafka and zookeeper.
> >
> > Regards,
> > Arjun S V
> >
>
Re: Digest-MD5 authentication between kafka and zookeeper
Posted by arjun s v <ar...@gmail.com>.
Team,
Please consider this as high priority, we need to enable authentication
ASAP. Please assist.
On Tue, Nov 7, 2023 at 4:38 PM arjun s v <ar...@gmail.com> wrote:
> Hi team,
>
> I'm trying to configure *Digest-MD5* authentication between kafka and
> zookeeper.
> Also I need to set ACL with digest scheme and credentials.
> I don't want to enable SASL.
> I tried to follow this
> <https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication> doc
> from zookeeper,
>
> - If I configured a jaas file, I have to set zookeeper.sasl.client=true(if
> not kafka throws error from JaasUtils) which enables sasl authentication.
> - If I set zookeeper.set.acl=true, I'm forced to configure TLS.
>
> I couldn't find any doc by kafka to enable Digest-MD5 authentication.
> I cannot configure kerberos or TLS, just a Digest-MD5 is sufficient for my
> usecase.
> Please let me know if there are any docs to enable Digest-MD5 auth between
> kafka and zookeeper.
>
> Regards,
> Arjun S V
>