You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Sebastian Schmitz <se...@propellerhead.co.nz> on 2020/02/11 01:55:50 UTC
Enabling Auth between Zookeeper Servers
Hello,
I'm currently looking into enabling the Auth between Zookeeper-Servers
and found this documentation:
https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
However, when I use the config from the document (for Digest-MD5) I get
this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried because
I thought using latest version could help:
java.io.IOException: No JAAS configuration section named 'Server' was
found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
And of course that's right, because there's only QuorumServer and
QuorumClient in the jaas.conf:
jaas.conf:
QuorumServer {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test";
};
QuorumClient {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zookeeper"
password="test";
};
I also tried renaming the QuorumServer to just "Server". No change.
My zoo.cfg:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/mnt/zk_data
clientPort=2181
dataLogDir=/mnt/zk_data_log
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=false
quorum.auth.serverRequireSasl=false
quorum.auth.learner.loginContext=QuorumLearner
quorum.auth.server.loginContext=QuorumServer
quorum.cnxn.threads.size=20
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
secureClientPort=2281
server.1=0.0.0.0:2888:3888
Any idea what I could try? Or maybe there's some better document on how
to achieve this?
Thank you
Sebastian
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: Enabling Auth between Zookeeper Servers
Posted by Szalay-Bekő Máté <sz...@gmail.com>.
congrats! :)
> @Mate: as I copied the jaas.conf from your repo is that the exact file
you used for testing? Because changing the "user_zookeeper" to "user_kafka"
in the server-part fixed it.
if you mean this file, then yes, I used this for testing:
https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_jaas.conf
and it worked for me... strange that in your case you had to change it.
Regarding the usefulness of the error message: I am not sure if we can
change that, it should come from a java system library. ZooKeeper just
catch the SecurityException, and doesn't really analyze its content.
Still, it would be great at least to at least print out the exact security
exception (at least with debug logging) and also update the wiki /
documentation and highlighting that different user names may need to be
used.
If you create such a ticket, please also add the exact java version you
used in the docker image.
Good luck for the Kafka work! :)
Mate
On Mon, Feb 17, 2020 at 8:40 PM Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> Hello,
>
> I think I found the issue...
>
> One can't use the same username for clients and quorums. I configured
> all of them to be "zookeeper", but in the server-part of the jaas.conf
> it should probably be more like "kafka" as it's Kafka which
> authenticates to the zookeeper in that case and zookeepers are using the
> qorum-part to authenticate to each other. Correct?
> If that's correct the exception message is completely wrong. It can find
> the file, it can read the file and it even finds the server-part, but
> the server-part itself has wrong configuration.
>
> At least with the hanged username in the server-part I got a new exception:
>
> 2020-02-17 19:28:17,994 [myid:1] - ERROR [main:ZooKeeperServerMain@83] -
> Unexpected exception, exiting abnormally
> java.io.IOException: No snapshot found, but there are log entries.
> Something is broken!
>
> Which was probably caused by non-cleaned folders of some previous
> deployments. So I added the "snapshot.trust.empty=true" to the config to
> have it start and rebuild the snapshot. And now my zookeeper is running
> just fine! :)
>
> @Mate: as I copied the jaas.conf from your repo is that the exact file
> you used for testing? Because changing the "user_zookeeper" to
> "user_kafka" in the server-part fixed it.
>
> My next task now is to get Kafka authenticated to zookeeper and get ACLs
> working. Will be fun :)
> And I should probably create a ticket to get the jaas.conf-error message
> fixed!?
>
> Best regards
>
> Sebastian
>
>
> On 17-Feb-20 1:50 PM, Sebastian Schmitz wrote:
> > Hey,
> >
> > I also just tried using 3.5.7, but same problem...
> >
> > Best regards
> >
> > Sebastian
> >
> >
> > On 17-Feb-20 11:34 AM, Sebastian Schmitz wrote:
> >> Hi Mate,
> >>
> >> that's what I also tried. I copied it to the
> >> /opt/zookeeper-cluster/-folder and got the same exception just with
> >> the new path.
> >>
> >> So, if that config works on your side it might be my environment
> >> then!? Maybe it's a problem with the base-image
> >> openjdk:11-jre-stretch which I use for the container... I'll try
> >> using the openjdk:8u222-jre you're using.
> >>
> >> Best regards
> >>
> >> Sebastian
> >>
> >>
> >> On 17-Feb-20 9:19 AM, Szalay-Bekő Máté wrote:
> >>> Hi Sebastian,
> >>>
> >>> It's strange indeed... I also see the owner is root. That should
> >>> work in
> >>> docker usually, given that you run the zookeeper process with the root
> >>> user. Maybe copying it to a different folder? I see that the conf
> >>> folder
> >>> has different owner, maybe the java security library doesn't like that?
> >>>
> >>> But honestly, I don't have any useful explanation.
> >>>
> >>> Good luck!
> >>> Mate
> >>>
> >>> On Sun, Feb 16, 2020, 20:06 Sebastian Schmitz <
> >>> sebastian.schmitz@propellerhead.co.nz> wrote:
> >>>
> >>>> Hey Mate,
> >>>>
> >>>> now it gets really weird. I get the file not found exception:
> >>>>
> >>>> '.20-02-16 18:27:50,530 [myid:1] - ERROR
> >>>> [main:ServerCnxnFactory@246] -
> >>>> No JAAS configuration section named 'Server' was found in
> >>>> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >>>> java.lang.SecurityException: java.io.IOException:
> >>>> /opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >>>> (No such file or directory)
> >>>> at
> >>>> java.base/sun.security.provider.ConfigFile$Spi.<init>(Unknown Source)
> >>>> at java.base/sun.security.provider.ConfigFile.<init>(Unknown
> >>>> Source)
> >>>> at
> >>>>
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>
> >>>>
> >>>>
> >>>> Method)
> >>>> at
> >>>>
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>
> >>>>
> >>>>
> >>>> Source)
> >>>> at
> >>>>
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>
> >>>>
> >>>>
> >>>> Source)
> >>>> at
> >>>> java.base/java.lang.reflect.Constructor.newInstance(Unknown
> >>>> Source)
> >>>> at java.base/java.lang.Class.newInstance(Unknown Source)
> >>>> at
> >>>> java.base/javax.security.auth.login.Configuration$2.run(Unknown
> >>>> Source)
> >>>> at
> >>>> java.base/javax.security.auth.login.Configuration$2.run(Unknown
> >>>> Source)
> >>>> at
> >>>> java.base/java.security.AccessController.doPrivileged(Native
> >>>> Method)
> >>>> at
> >>>>
> java.base/javax.security.auth.login.Configuration.getConfiguration(Unknown
> >>>>
> >>>> Source)
> >>>> at
> >>>>
> >>>>
> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
>
> >>>>
> >>>> at
> >>>>
> >>>>
> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
>
> >>>>
> >>>> at
> >>>>
> >>>>
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
>
> >>>>
> >>>> at
> >>>>
> >>>>
> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
>
> >>>>
> >>>> at
> >>>>
> >>>>
> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
>
> >>>>
> >>>> at
> >>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
>
> >>>>
> >>>> at
> >>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
>
> >>>>
> >>>> Caused by: java.io.IOException:
> >>>> /opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >>>> (No such file or directory)
> >>>> at
> >>>> java.base/sun.security.provider.ConfigFile$Spi.ioException(Unknown
> >>>> Source)
> >>>> at
> >>>> java.base/sun.security.provider.ConfigFile$Spi.init(Unknown
> >>>> Source)
> >>>> ... 18 more
> >>>> 2020-02-16 18:27:50,566 [myid:1] - ERROR
> >>>> [main:ZooKeeperServerMain@83] -
> >>>> Unexpected exception, exiting abnormally
> >>>> java.io.IOException: No JAAS configuration section named 'Server' was
> >>>> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >>>> '.
> >>>> at
> >>>>
> >>>>
> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:247)
>
> >>>>
> >>>> at
> >>>>
> >>>>
> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
>
> >>>>
> >>>> at
> >>>>
> >>>>
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
>
> >>>>
> >>>> at
> >>>>
> >>>>
> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
>
> >>>>
> >>>> at
> >>>>
> >>>>
> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
>
> >>>>
> >>>> at
> >>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
>
> >>>>
> >>>> at
> >>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
>
> >>>>
> >>>>
> >>>> So I checked the image:
> >>>>
> >>>> root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# ls -la
> >>>> total 32
> >>>> drwxr-xr-x 2 1010 1011 4096 Feb 16 18:27 .
> >>>> drwxr-xr-x 12 root root 4096 Feb 16 18:27 ..
> >>>> -rw-r--r-- 1 1010 1011 535 Jan 30 12:18 configuration.xsl
> >>>> -rw-r--r-- 1 root root 600 Feb 13 18:32 jaas.conf
> >>>> -rw-r--r-- 1 root root 101 Feb 11 00:05 java.env
> >>>> -rw-r--r-- 1 1010 1011 2712 Feb 14 05:49 log4j.properties
> >>>> -rw-r--r-- 1 root root 1255 Feb 16 18:27 zoo.cfg
> >>>> -rw-r--r-- 1 1010 1011 922 Feb 14 05:49 zoo_sample.cfg
> >>>>
> >>>> And tried to output the file it states in the error:
> >>>>
> >>>> root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# cat
> >>>> /opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >>>> QuorumServer {
> >>>> org.apache.zookeeper.server.auth.DigestLoginModule required
> >>>> user_zookeeper="test";
> >>>> };
> >>>> QuorumClient {
> >>>> org.apache.zookeeper.server.auth.DigestLoginModule required
> >>>> username="zookeeper"
> >>>> password="test";
> >>>> };
> >>>> Server {
> >>>> org.apache.zookeeper.server.auth.DigestLoginModule required
> >>>> user_zookeeper="test";
> >>>> };
> >>>> Client {
> >>>> org.apache.zookeeper.server.auth.DigestLoginModule required
> >>>> username="zookeeper"
> >>>> password="test";
> >>>> };
> >>>>
> >>>> The weird part now is that the access is set exactly the same as the
> >>>> zoo.cfg which it can read without problems.
> >>>>
> >>>> Also changing the access to 666 doesn't change anything. And using
> >>>> your
> >>>> config doesn't help either:
> >>>>
> >>>> jaas.conf:
> >>>> QuorumServer {
> >>>> org.apache.zookeeper.server.auth.DigestLoginModule required
> >>>> user_zookeeper="test";
> >>>> };
> >>>> QuorumLearner {
> >>>> org.apache.zookeeper.server.auth.DigestLoginModule required
> >>>> username="zookeeper"
> >>>> password="test";
> >>>> };
> >>>> Server {
> >>>> org.apache.zookeeper.server.auth.DigestLoginModule required
> >>>> user_zookeeper="test";
> >>>> };
> >>>>
> >>>> zoo.cfg:
> >>>> tickTime=2000
> >>>> initLimit=10
> >>>> syncLimit=5
> >>>>
> >>>> dataDir=/mnt/zk_data
> >>>>
> >>>> clientPort=2181
> >>>>
> >>>> standaloneEnabled=true
> >>>> admin.enableServer=true
> >>>> localSessionsEnabled=true
> >>>> localSessionsUpgradingEnabled=true
> >>>>
> >>>> 4lw.commands.whitelist=stat, ruok, conf, isro, wchc, wchp, srvr,
> >>>> mntr, cons
> >>>>
> >>>> clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
> >>>> serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
> >>>>
> >>>>
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> >>>>
> >>>> quorum.auth.enableSasl=true
> >>>> quorum.auth.learnerRequireSasl=false
> >>>> quorum.auth.serverRequireSasl=false
> >>>> quorum.auth.learner.saslLoginContext=QuorumLearner
> >>>> quorum.auth.server.saslLoginContext=QuorumServer
> >>>> dataLogDir=/mnt/zk_data_log
> >>>> autopurge.snapRetainCount=3
> >>>> autopurge.purgeInterval=24
> >>>> quorum.cnxn.threads.size=20
> >>>> server.1=0.0.0.0:2888:3888
> >>>>
> >>>> I have no idea what's different now. I'll try to run the stuff from
> >>>> your
> >>>> repo and see if that works.
> >>>>
> >>>> Best regards
> >>>>
> >>>> Sebastian
> >>>>
> >>>> On 14-Feb-20 8:11 PM, Szalay-Bekő Máté wrote:
> >>>>> Hi Sebastian!
> >>>>>
> >>>>> I was able to setup digest authentication, uploaded my results here:
> >>>>> https://github.com/symat/zookeeper-docker-test
> >>>>> You can see my docker compose file:
> >>>>>
> >>>>
> https://github.com/symat/zookeeper-docker-test/blob/master/3_nodes_digest_quorum_auth.yml
> >>>>
> >>>>> also the zoo.cfg template:
> >>>>>
> >>>>
> https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_zoo.cfg
> >>>>
> >>>>> and the jaas.cfg file:
> >>>>>
> >>>>
> https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_jaas.conf
> >>>>
> >>>>> It works for me, using ZooKeeper 3.5.6. Although I haven't follow
> >>>>> your
> >>>>> config everywhere.
> >>>>>
> >>>>> Still, I wasn't able to reproduce your exception, only when I
> >>>>> actually
> >>>>> deleted the jaas config file. Are you sure that the ZooKeeper
> >>>>> process in
> >>>>> docker can see / open that file?
> >>>>>
> >>>>> I created a patched ZooKeeper 3.5.6 for you (you can download from
> >>>>> here:
> >>>>> https://drive.google.com/open?id=1KEPjNkiKf937jMJHAicwW9WATEuyRZIo),
> >>>> where
> >>>>> more details are printed in case of errors. E.g. in my case when I
> >>>> deleted
> >>>>> the jaas config file, I get:
> >>>>>
> >>>>> zoo1_1 | 2020-02-14 07:04:33,288 [myid:1] - ERROR
> >>>>> [main:ServerCnxnFactory@246] - No JAAS configuration section named
> >>>> 'Server'
> >>>>> was found in '/scripts/conf/digest_jaas.conf'.
> >>>>> zoo1_1 | java.lang.SecurityException: java.io.IOException:
> >>>>> /scripts/conf/digest_jaas.conf (No such file or directory)
> >>>>> zoo1_1 | at
> >>>>> sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:137)
> >>>>> zoo1_1 | at
> >>>>> sun.security.provider.ConfigFile.<init>(ConfigFile.java:102)
> >>>>> zoo1_1 | at
> >>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> >>>>> zoo1_1 | at
> >>>>>
> >>>>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
> >>>>
> >>>>> zoo1_1 | at
> >>>>>
> >>>>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> >>>>
> >>>>> zoo1_1 | at
> >>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> >>>>> zoo1_1 | at java.lang.Class.newInstance(Class.java:442)
> >>>>> zoo1_1 | at
> >>>>> javax.security.auth.login.Configuration$2.run(Configuration.java:255)
> >>>>> zoo1_1 | at
> >>>>> javax.security.auth.login.Configuration$2.run(Configuration.java:247)
> >>>>> zoo1_1 | at java.security.AccessController.doPrivileged(Native
> >>>>> Method)
> >>>>> zoo1_1 | at
> >>>>>
> >>>>
> javax.security.auth.login.Configuration.getConfiguration(Configuration.java:246)
>
> >>>>
> >>>>> zoo1_1 | at
> >>>>>
> >>>>
> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
>
> >>>>
> >>>>> zoo1_1 | at
> >>>>>
> >>>>
> org.apache.zookeeper.server.NettyServerCnxnFactory.configure(NettyServerCnxnFactory.java:383)
>
> >>>>
> >>>>> zoo1_1 | at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:148)
>
> >>>>
> >>>>> zoo1_1 | at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
>
> >>>>
> >>>>> zoo1_1 | at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
>
> >>>>
> >>>>> zoo1_1 | Caused by: java.io.IOException:
> >>>>> /scripts/conf/digest_jaas.conf
> >>>>> (No such file or directory)
> >>>>> zoo1_1 | at
> >>>>> sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:666)
> >>>>> zoo1_1 | at
> >>>>> sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:262)
> >>>>> zoo1_1 | at
> >>>>> sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:135)
> >>>>> zoo1_1 | ... 15 more
> >>>>> z
> >>>>>
> >>>>> Kind regards,
> >>>>> Mate
> >>>>>
> >>>>> On Fri, Feb 14, 2020 at 7:12 AM sagar shukla
> >>>>> <sa_shukla@yahoo.com.invalid
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> O
> >>>>>> Sent from Yahoo Mail on Android
> >>>>>>
> >>>>>> On Fri, Feb 14, 2020 at 11:02 AM, Szalay-Bekő Máté<
> >>>>>> szalay.beko.mate@gmail.com> wrote: Hi Sebastian,
> >>>>>>
> >>>>>>> But I still get the same exception.
> >>>>>> at this point I don't know why this happen... Adding the Server
> >>>>>> section
> >>>> to
> >>>>>> the jaas config should have helped. Unfortunately the exact
> >>>>>> exception is
> >>>>>> not printed out into the logs, just the error message, so it is
> >>>>>> hard to
> >>>>>> find out more details.
> >>>>>>
> >>>>>> I will try to reproduce your case with 3.5.6 locally and see if it
> >>>> works. I
> >>>>>> never actually used digest authentication before... we always use
> >>>> kerberos
> >>>>>> in production. If it works, I will share my configs / dockerfiles
> >>>>>> and
> >>>> send
> >>>>>> you a patched version with more debug info printed out.
> >>>>>>
> >>>>>>> Why would configuring quorum-auth also enable client-server-auth?
> >>>>>> it is not very logical indeed... if I see it right, based on the
> >>>>>> code
> >>>> once
> >>>>>> you set the java.security.auth.login.config property, then ZooKeeper
> >>>>>> assumes you want to use server-client sasl authentication. I
> >>>>>> guess the
> >>>>>> quorum-auth feature was added later and they introduced 'enable'
> >>>>>> config
> >>>>>> property for this, but forget to introduce the same config for the
> >>>> client
> >>>>>> authentication. I also guess most of the people are interested in
> >>>>>> the
> >>>>>> client authentication and it is rare that someone does't need
> >>>>>> that but
> >>>>>> needs quorum auth. Still, the current behaviour is not good I
> >>>>>> think. I
> >>>> will
> >>>>>> submit a jira ticket requesting an improvement here when I will have
> >>>> time,
> >>>>>> but feel free to submit it yourself if you wish.
> >>>>>>
> >>>>>> Kind regards,
> >>>>>> Mate
> >>>>>>
> >>>>>> On Thu, Feb 13, 2020 at 7:41 PM Sebastian Schmitz <
> >>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
> >>>>>>
> >>>>>>> Hey Mate,
> >>>>>>>
> >>>>>>> I checked the java.env-file and it contains:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>
> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"
>
> >>>>
> >>>>>>> which is exactly the place where the pasted jaas.conf is placed.
> >>>>>>>
> >>>>>>> I also just changed the config to be saslLoginContext and added the
> >>>>>>> missing semicolon.
> >>>>>>>
> >>>>>>> But I still get the same exception.
> >>>>>>>
> >>>>>>> Why would configuring quorum-auth also enable client-server-auth?
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>>
> >>>>>>> Sebastian
> >>>>>>>
> >>>>>>>
> >>>>>>> On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:
> >>>>>>>> Hi Sebastian,
> >>>>>>>>
> >>>>>>>> thanks for the more details!
> >>>>>>>>
> >>>>>>>> One thing I found in your config is that you should use:
> >>>>>>>> quorum.auth.learner.saslLoginContext=QuorumLearner
> >>>>>>>> quorum.auth.server.saslLoginContext=QuorumServer
> >>>>>>>>
> >>>>>>>> so instead of loginContext, use saslLoginContext in both
> >>>>>>>> lines. I
> >>>>>>> found
> >>>>>>>> this in the source code, I think the wiki is wrong (I will fix it
> >>>>>> later).
> >>>>>>>> However, actually this didn't really change anything, as the
> >>>>>>>> default
> >>>>>>> values
> >>>>>>>> are anyway
> >>>>>>>> QuorumLearner and QuorumServer, so you can even skip these
> >>>>>>>> lines from
> >>>>>>> the
> >>>>>>>> config.
> >>>>>>>>
> >>>>>>>> I think Rakesh is right, you are seeing exceptions related to
> >>>>>>>> not the
> >>>>>>>> QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to
> >>>>>>>> find
> >>>> the
> >>>>>>>> 'Server' section (what is configuring the server during the
> >>>>>> client-server
> >>>>>>>> authentication). The name of this section can be overwritten by
> >>>>>>>> the
> >>>>>>>> "zookeeper.sasl.serverconfig" system property.
> >>>>>>>>
> >>>>>>>> Based on the exception, ZooKeeper can not find the 'Server'
> >>>>>>>> section in
> >>>>>>>> the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are
> >>>>>>>> you sure
> >>>>>>> this
> >>>>>>>> is the correct jaas.conf? Does the ZooKeeper process have the
> >>>>>> permissions
> >>>>>>>> to open this file? You can specify the jaas config file path for
> >>>>>>> ZooKeeper
> >>>>>>>> by providing custom system property e.g. by exporting
> >>>>>>>>
> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
> >>>>>>>>
> >>>>>>>> before starting zkServer.sh
> >>>>>>>>
> >>>>>>>> Also in the jaas.conf you copied here, you are missing a semicolon
> >>>> from
> >>>>>>> the
> >>>>>>>> end of the last line in the Server block. I am not sure if it is
> >>>>>> causing
> >>>>>>>> any parsing error, but I always add the semicolon to the end of
> >>>>>>>> the
> >>>>>> last
> >>>>>>>> line in the block.
> >>>>>>>>
> >>>>>>>> Mate
> >>>>>>>>
> >>>>>>>> On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
> >>>>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
> >>>>>>>>
> >>>>>>>>> Hello Rakesh,
> >>>>>>>>>
> >>>>>>>>> as mentioned in the other mail adding the "Server"to jaas.conf
> >>>>>>>>> didn't
> >>>>>>> help.
> >>>>>>>>> Here are the Configs and Logs (with the Server-part included):
> >>>>>>>>>
> >>>>>>>>> jaas.conf:
> >>>>>>>>> QuorumServer {
> >>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
> >>>>>>>>> user_zookeeper="test";
> >>>>>>>>> };
> >>>>>>>>>
> >>>>>>>>> QuorumClient {
> >>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
> >>>>>>>>> username="zookeeper"
> >>>>>>>>> password="test";
> >>>>>>>>> };
> >>>>>>>>>
> >>>>>>>>> Server {
> >>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
> >>>>>>>>> user_zookeeper="test"
> >>>>>>>>> };
> >>>>>>>>>
> >>>>>>>>> Client {
> >>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
> >>>>>>>>> username="zookeeper"
> >>>>>>>>> password="test";
> >>>>>>>>> };
> >>>>>>>>>
> >>>>>>>>> zoo.cfg:
> >>>>>>>>> # The number of milliseconds of each tick
> >>>>>>>>> tickTime=2000
> >>>>>>>>> # The number of ticks that the initial
> >>>>>>>>> # synchronization phase can take
> >>>>>>>>> initLimit=10
> >>>>>>>>> # The number of ticks that can pass between
> >>>>>>>>> # sending a request and getting an acknowledgement
> >>>>>>>>> syncLimit=5
> >>>>>>>>> # the directory where the snapshot is stored.
> >>>>>>>>> # do not use /tmp for storage, /tmp here is just
> >>>>>>>>> # example sakes.
> >>>>>>>>> dataDir=/mnt/zk_data
> >>>>>>>>> # the port at which the clients will connect
> >>>>>>>>> clientPort=2181
> >>>>>>>>> # the maximum number of client connections.
> >>>>>>>>> # increase this if you need to handle more clients
> >>>>>>>>> #maxClientCnxns=60
> >>>>>>>>> #
> >>>>>>>>> # Be sure to read the maintenance section of the
> >>>>>>>>> # administrator guide before turning on autopurge.
> >>>>>>>>> #
> >>>>>>>>> #
> >>>>>>>>>
> >>>>
> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
> >>>>
> >>>>>>>>> #
> >>>>>>>>> # The number of snapshots to retain in dataDir
> >>>>>>>>> #autopurge.snapRetainCount=3
> >>>>>>>>> # Purge task interval in hours
> >>>>>>>>> # Set to "0" to disable auto purge feature
> >>>>>>>>> #autopurge.purgeInterval=1
> >>>>>>>>> dataLogDir=/mnt/zk_data_log
> >>>>>>>>> autopurge.snapRetainCount=3
> >>>>>>>>> autopurge.purgeInterval=24
> >>>>>>>>> quorum.auth.enableSasl=true
> >>>>>>>>> quorum.auth.learnerRequireSasl=false
> >>>>>>>>> quorum.auth.serverRequireSasl=false
> >>>>>>>>> quorum.auth.learner.loginContext=QuorumLearner
> >>>>>>>>> quorum.auth.server.loginContext=QuorumServer
> >>>>>>>>> quorum.cnxn.threads.size=20
> >>>>>>>>>
> >>>>
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> >>>>
> >>>>>>>>> secureClientPort=2281
> >>>>>>>>> server.1=0.0.0.0:2888:3888
> >>>>>>>>> server.2=kafkad02.x.azure.com:2888:3888
> >>>>>>>>> server.3=kafkad03.x.azure.com:2888:3888
> >>>>>>>>>
> >>>>>>>>> Server-Log:
> >>>>>>>>> Using config:
> >>>>>>>>> /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
> >>>>>>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic
> >>>>>>>>> Agent:
> >>>>>>>>> Loading configuration file
> >>>>>>> "/opt/zookeeper-cluster/newrelic/./newrelic.yml"
> >>>>>>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: Using
> >>>>>>>>> default
> >>>>>>>>> collector host: collector.newrelic.com
> >>>>>>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic
> >>>>>>>>> Agent:
> >>>>>>>>> Writing to log file:
> >>>>>>>>> /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
> >>>>>>>>> WARNING: An illegal reflective access operation has occurred
> >>>>>>>>> WARNING: Illegal reflective access by
> >>>>>>>>> com.newrelic.weave.weavepackage.NewClassAppender
> >>>>>>>>> (file:/opt/zookeeper-cluster/newrelic/newrelic.jar) to method
> >>>>>>>>> java.net.URLClassLoader.addURL(java.net.URL)
> >>>>>>>>> WARNING: Please consider reporting this to the maintainers of
> >>>>>>>>> com.newrelic.weave.weavepackage.NewClassAppender
> >>>>>>>>> WARNING: Use --illegal-access=warn to enable warnings of further
> >>>>>> illegal
> >>>>>>>>> reflective access operations
> >>>>>>>>> WARNING: All illegal access operations will be denied in a future
> >>>>>>> release
> >>>>>>>>> 2020-02-11 18:43:59,257 [myid:] - INFO
> >>>>>>>>> [main:QuorumPeerConfig@136] -
> >>>>>>>>> Reading configuration from:
> >>>>>>>>> /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
> >>>>>>>>> 2020-02-11 18:43:59,477 [myid:] - INFO
> >>>>>>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
> >>>>>>>>> kafkad02.x.azure.com to address: kafkad02.x.azure.com/1.2.3.4
> >>>>>>>>> 2020-02-11 18:43:59,477 [myid:] - INFO
> >>>>>>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
> >>>>>>>>> 0.0.0.0 to
> >>>>>>>>> address: /0.0.0.0
> >>>>>>>>> 2020-02-11 18:43:59,666 [myid:] - INFO
> >>>>>>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
> >>>>>>>>> kafkad03.x.azure.com to address: kafkad03.x.azure.com/1.2.3.5
> >>>>>>>>> 2020-02-11 18:43:59,666 [myid:] - INFO
> >>>>>>>>> [main:QuorumPeerConfig@398] -
> >>>>>>>>> Defaulting to majority quorums
> >>>>>>>>> 2020-02-11 18:43:59,677 [myid:1] - INFO
> >>>> [main:DatadirCleanupManager@78
> >>>>>> ]
> >>>>>>>>> - autopurge.snapRetainCount set to 3
> >>>>>>>>> 2020-02-11 18:43:59,677 [myid:1] - INFO
> >>>> [main:DatadirCleanupManager@79
> >>>>>> ]
> >>>>>>>>> - autopurge.purgeInterval set to 24
> >>>>>>>>> 2020-02-11 18:43:59,732 [myid:1] - INFO
> >>>>>>>>> [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task
> >>>> started.
> >>>>>>>>> 2020-02-11 18:43:59,749 [myid:1] - INFO
> >>>>>>>>> [main:QuorumPeerMain@130] -
> >>>>>>>>> Starting quorum peer
> >>>>>>>>> 2020-02-11 18:43:59,788 [myid:1] - INFO
> >>>>>>>>> [main:ServerCnxnFactory@117]
> >>>>>> -
> >>>>>>>>> Using org.apache.zookeeper.server.NIOServerCnxnFactory as server
> >>>>>>>>> connection factory
> >>>>>>>>> 2020-02-11 18:43:59,804 [myid:1] - INFO
> >>>>>>>>> [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task
> >>>>>> completed.
> >>>>>>>>> '.20-02-11 18:43:59,826 [myid:1] - ERROR
> >>>>>>>>> [main:ServerCnxnFactory@210
> >>>> ]
> >>>>>> -
> >>>>>>>>> No JAAS configuration section named 'Server' was foundin
> >>>>>>>>> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >>>>>>>>> 2020-02-11 18:43:59,827 [myid:1] - ERROR
> >>>>>>>>> [main:QuorumPeerMain@92] -
> >>>>>>>>> Unexpected exception, exiting abnormally
> >>>>>>>>> java.io.IOException: No JAAS configuration section named
> >>>>>>>>> 'Server' was
> >>>>>>>>> foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >>>>>>>>> '.
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>
> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
>
> >>>>
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>
> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
>
> >>>>
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
>
> >>>>
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
>
> >>>>
> >>>>>>>>> at
> >>>>>>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
>
> >>>>
> >>>>>>>>>
> >>>>>>>>> Best regards
> >>>>>>>>>
> >>>>>>>>> Sebastian
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 12-Feb-20 2:36 AM, Rakesh Radhakrishnan wrote:
> >>>>>>>>>>>>>>> java.io.IOException: No JAAS configuration section named
> >>>>>> 'Server'
> >>>>>>>>>> I could see you have enabled client-server authentication as
> >>>>>>>>>> well.
> >>>> It
> >>>>>>>>>> looks to me that the error is coming from that. Please share the
> >>>>>>>>>> complete error logs to trace it.
> >>>>>>>>>> Have you configured "*Server*" section along with the
> >>>>>> "*QuorumServer*"
> >>>>>>>>>> and "*QuorumClient*" sections? If not, please configure
> >>>>>>>>>> "*Server*"
> >>>>>>>>>> section along with others and try it out.
> >>>>>>>>>>
> >>>>>>>>>> Reference:
> >>>>>>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
> >>>>
> >>>>>>>>>> image.png
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Rakesh
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz
> >>>>>>>>>> <sebastian.schmitz@propellerhead.co.nz
> >>>>>>>>>> <ma...@propellerhead.co.nz>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hello,
> >>>>>>>>>>
> >>>>>>>>>> I'm currently looking into enabling the Auth between
> >>>>>>>>>> Zookeeper-Servers
> >>>>>>>>>> and found this documentation:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
> >>>>
> >>>>>>>>>> However, when I use the config from the document (for
> >>>>>> Digest-MD5)
> >>>>>>>>>> I get
> >>>>>>>>>> this exception in Zookeeper 3.4.14 and also 3.5.6,
> >>>>>>>>>> which I
> >>>> tried
> >>>>>>>>>> because
> >>>>>>>>>> I thought using latest version could help:
> >>>>>>>>>> java.io.IOException: No JAAS configuration section named
> >>>>>> 'Server'
> >>>>>>> was
> >>>>>>>>>> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >>>>>>>>>>
> >>>>>>>>>> And of course that's right, because there's only
> >>>>>>>>>> QuorumServer
> >>>>>> and
> >>>>>>>>>> QuorumClient in the jaas.conf:
> >>>>>>>>>>
> >>>>>>>>>> jaas.conf:
> >>>>>>>>>> QuorumServer {
> >>>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule
> >>>>>>> required
> >>>>>>>>>> user_zookeeper="test";
> >>>>>>>>>> };
> >>>>>>>>>>
> >>>>>>>>>> QuorumClient {
> >>>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule
> >>>>>>> required
> >>>>>>>>>> username="zookeeper"
> >>>>>>>>>> password="test";
> >>>>>>>>>> };
> >>>>>>>>>>
> >>>>>>>>>> I also tried renaming the QuorumServer to just
> >>>>>>>>>> "Server". No
> >>>>>>> change.
> >>>>>>>>>> My zoo.cfg:
> >>>>>>>>>> tickTime=2000
> >>>>>>>>>> initLimit=10
> >>>>>>>>>> syncLimit=5
> >>>>>>>>>> dataDir=/mnt/zk_data
> >>>>>>>>>> clientPort=2181
> >>>>>>>>>> dataLogDir=/mnt/zk_data_log
> >>>>>>>>>> autopurge.snapRetainCount=3
> >>>>>>>>>> autopurge.purgeInterval=24
> >>>>>>>>>> quorum.auth.enableSasl=true
> >>>>>>>>>> quorum.auth.learnerRequireSasl=false
> >>>>>>>>>> quorum.auth.serverRequireSasl=false
> >>>>>>>>>> quorum.auth.learner.loginContext=QuorumLearner
> >>>>>>>>>> quorum.auth.server.loginContext=QuorumServer
> >>>>>>>>>> quorum.cnxn.threads.size=20
> >>>>>>>>>>
> >>>>
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> >>>>
> >>>>>>>>>> �� secureClientPort=2281
> >>>>>>>>>> server.1=0.0.0.0:2888:3888
> >>>>>>>>>>
> >>>>>>>>>> Any idea what I could try? Or maybe there's some better
> >>>> document
> >>>>>>>>>> on how
> >>>>>>>>>> to achieve this?
> >>>>>>>>>>
> >>>>>>>>>> Thank you
> >>>>>>>>>>
> >>>>>>>>>> Sebastian
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> DISCLAIMER
> >>>>>>>>>> This email contains information that is confidential
> >>>>>>>>>> and which
> >>>>>>>>>> may be
> >>>>>>>>>> legally privileged. If you have received this email in
> >>>>>>>>>> error
> >>>>>>> please
> >>>>>>>>>> notify the sender immediately and delete the email.
> >>>>>>>>>> This email is intended
> >>>>>>>>>> solely for the use of the intended recipient and you
> >>>>>>>>>> may not
> >>>> use
> >>>>>>> or
> >>>>>>>>>> disclose this email in any way.
> >>>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> DISCLAIMER
> >>>>>>>>> This email contains information that is confidential and which
> >>>>>>>>> may be
> >>>>>>>>> legally privileged. If you have received this email in error
> >>>>>>>>> please
> >>>>>>>>>
> >>>>>>>>> notify the sender immediately and delete the email.
> >>>>>>>>> This email is intended
> >>>>>>>>> solely for the use of the intended recipient and you may not
> >>>>>>>>> use or
> >>>>>>>>> disclose this email in any way.
> >>>>>>>>>
> >>>>>>> --
> >>>>>>> DISCLAIMER
> >>>>>>> This email contains information that is confidential and which
> >>>>>>> may be
> >>>>>>> legally privileged. If you have received this email in error please
> >>>>>>>
> >>>>>>> notify the sender immediately and delete the email.
> >>>>>>> This email is intended
> >>>>>>> solely for the use of the intended recipient and you may not use or
> >>>>>>> disclose this email in any way.
> >>>>>>>
> >>>> --
> >>>> DISCLAIMER
> >>>> This email contains information that is confidential and which
> >>>> may be
> >>>> legally privileged. If you have received this email in error please
> >>>>
> >>>> notify the sender immediately and delete the email.
> >>>> This email is intended
> >>>> solely for the use of the intended recipient and you may not use or
> >>>> disclose this email in any way.
> >>>>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>
>
Re: Enabling Auth between Zookeeper Servers
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
Hello,
I think I found the issue...
One can't use the same username for clients and quorums. I configured
all of them to be "zookeeper", but in the server-part of the jaas.conf
it should probably be more like "kafka" as it's Kafka which
authenticates to the zookeeper in that case and zookeepers are using the
qorum-part to authenticate to each other. Correct?
If that's correct the exception message is completely wrong. It can find
the file, it can read the file and it even finds the server-part, but
the server-part itself has wrong configuration.
At least with the hanged username in the server-part I got a new exception:
2020-02-17 19:28:17,994 [myid:1] - ERROR [main:ZooKeeperServerMain@83] -
Unexpected exception, exiting abnormally
java.io.IOException: No snapshot found, but there are log entries.
Something is broken!
Which was probably caused by non-cleaned folders of some previous
deployments. So I added the "snapshot.trust.empty=true" to the config to
have it start and rebuild the snapshot. And now my zookeeper is running
just fine! :)
@Mate: as I copied the jaas.conf from your repo is that the exact file
you used for testing? Because changing the "user_zookeeper" to
"user_kafka" in the server-part fixed it.
My next task now is to get Kafka authenticated to zookeeper and get ACLs
working. Will be fun :)
And I should probably create a ticket to get the jaas.conf-error message
fixed!?
Best regards
Sebastian
On 17-Feb-20 1:50 PM, Sebastian Schmitz wrote:
> Hey,
>
> I also just tried using 3.5.7, but same problem...
>
> Best regards
>
> Sebastian
>
>
> On 17-Feb-20 11:34 AM, Sebastian Schmitz wrote:
>> Hi Mate,
>>
>> that's what I also tried. I copied it to the
>> /opt/zookeeper-cluster/-folder and got the same exception just with
>> the new path.
>>
>> So, if that config works on your side it might be my environment
>> then!? Maybe it's a problem with the base-image
>> openjdk:11-jre-stretch which I use for the container... I'll try
>> using the openjdk:8u222-jre you're using.
>>
>> Best regards
>>
>> Sebastian
>>
>>
>> On 17-Feb-20 9:19 AM, Szalay-Bekő Máté wrote:
>>> Hi Sebastian,
>>>
>>> It's strange indeed... I also see the owner is root. That should
>>> work in
>>> docker usually, given that you run the zookeeper process with the root
>>> user. Maybe copying it to a different folder? I see that the conf
>>> folder
>>> has different owner, maybe the java security library doesn't like that?
>>>
>>> But honestly, I don't have any useful explanation.
>>>
>>> Good luck!
>>> Mate
>>>
>>> On Sun, Feb 16, 2020, 20:06 Sebastian Schmitz <
>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>
>>>> Hey Mate,
>>>>
>>>> now it gets really weird. I get the file not found exception:
>>>>
>>>> '.20-02-16 18:27:50,530 [myid:1] - ERROR
>>>> [main:ServerCnxnFactory@246] -
>>>> No JAAS configuration section named 'Server' was found in
>>>> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>> java.lang.SecurityException: java.io.IOException:
>>>> /opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>> (No such file or directory)
>>>> at
>>>> java.base/sun.security.provider.ConfigFile$Spi.<init>(Unknown Source)
>>>> at java.base/sun.security.provider.ConfigFile.<init>(Unknown
>>>> Source)
>>>> at
>>>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>
>>>>
>>>> Method)
>>>> at
>>>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>>>>
>>>>
>>>> Source)
>>>> at
>>>> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>>>>
>>>>
>>>> Source)
>>>> at
>>>> java.base/java.lang.reflect.Constructor.newInstance(Unknown
>>>> Source)
>>>> at java.base/java.lang.Class.newInstance(Unknown Source)
>>>> at
>>>> java.base/javax.security.auth.login.Configuration$2.run(Unknown
>>>> Source)
>>>> at
>>>> java.base/javax.security.auth.login.Configuration$2.run(Unknown
>>>> Source)
>>>> at
>>>> java.base/java.security.AccessController.doPrivileged(Native
>>>> Method)
>>>> at
>>>> java.base/javax.security.auth.login.Configuration.getConfiguration(Unknown
>>>>
>>>> Source)
>>>> at
>>>>
>>>> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
>>>>
>>>> at
>>>>
>>>> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
>>>>
>>>> at
>>>>
>>>> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
>>>>
>>>> at
>>>>
>>>> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
>>>>
>>>> at
>>>>
>>>> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
>>>>
>>>> at
>>>>
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
>>>>
>>>> at
>>>>
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
>>>>
>>>> Caused by: java.io.IOException:
>>>> /opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>> (No such file or directory)
>>>> at
>>>> java.base/sun.security.provider.ConfigFile$Spi.ioException(Unknown
>>>> Source)
>>>> at
>>>> java.base/sun.security.provider.ConfigFile$Spi.init(Unknown
>>>> Source)
>>>> ... 18 more
>>>> 2020-02-16 18:27:50,566 [myid:1] - ERROR
>>>> [main:ZooKeeperServerMain@83] -
>>>> Unexpected exception, exiting abnormally
>>>> java.io.IOException: No JAAS configuration section named 'Server' was
>>>> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>> '.
>>>> at
>>>>
>>>> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:247)
>>>>
>>>> at
>>>>
>>>> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
>>>>
>>>> at
>>>>
>>>> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
>>>>
>>>> at
>>>>
>>>> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
>>>>
>>>> at
>>>>
>>>> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
>>>>
>>>> at
>>>>
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
>>>>
>>>> at
>>>>
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
>>>>
>>>>
>>>> So I checked the image:
>>>>
>>>> root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# ls -la
>>>> total 32
>>>> drwxr-xr-x 2 1010 1011 4096 Feb 16 18:27 .
>>>> drwxr-xr-x 12 root root 4096 Feb 16 18:27 ..
>>>> -rw-r--r-- 1 1010 1011 535 Jan 30 12:18 configuration.xsl
>>>> -rw-r--r-- 1 root root 600 Feb 13 18:32 jaas.conf
>>>> -rw-r--r-- 1 root root 101 Feb 11 00:05 java.env
>>>> -rw-r--r-- 1 1010 1011 2712 Feb 14 05:49 log4j.properties
>>>> -rw-r--r-- 1 root root 1255 Feb 16 18:27 zoo.cfg
>>>> -rw-r--r-- 1 1010 1011 922 Feb 14 05:49 zoo_sample.cfg
>>>>
>>>> And tried to output the file it states in the error:
>>>>
>>>> root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# cat
>>>> /opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>> QuorumServer {
>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>> user_zookeeper="test";
>>>> };
>>>> QuorumClient {
>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>> username="zookeeper"
>>>> password="test";
>>>> };
>>>> Server {
>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>> user_zookeeper="test";
>>>> };
>>>> Client {
>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>> username="zookeeper"
>>>> password="test";
>>>> };
>>>>
>>>> The weird part now is that the access is set exactly the same as the
>>>> zoo.cfg which it can read without problems.
>>>>
>>>> Also changing the access to 666 doesn't change anything. And using
>>>> your
>>>> config doesn't help either:
>>>>
>>>> jaas.conf:
>>>> QuorumServer {
>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>> user_zookeeper="test";
>>>> };
>>>> QuorumLearner {
>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>> username="zookeeper"
>>>> password="test";
>>>> };
>>>> Server {
>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>> user_zookeeper="test";
>>>> };
>>>>
>>>> zoo.cfg:
>>>> tickTime=2000
>>>> initLimit=10
>>>> syncLimit=5
>>>>
>>>> dataDir=/mnt/zk_data
>>>>
>>>> clientPort=2181
>>>>
>>>> standaloneEnabled=true
>>>> admin.enableServer=true
>>>> localSessionsEnabled=true
>>>> localSessionsUpgradingEnabled=true
>>>>
>>>> 4lw.commands.whitelist=stat, ruok, conf, isro, wchc, wchp, srvr,
>>>> mntr, cons
>>>>
>>>> clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
>>>> serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
>>>>
>>>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>>>>
>>>> quorum.auth.enableSasl=true
>>>> quorum.auth.learnerRequireSasl=false
>>>> quorum.auth.serverRequireSasl=false
>>>> quorum.auth.learner.saslLoginContext=QuorumLearner
>>>> quorum.auth.server.saslLoginContext=QuorumServer
>>>> dataLogDir=/mnt/zk_data_log
>>>> autopurge.snapRetainCount=3
>>>> autopurge.purgeInterval=24
>>>> quorum.cnxn.threads.size=20
>>>> server.1=0.0.0.0:2888:3888
>>>>
>>>> I have no idea what's different now. I'll try to run the stuff from
>>>> your
>>>> repo and see if that works.
>>>>
>>>> Best regards
>>>>
>>>> Sebastian
>>>>
>>>> On 14-Feb-20 8:11 PM, Szalay-Bekő Máté wrote:
>>>>> Hi Sebastian!
>>>>>
>>>>> I was able to setup digest authentication, uploaded my results here:
>>>>> https://github.com/symat/zookeeper-docker-test
>>>>> You can see my docker compose file:
>>>>>
>>>> https://github.com/symat/zookeeper-docker-test/blob/master/3_nodes_digest_quorum_auth.yml
>>>>
>>>>> also the zoo.cfg template:
>>>>>
>>>> https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_zoo.cfg
>>>>
>>>>> and the jaas.cfg file:
>>>>>
>>>> https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_jaas.conf
>>>>
>>>>> It works for me, using ZooKeeper 3.5.6. Although I haven't follow
>>>>> your
>>>>> config everywhere.
>>>>>
>>>>> Still, I wasn't able to reproduce your exception, only when I
>>>>> actually
>>>>> deleted the jaas config file. Are you sure that the ZooKeeper
>>>>> process in
>>>>> docker can see / open that file?
>>>>>
>>>>> I created a patched ZooKeeper 3.5.6 for you (you can download from
>>>>> here:
>>>>> https://drive.google.com/open?id=1KEPjNkiKf937jMJHAicwW9WATEuyRZIo),
>>>> where
>>>>> more details are printed in case of errors. E.g. in my case when I
>>>> deleted
>>>>> the jaas config file, I get:
>>>>>
>>>>> zoo1_1 | 2020-02-14 07:04:33,288 [myid:1] - ERROR
>>>>> [main:ServerCnxnFactory@246] - No JAAS configuration section named
>>>> 'Server'
>>>>> was found in '/scripts/conf/digest_jaas.conf'.
>>>>> zoo1_1 | java.lang.SecurityException: java.io.IOException:
>>>>> /scripts/conf/digest_jaas.conf (No such file or directory)
>>>>> zoo1_1 | at
>>>>> sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:137)
>>>>> zoo1_1 | at
>>>>> sun.security.provider.ConfigFile.<init>(ConfigFile.java:102)
>>>>> zoo1_1 | at
>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>>>> zoo1_1 | at
>>>>>
>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>>>>
>>>>> zoo1_1 | at
>>>>>
>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>
>>>>> zoo1_1 | at
>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>>>>> zoo1_1 | at java.lang.Class.newInstance(Class.java:442)
>>>>> zoo1_1 | at
>>>>> javax.security.auth.login.Configuration$2.run(Configuration.java:255)
>>>>> zoo1_1 | at
>>>>> javax.security.auth.login.Configuration$2.run(Configuration.java:247)
>>>>> zoo1_1 | at java.security.AccessController.doPrivileged(Native
>>>>> Method)
>>>>> zoo1_1 | at
>>>>>
>>>> javax.security.auth.login.Configuration.getConfiguration(Configuration.java:246)
>>>>
>>>>> zoo1_1 | at
>>>>>
>>>> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
>>>>
>>>>> zoo1_1 | at
>>>>>
>>>> org.apache.zookeeper.server.NettyServerCnxnFactory.configure(NettyServerCnxnFactory.java:383)
>>>>
>>>>> zoo1_1 | at
>>>>>
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:148)
>>>>
>>>>> zoo1_1 | at
>>>>>
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
>>>>
>>>>> zoo1_1 | at
>>>>>
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
>>>>
>>>>> zoo1_1 | Caused by: java.io.IOException:
>>>>> /scripts/conf/digest_jaas.conf
>>>>> (No such file or directory)
>>>>> zoo1_1 | at
>>>>> sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:666)
>>>>> zoo1_1 | at
>>>>> sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:262)
>>>>> zoo1_1 | at
>>>>> sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:135)
>>>>> zoo1_1 | ... 15 more
>>>>> z
>>>>>
>>>>> Kind regards,
>>>>> Mate
>>>>>
>>>>> On Fri, Feb 14, 2020 at 7:12 AM sagar shukla
>>>>> <sa_shukla@yahoo.com.invalid
>>>>>
>>>>> wrote:
>>>>>
>>>>>> O
>>>>>> Sent from Yahoo Mail on Android
>>>>>>
>>>>>> On Fri, Feb 14, 2020 at 11:02 AM, Szalay-Bekő Máté<
>>>>>> szalay.beko.mate@gmail.com> wrote: Hi Sebastian,
>>>>>>
>>>>>>> But I still get the same exception.
>>>>>> at this point I don't know why this happen... Adding the Server
>>>>>> section
>>>> to
>>>>>> the jaas config should have helped. Unfortunately the exact
>>>>>> exception is
>>>>>> not printed out into the logs, just the error message, so it is
>>>>>> hard to
>>>>>> find out more details.
>>>>>>
>>>>>> I will try to reproduce your case with 3.5.6 locally and see if it
>>>> works. I
>>>>>> never actually used digest authentication before... we always use
>>>> kerberos
>>>>>> in production. If it works, I will share my configs / dockerfiles
>>>>>> and
>>>> send
>>>>>> you a patched version with more debug info printed out.
>>>>>>
>>>>>>> Why would configuring quorum-auth also enable client-server-auth?
>>>>>> it is not very logical indeed... if I see it right, based on the
>>>>>> code
>>>> once
>>>>>> you set the java.security.auth.login.config property, then ZooKeeper
>>>>>> assumes you want to use server-client sasl authentication. I
>>>>>> guess the
>>>>>> quorum-auth feature was added later and they introduced 'enable'
>>>>>> config
>>>>>> property for this, but forget to introduce the same config for the
>>>> client
>>>>>> authentication. I also guess most of the people are interested in
>>>>>> the
>>>>>> client authentication and it is rare that someone does't need
>>>>>> that but
>>>>>> needs quorum auth. Still, the current behaviour is not good I
>>>>>> think. I
>>>> will
>>>>>> submit a jira ticket requesting an improvement here when I will have
>>>> time,
>>>>>> but feel free to submit it yourself if you wish.
>>>>>>
>>>>>> Kind regards,
>>>>>> Mate
>>>>>>
>>>>>> On Thu, Feb 13, 2020 at 7:41 PM Sebastian Schmitz <
>>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>>>>
>>>>>>> Hey Mate,
>>>>>>>
>>>>>>> I checked the java.env-file and it contains:
>>>>>>>
>>>>>>>
>>>>>>>
>>>> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"
>>>>
>>>>>>> which is exactly the place where the pasted jaas.conf is placed.
>>>>>>>
>>>>>>> I also just changed the config to be saslLoginContext and added the
>>>>>>> missing semicolon.
>>>>>>>
>>>>>>> But I still get the same exception.
>>>>>>>
>>>>>>> Why would configuring quorum-auth also enable client-server-auth?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Sebastian
>>>>>>>
>>>>>>>
>>>>>>> On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:
>>>>>>>> Hi Sebastian,
>>>>>>>>
>>>>>>>> thanks for the more details!
>>>>>>>>
>>>>>>>> One thing I found in your config is that you should use:
>>>>>>>> quorum.auth.learner.saslLoginContext=QuorumLearner
>>>>>>>> quorum.auth.server.saslLoginContext=QuorumServer
>>>>>>>>
>>>>>>>> so instead of loginContext, use saslLoginContext in both
>>>>>>>> lines. I
>>>>>>> found
>>>>>>>> this in the source code, I think the wiki is wrong (I will fix it
>>>>>> later).
>>>>>>>> However, actually this didn't really change anything, as the
>>>>>>>> default
>>>>>>> values
>>>>>>>> are anyway
>>>>>>>> QuorumLearner and QuorumServer, so you can even skip these
>>>>>>>> lines from
>>>>>>> the
>>>>>>>> config.
>>>>>>>>
>>>>>>>> I think Rakesh is right, you are seeing exceptions related to
>>>>>>>> not the
>>>>>>>> QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to
>>>>>>>> find
>>>> the
>>>>>>>> 'Server' section (what is configuring the server during the
>>>>>> client-server
>>>>>>>> authentication). The name of this section can be overwritten by
>>>>>>>> the
>>>>>>>> "zookeeper.sasl.serverconfig" system property.
>>>>>>>>
>>>>>>>> Based on the exception, ZooKeeper can not find the 'Server'
>>>>>>>> section in
>>>>>>>> the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are
>>>>>>>> you sure
>>>>>>> this
>>>>>>>> is the correct jaas.conf? Does the ZooKeeper process have the
>>>>>> permissions
>>>>>>>> to open this file? You can specify the jaas config file path for
>>>>>>> ZooKeeper
>>>>>>>> by providing custom system property e.g. by exporting
>>>>>>>> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
>>>>>>>>
>>>>>>>> before starting zkServer.sh
>>>>>>>>
>>>>>>>> Also in the jaas.conf you copied here, you are missing a semicolon
>>>> from
>>>>>>> the
>>>>>>>> end of the last line in the Server block. I am not sure if it is
>>>>>> causing
>>>>>>>> any parsing error, but I always add the semicolon to the end of
>>>>>>>> the
>>>>>> last
>>>>>>>> line in the block.
>>>>>>>>
>>>>>>>> Mate
>>>>>>>>
>>>>>>>> On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
>>>>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>>>>>>
>>>>>>>>> Hello Rakesh,
>>>>>>>>>
>>>>>>>>> as mentioned in the other mail adding the "Server"to jaas.conf
>>>>>>>>> didn't
>>>>>>> help.
>>>>>>>>> Here are the Configs and Logs (with the Server-part included):
>>>>>>>>>
>>>>>>>>> jaas.conf:
>>>>>>>>> QuorumServer {
>>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>>>>>> user_zookeeper="test";
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> QuorumClient {
>>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>>>>>> username="zookeeper"
>>>>>>>>> password="test";
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> Server {
>>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>>>>>> user_zookeeper="test"
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> Client {
>>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>>>>>> username="zookeeper"
>>>>>>>>> password="test";
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> zoo.cfg:
>>>>>>>>> # The number of milliseconds of each tick
>>>>>>>>> tickTime=2000
>>>>>>>>> # The number of ticks that the initial
>>>>>>>>> # synchronization phase can take
>>>>>>>>> initLimit=10
>>>>>>>>> # The number of ticks that can pass between
>>>>>>>>> # sending a request and getting an acknowledgement
>>>>>>>>> syncLimit=5
>>>>>>>>> # the directory where the snapshot is stored.
>>>>>>>>> # do not use /tmp for storage, /tmp here is just
>>>>>>>>> # example sakes.
>>>>>>>>> dataDir=/mnt/zk_data
>>>>>>>>> # the port at which the clients will connect
>>>>>>>>> clientPort=2181
>>>>>>>>> # the maximum number of client connections.
>>>>>>>>> # increase this if you need to handle more clients
>>>>>>>>> #maxClientCnxns=60
>>>>>>>>> #
>>>>>>>>> # Be sure to read the maintenance section of the
>>>>>>>>> # administrator guide before turning on autopurge.
>>>>>>>>> #
>>>>>>>>> #
>>>>>>>>>
>>>> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>>>>
>>>>>>>>> #
>>>>>>>>> # The number of snapshots to retain in dataDir
>>>>>>>>> #autopurge.snapRetainCount=3
>>>>>>>>> # Purge task interval in hours
>>>>>>>>> # Set to "0" to disable auto purge feature
>>>>>>>>> #autopurge.purgeInterval=1
>>>>>>>>> dataLogDir=/mnt/zk_data_log
>>>>>>>>> autopurge.snapRetainCount=3
>>>>>>>>> autopurge.purgeInterval=24
>>>>>>>>> quorum.auth.enableSasl=true
>>>>>>>>> quorum.auth.learnerRequireSasl=false
>>>>>>>>> quorum.auth.serverRequireSasl=false
>>>>>>>>> quorum.auth.learner.loginContext=QuorumLearner
>>>>>>>>> quorum.auth.server.loginContext=QuorumServer
>>>>>>>>> quorum.cnxn.threads.size=20
>>>>>>>>>
>>>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>>>>
>>>>>>>>> secureClientPort=2281
>>>>>>>>> server.1=0.0.0.0:2888:3888
>>>>>>>>> server.2=kafkad02.x.azure.com:2888:3888
>>>>>>>>> server.3=kafkad03.x.azure.com:2888:3888
>>>>>>>>>
>>>>>>>>> Server-Log:
>>>>>>>>> Using config:
>>>>>>>>> /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
>>>>>>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic
>>>>>>>>> Agent:
>>>>>>>>> Loading configuration file
>>>>>>> "/opt/zookeeper-cluster/newrelic/./newrelic.yml"
>>>>>>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: Using
>>>>>>>>> default
>>>>>>>>> collector host: collector.newrelic.com
>>>>>>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic
>>>>>>>>> Agent:
>>>>>>>>> Writing to log file:
>>>>>>>>> /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
>>>>>>>>> WARNING: An illegal reflective access operation has occurred
>>>>>>>>> WARNING: Illegal reflective access by
>>>>>>>>> com.newrelic.weave.weavepackage.NewClassAppender
>>>>>>>>> (file:/opt/zookeeper-cluster/newrelic/newrelic.jar) to method
>>>>>>>>> java.net.URLClassLoader.addURL(java.net.URL)
>>>>>>>>> WARNING: Please consider reporting this to the maintainers of
>>>>>>>>> com.newrelic.weave.weavepackage.NewClassAppender
>>>>>>>>> WARNING: Use --illegal-access=warn to enable warnings of further
>>>>>> illegal
>>>>>>>>> reflective access operations
>>>>>>>>> WARNING: All illegal access operations will be denied in a future
>>>>>>> release
>>>>>>>>> 2020-02-11 18:43:59,257 [myid:] - INFO
>>>>>>>>> [main:QuorumPeerConfig@136] -
>>>>>>>>> Reading configuration from:
>>>>>>>>> /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
>>>>>>>>> 2020-02-11 18:43:59,477 [myid:] - INFO
>>>>>>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
>>>>>>>>> kafkad02.x.azure.com to address: kafkad02.x.azure.com/1.2.3.4
>>>>>>>>> 2020-02-11 18:43:59,477 [myid:] - INFO
>>>>>>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
>>>>>>>>> 0.0.0.0 to
>>>>>>>>> address: /0.0.0.0
>>>>>>>>> 2020-02-11 18:43:59,666 [myid:] - INFO
>>>>>>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
>>>>>>>>> kafkad03.x.azure.com to address: kafkad03.x.azure.com/1.2.3.5
>>>>>>>>> 2020-02-11 18:43:59,666 [myid:] - INFO
>>>>>>>>> [main:QuorumPeerConfig@398] -
>>>>>>>>> Defaulting to majority quorums
>>>>>>>>> 2020-02-11 18:43:59,677 [myid:1] - INFO
>>>> [main:DatadirCleanupManager@78
>>>>>> ]
>>>>>>>>> - autopurge.snapRetainCount set to 3
>>>>>>>>> 2020-02-11 18:43:59,677 [myid:1] - INFO
>>>> [main:DatadirCleanupManager@79
>>>>>> ]
>>>>>>>>> - autopurge.purgeInterval set to 24
>>>>>>>>> 2020-02-11 18:43:59,732 [myid:1] - INFO
>>>>>>>>> [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task
>>>> started.
>>>>>>>>> 2020-02-11 18:43:59,749 [myid:1] - INFO
>>>>>>>>> [main:QuorumPeerMain@130] -
>>>>>>>>> Starting quorum peer
>>>>>>>>> 2020-02-11 18:43:59,788 [myid:1] - INFO
>>>>>>>>> [main:ServerCnxnFactory@117]
>>>>>> -
>>>>>>>>> Using org.apache.zookeeper.server.NIOServerCnxnFactory as server
>>>>>>>>> connection factory
>>>>>>>>> 2020-02-11 18:43:59,804 [myid:1] - INFO
>>>>>>>>> [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task
>>>>>> completed.
>>>>>>>>> '.20-02-11 18:43:59,826 [myid:1] - ERROR
>>>>>>>>> [main:ServerCnxnFactory@210
>>>> ]
>>>>>> -
>>>>>>>>> No JAAS configuration section named 'Server' was foundin
>>>>>>>>> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>>>>>>> 2020-02-11 18:43:59,827 [myid:1] - ERROR
>>>>>>>>> [main:QuorumPeerMain@92] -
>>>>>>>>> Unexpected exception, exiting abnormally
>>>>>>>>> java.io.IOException: No JAAS configuration section named
>>>>>>>>> 'Server' was
>>>>>>>>> foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>>>>>>> '.
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
>>>>
>>>>>>>>> at
>>>>>>>>>
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
>>>>
>>>>>>>>>
>>>>>>>>> Best regards
>>>>>>>>>
>>>>>>>>> Sebastian
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 12-Feb-20 2:36 AM, Rakesh Radhakrishnan wrote:
>>>>>>>>>>>>>>> java.io.IOException: No JAAS configuration section named
>>>>>> 'Server'
>>>>>>>>>> I could see you have enabled client-server authentication as
>>>>>>>>>> well.
>>>> It
>>>>>>>>>> looks to me that the error is coming from that. Please share the
>>>>>>>>>> complete error logs to trace it.
>>>>>>>>>> Have you configured "*Server*" section along with the
>>>>>> "*QuorumServer*"
>>>>>>>>>> and "*QuorumClient*" sections? If not, please configure
>>>>>>>>>> "*Server*"
>>>>>>>>>> section along with others and try it out.
>>>>>>>>>>
>>>>>>>>>> Reference:
>>>>>>>>>>
>>>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
>>>>
>>>>>>>>>> image.png
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Rakesh
>>>>>>>>>>
>>>>>>>>>> On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz
>>>>>>>>>> <sebastian.schmitz@propellerhead.co.nz
>>>>>>>>>> <ma...@propellerhead.co.nz>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I'm currently looking into enabling the Auth between
>>>>>>>>>> Zookeeper-Servers
>>>>>>>>>> and found this documentation:
>>>>>>>>>>
>>>>>>>>>>
>>>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
>>>>
>>>>>>>>>> However, when I use the config from the document (for
>>>>>> Digest-MD5)
>>>>>>>>>> I get
>>>>>>>>>> this exception in Zookeeper 3.4.14 and also 3.5.6,
>>>>>>>>>> which I
>>>> tried
>>>>>>>>>> because
>>>>>>>>>> I thought using latest version could help:
>>>>>>>>>> java.io.IOException: No JAAS configuration section named
>>>>>> 'Server'
>>>>>>> was
>>>>>>>>>> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>>>>>>>>
>>>>>>>>>> And of course that's right, because there's only
>>>>>>>>>> QuorumServer
>>>>>> and
>>>>>>>>>> QuorumClient in the jaas.conf:
>>>>>>>>>>
>>>>>>>>>> jaas.conf:
>>>>>>>>>> QuorumServer {
>>>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule
>>>>>>> required
>>>>>>>>>> user_zookeeper="test";
>>>>>>>>>> };
>>>>>>>>>>
>>>>>>>>>> QuorumClient {
>>>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule
>>>>>>> required
>>>>>>>>>> username="zookeeper"
>>>>>>>>>> password="test";
>>>>>>>>>> };
>>>>>>>>>>
>>>>>>>>>> I also tried renaming the QuorumServer to just
>>>>>>>>>> "Server". No
>>>>>>> change.
>>>>>>>>>> My zoo.cfg:
>>>>>>>>>> tickTime=2000
>>>>>>>>>> initLimit=10
>>>>>>>>>> syncLimit=5
>>>>>>>>>> dataDir=/mnt/zk_data
>>>>>>>>>> clientPort=2181
>>>>>>>>>> dataLogDir=/mnt/zk_data_log
>>>>>>>>>> autopurge.snapRetainCount=3
>>>>>>>>>> autopurge.purgeInterval=24
>>>>>>>>>> quorum.auth.enableSasl=true
>>>>>>>>>> quorum.auth.learnerRequireSasl=false
>>>>>>>>>> quorum.auth.serverRequireSasl=false
>>>>>>>>>> quorum.auth.learner.loginContext=QuorumLearner
>>>>>>>>>> quorum.auth.server.loginContext=QuorumServer
>>>>>>>>>> quorum.cnxn.threads.size=20
>>>>>>>>>>
>>>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>>>>
>>>>>>>>>> �� secureClientPort=2281
>>>>>>>>>> server.1=0.0.0.0:2888:3888
>>>>>>>>>>
>>>>>>>>>> Any idea what I could try? Or maybe there's some better
>>>> document
>>>>>>>>>> on how
>>>>>>>>>> to achieve this?
>>>>>>>>>>
>>>>>>>>>> Thank you
>>>>>>>>>>
>>>>>>>>>> Sebastian
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> DISCLAIMER
>>>>>>>>>> This email contains information that is confidential
>>>>>>>>>> and which
>>>>>>>>>> may be
>>>>>>>>>> legally privileged. If you have received this email in
>>>>>>>>>> error
>>>>>>> please
>>>>>>>>>> notify the sender immediately and delete the email.
>>>>>>>>>> This email is intended
>>>>>>>>>> solely for the use of the intended recipient and you
>>>>>>>>>> may not
>>>> use
>>>>>>> or
>>>>>>>>>> disclose this email in any way.
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> DISCLAIMER
>>>>>>>>> This email contains information that is confidential and which
>>>>>>>>> may be
>>>>>>>>> legally privileged. If you have received this email in error
>>>>>>>>> please
>>>>>>>>>
>>>>>>>>> notify the sender immediately and delete the email.
>>>>>>>>> This email is intended
>>>>>>>>> solely for the use of the intended recipient and you may not
>>>>>>>>> use or
>>>>>>>>> disclose this email in any way.
>>>>>>>>>
>>>>>>> --
>>>>>>> DISCLAIMER
>>>>>>> This email contains information that is confidential and which
>>>>>>> may be
>>>>>>> legally privileged. If you have received this email in error please
>>>>>>>
>>>>>>> notify the sender immediately and delete the email.
>>>>>>> This email is intended
>>>>>>> solely for the use of the intended recipient and you may not use or
>>>>>>> disclose this email in any way.
>>>>>>>
>>>> --
>>>> DISCLAIMER
>>>> This email contains information that is confidential and which
>>>> may be
>>>> legally privileged. If you have received this email in error please
>>>>
>>>> notify the sender immediately and delete the email.
>>>> This email is intended
>>>> solely for the use of the intended recipient and you may not use or
>>>> disclose this email in any way.
>>>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: Enabling Auth between Zookeeper Servers
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
Hey,
I also just tried using 3.5.7, but same problem...
Best regards
Sebastian
On 17-Feb-20 11:34 AM, Sebastian Schmitz wrote:
> Hi Mate,
>
> that's what I also tried. I copied it to the
> /opt/zookeeper-cluster/-folder and got the same exception just with
> the new path.
>
> So, if that config works on your side it might be my environment
> then!? Maybe it's a problem with the base-image openjdk:11-jre-stretch
> which I use for the container... I'll try using the openjdk:8u222-jre
> you're using.
>
> Best regards
>
> Sebastian
>
>
> On 17-Feb-20 9:19 AM, Szalay-Bekő Máté wrote:
>> Hi Sebastian,
>>
>> It's strange indeed... I also see the owner is root. That should work in
>> docker usually, given that you run the zookeeper process with the root
>> user. Maybe copying it to a different folder? I see that the conf folder
>> has different owner, maybe the java security library doesn't like that?
>>
>> But honestly, I don't have any useful explanation.
>>
>> Good luck!
>> Mate
>>
>> On Sun, Feb 16, 2020, 20:06 Sebastian Schmitz <
>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>
>>> Hey Mate,
>>>
>>> now it gets really weird. I get the file not found exception:
>>>
>>> '.20-02-16 18:27:50,530 [myid:1] - ERROR [main:ServerCnxnFactory@246] -
>>> No JAAS configuration section named 'Server' was found in
>>> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>> java.lang.SecurityException: java.io.IOException:
>>> /opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>> (No such file or directory)
>>> at
>>> java.base/sun.security.provider.ConfigFile$Spi.<init>(Unknown Source)
>>> at java.base/sun.security.provider.ConfigFile.<init>(Unknown
>>> Source)
>>> at
>>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>
>>>
>>> Method)
>>> at
>>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>>>
>>>
>>> Source)
>>> at
>>> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>>>
>>>
>>> Source)
>>> at
>>> java.base/java.lang.reflect.Constructor.newInstance(Unknown
>>> Source)
>>> at java.base/java.lang.Class.newInstance(Unknown Source)
>>> at
>>> java.base/javax.security.auth.login.Configuration$2.run(Unknown Source)
>>> at
>>> java.base/javax.security.auth.login.Configuration$2.run(Unknown Source)
>>> at
>>> java.base/java.security.AccessController.doPrivileged(Native
>>> Method)
>>> at
>>> java.base/javax.security.auth.login.Configuration.getConfiguration(Unknown
>>>
>>> Source)
>>> at
>>>
>>> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
>>>
>>> at
>>>
>>> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
>>>
>>> at
>>>
>>> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
>>>
>>> at
>>>
>>> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
>>>
>>> at
>>>
>>> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
>>>
>>> at
>>>
>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
>>>
>>> at
>>>
>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
>>>
>>> Caused by: java.io.IOException:
>>> /opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>> (No such file or directory)
>>> at
>>> java.base/sun.security.provider.ConfigFile$Spi.ioException(Unknown
>>> Source)
>>> at
>>> java.base/sun.security.provider.ConfigFile$Spi.init(Unknown
>>> Source)
>>> ... 18 more
>>> 2020-02-16 18:27:50,566 [myid:1] - ERROR
>>> [main:ZooKeeperServerMain@83] -
>>> Unexpected exception, exiting abnormally
>>> java.io.IOException: No JAAS configuration section named 'Server' was
>>> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>> '.
>>> at
>>>
>>> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:247)
>>>
>>> at
>>>
>>> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
>>>
>>> at
>>>
>>> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
>>>
>>> at
>>>
>>> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
>>>
>>> at
>>>
>>> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
>>>
>>> at
>>>
>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
>>>
>>> at
>>>
>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
>>>
>>>
>>> So I checked the image:
>>>
>>> root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# ls -la
>>> total 32
>>> drwxr-xr-x 2 1010 1011 4096 Feb 16 18:27 .
>>> drwxr-xr-x 12 root root 4096 Feb 16 18:27 ..
>>> -rw-r--r-- 1 1010 1011 535 Jan 30 12:18 configuration.xsl
>>> -rw-r--r-- 1 root root 600 Feb 13 18:32 jaas.conf
>>> -rw-r--r-- 1 root root 101 Feb 11 00:05 java.env
>>> -rw-r--r-- 1 1010 1011 2712 Feb 14 05:49 log4j.properties
>>> -rw-r--r-- 1 root root 1255 Feb 16 18:27 zoo.cfg
>>> -rw-r--r-- 1 1010 1011 922 Feb 14 05:49 zoo_sample.cfg
>>>
>>> And tried to output the file it states in the error:
>>>
>>> root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# cat
>>> /opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>> QuorumServer {
>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>> user_zookeeper="test";
>>> };
>>> QuorumClient {
>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>> username="zookeeper"
>>> password="test";
>>> };
>>> Server {
>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>> user_zookeeper="test";
>>> };
>>> Client {
>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>> username="zookeeper"
>>> password="test";
>>> };
>>>
>>> The weird part now is that the access is set exactly the same as the
>>> zoo.cfg which it can read without problems.
>>>
>>> Also changing the access to 666 doesn't change anything. And using your
>>> config doesn't help either:
>>>
>>> jaas.conf:
>>> QuorumServer {
>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>> user_zookeeper="test";
>>> };
>>> QuorumLearner {
>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>> username="zookeeper"
>>> password="test";
>>> };
>>> Server {
>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>> user_zookeeper="test";
>>> };
>>>
>>> zoo.cfg:
>>> tickTime=2000
>>> initLimit=10
>>> syncLimit=5
>>>
>>> dataDir=/mnt/zk_data
>>>
>>> clientPort=2181
>>>
>>> standaloneEnabled=true
>>> admin.enableServer=true
>>> localSessionsEnabled=true
>>> localSessionsUpgradingEnabled=true
>>>
>>> 4lw.commands.whitelist=stat, ruok, conf, isro, wchc, wchp, srvr,
>>> mntr, cons
>>>
>>> clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
>>> serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
>>>
>>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>>>
>>> quorum.auth.enableSasl=true
>>> quorum.auth.learnerRequireSasl=false
>>> quorum.auth.serverRequireSasl=false
>>> quorum.auth.learner.saslLoginContext=QuorumLearner
>>> quorum.auth.server.saslLoginContext=QuorumServer
>>> dataLogDir=/mnt/zk_data_log
>>> autopurge.snapRetainCount=3
>>> autopurge.purgeInterval=24
>>> quorum.cnxn.threads.size=20
>>> server.1=0.0.0.0:2888:3888
>>>
>>> I have no idea what's different now. I'll try to run the stuff from
>>> your
>>> repo and see if that works.
>>>
>>> Best regards
>>>
>>> Sebastian
>>>
>>> On 14-Feb-20 8:11 PM, Szalay-Bekő Máté wrote:
>>>> Hi Sebastian!
>>>>
>>>> I was able to setup digest authentication, uploaded my results here:
>>>> https://github.com/symat/zookeeper-docker-test
>>>> You can see my docker compose file:
>>>>
>>> https://github.com/symat/zookeeper-docker-test/blob/master/3_nodes_digest_quorum_auth.yml
>>>
>>>> also the zoo.cfg template:
>>>>
>>> https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_zoo.cfg
>>>
>>>> and the jaas.cfg file:
>>>>
>>> https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_jaas.conf
>>>
>>>> It works for me, using ZooKeeper 3.5.6. Although I haven't follow your
>>>> config everywhere.
>>>>
>>>> Still, I wasn't able to reproduce your exception, only when I actually
>>>> deleted the jaas config file. Are you sure that the ZooKeeper
>>>> process in
>>>> docker can see / open that file?
>>>>
>>>> I created a patched ZooKeeper 3.5.6 for you (you can download from
>>>> here:
>>>> https://drive.google.com/open?id=1KEPjNkiKf937jMJHAicwW9WATEuyRZIo),
>>> where
>>>> more details are printed in case of errors. E.g. in my case when I
>>> deleted
>>>> the jaas config file, I get:
>>>>
>>>> zoo1_1 | 2020-02-14 07:04:33,288 [myid:1] - ERROR
>>>> [main:ServerCnxnFactory@246] - No JAAS configuration section named
>>> 'Server'
>>>> was found in '/scripts/conf/digest_jaas.conf'.
>>>> zoo1_1 | java.lang.SecurityException: java.io.IOException:
>>>> /scripts/conf/digest_jaas.conf (No such file or directory)
>>>> zoo1_1 | at
>>>> sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:137)
>>>> zoo1_1 | at
>>>> sun.security.provider.ConfigFile.<init>(ConfigFile.java:102)
>>>> zoo1_1 | at
>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>>> zoo1_1 | at
>>>>
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>>>
>>>> zoo1_1 | at
>>>>
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>
>>>> zoo1_1 | at
>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>>>> zoo1_1 | at java.lang.Class.newInstance(Class.java:442)
>>>> zoo1_1 | at
>>>> javax.security.auth.login.Configuration$2.run(Configuration.java:255)
>>>> zoo1_1 | at
>>>> javax.security.auth.login.Configuration$2.run(Configuration.java:247)
>>>> zoo1_1 | at java.security.AccessController.doPrivileged(Native
>>>> Method)
>>>> zoo1_1 | at
>>>>
>>> javax.security.auth.login.Configuration.getConfiguration(Configuration.java:246)
>>>
>>>> zoo1_1 | at
>>>>
>>> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
>>>
>>>> zoo1_1 | at
>>>>
>>> org.apache.zookeeper.server.NettyServerCnxnFactory.configure(NettyServerCnxnFactory.java:383)
>>>
>>>> zoo1_1 | at
>>>>
>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:148)
>>>
>>>> zoo1_1 | at
>>>>
>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
>>>
>>>> zoo1_1 | at
>>>>
>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
>>>
>>>> zoo1_1 | Caused by: java.io.IOException:
>>>> /scripts/conf/digest_jaas.conf
>>>> (No such file or directory)
>>>> zoo1_1 | at
>>>> sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:666)
>>>> zoo1_1 | at
>>>> sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:262)
>>>> zoo1_1 | at
>>>> sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:135)
>>>> zoo1_1 | ... 15 more
>>>> z
>>>>
>>>> Kind regards,
>>>> Mate
>>>>
>>>> On Fri, Feb 14, 2020 at 7:12 AM sagar shukla
>>>> <sa_shukla@yahoo.com.invalid
>>>>
>>>> wrote:
>>>>
>>>>> O
>>>>> Sent from Yahoo Mail on Android
>>>>>
>>>>> On Fri, Feb 14, 2020 at 11:02 AM, Szalay-Bekő Máté<
>>>>> szalay.beko.mate@gmail.com> wrote: Hi Sebastian,
>>>>>
>>>>>> But I still get the same exception.
>>>>> at this point I don't know why this happen... Adding the Server
>>>>> section
>>> to
>>>>> the jaas config should have helped. Unfortunately the exact
>>>>> exception is
>>>>> not printed out into the logs, just the error message, so it is
>>>>> hard to
>>>>> find out more details.
>>>>>
>>>>> I will try to reproduce your case with 3.5.6 locally and see if it
>>> works. I
>>>>> never actually used digest authentication before... we always use
>>> kerberos
>>>>> in production. If it works, I will share my configs / dockerfiles and
>>> send
>>>>> you a patched version with more debug info printed out.
>>>>>
>>>>>> Why would configuring quorum-auth also enable client-server-auth?
>>>>> it is not very logical indeed... if I see it right, based on the code
>>> once
>>>>> you set the java.security.auth.login.config property, then ZooKeeper
>>>>> assumes you want to use server-client sasl authentication. I guess
>>>>> the
>>>>> quorum-auth feature was added later and they introduced 'enable'
>>>>> config
>>>>> property for this, but forget to introduce the same config for the
>>> client
>>>>> authentication. I also guess most of the people are interested in the
>>>>> client authentication and it is rare that someone does't need that
>>>>> but
>>>>> needs quorum auth. Still, the current behaviour is not good I
>>>>> think. I
>>> will
>>>>> submit a jira ticket requesting an improvement here when I will have
>>> time,
>>>>> but feel free to submit it yourself if you wish.
>>>>>
>>>>> Kind regards,
>>>>> Mate
>>>>>
>>>>> On Thu, Feb 13, 2020 at 7:41 PM Sebastian Schmitz <
>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>>>
>>>>>> Hey Mate,
>>>>>>
>>>>>> I checked the java.env-file and it contains:
>>>>>>
>>>>>>
>>>>>>
>>> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"
>>>
>>>>>> which is exactly the place where the pasted jaas.conf is placed.
>>>>>>
>>>>>> I also just changed the config to be saslLoginContext and added the
>>>>>> missing semicolon.
>>>>>>
>>>>>> But I still get the same exception.
>>>>>>
>>>>>> Why would configuring quorum-auth also enable client-server-auth?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Sebastian
>>>>>>
>>>>>>
>>>>>> On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:
>>>>>>> Hi Sebastian,
>>>>>>>
>>>>>>> thanks for the more details!
>>>>>>>
>>>>>>> One thing I found in your config is that you should use:
>>>>>>> quorum.auth.learner.saslLoginContext=QuorumLearner
>>>>>>> quorum.auth.server.saslLoginContext=QuorumServer
>>>>>>>
>>>>>>> so instead of loginContext, use saslLoginContext in both
>>>>>>> lines. I
>>>>>> found
>>>>>>> this in the source code, I think the wiki is wrong (I will fix it
>>>>> later).
>>>>>>> However, actually this didn't really change anything, as the
>>>>>>> default
>>>>>> values
>>>>>>> are anyway
>>>>>>> QuorumLearner and QuorumServer, so you can even skip these
>>>>>>> lines from
>>>>>> the
>>>>>>> config.
>>>>>>>
>>>>>>> I think Rakesh is right, you are seeing exceptions related to
>>>>>>> not the
>>>>>>> QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find
>>> the
>>>>>>> 'Server' section (what is configuring the server during the
>>>>> client-server
>>>>>>> authentication). The name of this section can be overwritten by the
>>>>>>> "zookeeper.sasl.serverconfig" system property.
>>>>>>>
>>>>>>> Based on the exception, ZooKeeper can not find the 'Server'
>>>>>>> section in
>>>>>>> the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are
>>>>>>> you sure
>>>>>> this
>>>>>>> is the correct jaas.conf? Does the ZooKeeper process have the
>>>>> permissions
>>>>>>> to open this file? You can specify the jaas config file path for
>>>>>> ZooKeeper
>>>>>>> by providing custom system property e.g. by exporting
>>>>>>> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
>>>>>>>
>>>>>>> before starting zkServer.sh
>>>>>>>
>>>>>>> Also in the jaas.conf you copied here, you are missing a semicolon
>>> from
>>>>>> the
>>>>>>> end of the last line in the Server block. I am not sure if it is
>>>>> causing
>>>>>>> any parsing error, but I always add the semicolon to the end of the
>>>>> last
>>>>>>> line in the block.
>>>>>>>
>>>>>>> Mate
>>>>>>>
>>>>>>> On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
>>>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>>>>>
>>>>>>>> Hello Rakesh,
>>>>>>>>
>>>>>>>> as mentioned in the other mail adding the "Server"to jaas.conf
>>>>>>>> didn't
>>>>>> help.
>>>>>>>> Here are the Configs and Logs (with the Server-part included):
>>>>>>>>
>>>>>>>> jaas.conf:
>>>>>>>> QuorumServer {
>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>>>>> user_zookeeper="test";
>>>>>>>> };
>>>>>>>>
>>>>>>>> QuorumClient {
>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>>>>> username="zookeeper"
>>>>>>>> password="test";
>>>>>>>> };
>>>>>>>>
>>>>>>>> Server {
>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>>>>> user_zookeeper="test"
>>>>>>>> };
>>>>>>>>
>>>>>>>> Client {
>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>>>>> username="zookeeper"
>>>>>>>> password="test";
>>>>>>>> };
>>>>>>>>
>>>>>>>> zoo.cfg:
>>>>>>>> # The number of milliseconds of each tick
>>>>>>>> tickTime=2000
>>>>>>>> # The number of ticks that the initial
>>>>>>>> # synchronization phase can take
>>>>>>>> initLimit=10
>>>>>>>> # The number of ticks that can pass between
>>>>>>>> # sending a request and getting an acknowledgement
>>>>>>>> syncLimit=5
>>>>>>>> # the directory where the snapshot is stored.
>>>>>>>> # do not use /tmp for storage, /tmp here is just
>>>>>>>> # example sakes.
>>>>>>>> dataDir=/mnt/zk_data
>>>>>>>> # the port at which the clients will connect
>>>>>>>> clientPort=2181
>>>>>>>> # the maximum number of client connections.
>>>>>>>> # increase this if you need to handle more clients
>>>>>>>> #maxClientCnxns=60
>>>>>>>> #
>>>>>>>> # Be sure to read the maintenance section of the
>>>>>>>> # administrator guide before turning on autopurge.
>>>>>>>> #
>>>>>>>> #
>>>>>>>>
>>> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>>>
>>>>>>>> #
>>>>>>>> # The number of snapshots to retain in dataDir
>>>>>>>> #autopurge.snapRetainCount=3
>>>>>>>> # Purge task interval in hours
>>>>>>>> # Set to "0" to disable auto purge feature
>>>>>>>> #autopurge.purgeInterval=1
>>>>>>>> dataLogDir=/mnt/zk_data_log
>>>>>>>> autopurge.snapRetainCount=3
>>>>>>>> autopurge.purgeInterval=24
>>>>>>>> quorum.auth.enableSasl=true
>>>>>>>> quorum.auth.learnerRequireSasl=false
>>>>>>>> quorum.auth.serverRequireSasl=false
>>>>>>>> quorum.auth.learner.loginContext=QuorumLearner
>>>>>>>> quorum.auth.server.loginContext=QuorumServer
>>>>>>>> quorum.cnxn.threads.size=20
>>>>>>>>
>>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>>>
>>>>>>>> secureClientPort=2281
>>>>>>>> server.1=0.0.0.0:2888:3888
>>>>>>>> server.2=kafkad02.x.azure.com:2888:3888
>>>>>>>> server.3=kafkad03.x.azure.com:2888:3888
>>>>>>>>
>>>>>>>> Server-Log:
>>>>>>>> Using config: /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
>>>>>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic
>>>>>>>> Agent:
>>>>>>>> Loading configuration file
>>>>>> "/opt/zookeeper-cluster/newrelic/./newrelic.yml"
>>>>>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: Using default
>>>>>>>> collector host: collector.newrelic.com
>>>>>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic
>>>>>>>> Agent:
>>>>>>>> Writing to log file:
>>>>>>>> /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
>>>>>>>> WARNING: An illegal reflective access operation has occurred
>>>>>>>> WARNING: Illegal reflective access by
>>>>>>>> com.newrelic.weave.weavepackage.NewClassAppender
>>>>>>>> (file:/opt/zookeeper-cluster/newrelic/newrelic.jar) to method
>>>>>>>> java.net.URLClassLoader.addURL(java.net.URL)
>>>>>>>> WARNING: Please consider reporting this to the maintainers of
>>>>>>>> com.newrelic.weave.weavepackage.NewClassAppender
>>>>>>>> WARNING: Use --illegal-access=warn to enable warnings of further
>>>>> illegal
>>>>>>>> reflective access operations
>>>>>>>> WARNING: All illegal access operations will be denied in a future
>>>>>> release
>>>>>>>> 2020-02-11 18:43:59,257 [myid:] - INFO
>>>>>>>> [main:QuorumPeerConfig@136] -
>>>>>>>> Reading configuration from:
>>>>>>>> /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
>>>>>>>> 2020-02-11 18:43:59,477 [myid:] - INFO
>>>>>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
>>>>>>>> kafkad02.x.azure.com to address: kafkad02.x.azure.com/1.2.3.4
>>>>>>>> 2020-02-11 18:43:59,477 [myid:] - INFO
>>>>>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname: 0.0.0.0 to
>>>>>>>> address: /0.0.0.0
>>>>>>>> 2020-02-11 18:43:59,666 [myid:] - INFO
>>>>>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
>>>>>>>> kafkad03.x.azure.com to address: kafkad03.x.azure.com/1.2.3.5
>>>>>>>> 2020-02-11 18:43:59,666 [myid:] - INFO
>>>>>>>> [main:QuorumPeerConfig@398] -
>>>>>>>> Defaulting to majority quorums
>>>>>>>> 2020-02-11 18:43:59,677 [myid:1] - INFO
>>> [main:DatadirCleanupManager@78
>>>>> ]
>>>>>>>> - autopurge.snapRetainCount set to 3
>>>>>>>> 2020-02-11 18:43:59,677 [myid:1] - INFO
>>> [main:DatadirCleanupManager@79
>>>>> ]
>>>>>>>> - autopurge.purgeInterval set to 24
>>>>>>>> 2020-02-11 18:43:59,732 [myid:1] - INFO
>>>>>>>> [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task
>>> started.
>>>>>>>> 2020-02-11 18:43:59,749 [myid:1] - INFO
>>>>>>>> [main:QuorumPeerMain@130] -
>>>>>>>> Starting quorum peer
>>>>>>>> 2020-02-11 18:43:59,788 [myid:1] - INFO
>>>>>>>> [main:ServerCnxnFactory@117]
>>>>> -
>>>>>>>> Using org.apache.zookeeper.server.NIOServerCnxnFactory as server
>>>>>>>> connection factory
>>>>>>>> 2020-02-11 18:43:59,804 [myid:1] - INFO
>>>>>>>> [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task
>>>>> completed.
>>>>>>>> '.20-02-11 18:43:59,826 [myid:1] - ERROR
>>>>>>>> [main:ServerCnxnFactory@210
>>> ]
>>>>> -
>>>>>>>> No JAAS configuration section named 'Server' was foundin
>>>>>>>> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>>>>>> 2020-02-11 18:43:59,827 [myid:1] - ERROR
>>>>>>>> [main:QuorumPeerMain@92] -
>>>>>>>> Unexpected exception, exiting abnormally
>>>>>>>> java.io.IOException: No JAAS configuration section named
>>>>>>>> 'Server' was
>>>>>>>> foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>>>>>> '.
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
>>>
>>>>>>>> at
>>>>>>>>
>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
>>>
>>>>>>>>
>>>>>>>> Best regards
>>>>>>>>
>>>>>>>> Sebastian
>>>>>>>>
>>>>>>>>
>>>>>>>> On 12-Feb-20 2:36 AM, Rakesh Radhakrishnan wrote:
>>>>>>>>>>>>>> java.io.IOException: No JAAS configuration section named
>>>>> 'Server'
>>>>>>>>> I could see you have enabled client-server authentication as
>>>>>>>>> well.
>>> It
>>>>>>>>> looks to me that the error is coming from that. Please share the
>>>>>>>>> complete error logs to trace it.
>>>>>>>>> Have you configured "*Server*" section along with the
>>>>> "*QuorumServer*"
>>>>>>>>> and "*QuorumClient*" sections? If not, please configure
>>>>>>>>> "*Server*"
>>>>>>>>> section along with others and try it out.
>>>>>>>>>
>>>>>>>>> Reference:
>>>>>>>>>
>>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
>>>
>>>>>>>>> image.png
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Rakesh
>>>>>>>>>
>>>>>>>>> On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz
>>>>>>>>> <sebastian.schmitz@propellerhead.co.nz
>>>>>>>>> <ma...@propellerhead.co.nz>> wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I'm currently looking into enabling the Auth between
>>>>>>>>> Zookeeper-Servers
>>>>>>>>> and found this documentation:
>>>>>>>>>
>>>>>>>>>
>>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
>>>
>>>>>>>>> However, when I use the config from the document (for
>>>>> Digest-MD5)
>>>>>>>>> I get
>>>>>>>>> this exception in Zookeeper 3.4.14 and also 3.5.6, which I
>>> tried
>>>>>>>>> because
>>>>>>>>> I thought using latest version could help:
>>>>>>>>> java.io.IOException: No JAAS configuration section named
>>>>> 'Server'
>>>>>> was
>>>>>>>>> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>>>>>>>
>>>>>>>>> And of course that's right, because there's only
>>>>>>>>> QuorumServer
>>>>> and
>>>>>>>>> QuorumClient in the jaas.conf:
>>>>>>>>>
>>>>>>>>> jaas.conf:
>>>>>>>>> QuorumServer {
>>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule
>>>>>> required
>>>>>>>>> user_zookeeper="test";
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> QuorumClient {
>>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule
>>>>>> required
>>>>>>>>> username="zookeeper"
>>>>>>>>> password="test";
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> I also tried renaming the QuorumServer to just
>>>>>>>>> "Server". No
>>>>>> change.
>>>>>>>>> My zoo.cfg:
>>>>>>>>> tickTime=2000
>>>>>>>>> initLimit=10
>>>>>>>>> syncLimit=5
>>>>>>>>> dataDir=/mnt/zk_data
>>>>>>>>> clientPort=2181
>>>>>>>>> dataLogDir=/mnt/zk_data_log
>>>>>>>>> autopurge.snapRetainCount=3
>>>>>>>>> autopurge.purgeInterval=24
>>>>>>>>> quorum.auth.enableSasl=true
>>>>>>>>> quorum.auth.learnerRequireSasl=false
>>>>>>>>> quorum.auth.serverRequireSasl=false
>>>>>>>>> quorum.auth.learner.loginContext=QuorumLearner
>>>>>>>>> quorum.auth.server.loginContext=QuorumServer
>>>>>>>>> quorum.cnxn.threads.size=20
>>>>>>>>>
>>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>>>
>>>>>>>>> secureClientPort=2281
>>>>>>>>> server.1=0.0.0.0:2888:3888
>>>>>>>>>
>>>>>>>>> Any idea what I could try? Or maybe there's some better
>>> document
>>>>>>>>> on how
>>>>>>>>> to achieve this?
>>>>>>>>>
>>>>>>>>> Thank you
>>>>>>>>>
>>>>>>>>> Sebastian
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> DISCLAIMER
>>>>>>>>> This email contains information that is confidential
>>>>>>>>> and which
>>>>>>>>> may be
>>>>>>>>> legally privileged. If you have received this email in
>>>>>>>>> error
>>>>>> please
>>>>>>>>> notify the sender immediately and delete the email.
>>>>>>>>> This email is intended
>>>>>>>>> solely for the use of the intended recipient and you
>>>>>>>>> may not
>>> use
>>>>>> or
>>>>>>>>> disclose this email in any way.
>>>>>>>>>
>>>>>>>> --
>>>>>>>> DISCLAIMER
>>>>>>>> This email contains information that is confidential and which
>>>>>>>> may be
>>>>>>>> legally privileged. If you have received this email in error
>>>>>>>> please
>>>>>>>>
>>>>>>>> notify the sender immediately and delete the email.
>>>>>>>> This email is intended
>>>>>>>> solely for the use of the intended recipient and you may not
>>>>>>>> use or
>>>>>>>> disclose this email in any way.
>>>>>>>>
>>>>>> --
>>>>>> DISCLAIMER
>>>>>> This email contains information that is confidential and which
>>>>>> may be
>>>>>> legally privileged. If you have received this email in error please
>>>>>>
>>>>>> notify the sender immediately and delete the email.
>>>>>> This email is intended
>>>>>> solely for the use of the intended recipient and you may not use or
>>>>>> disclose this email in any way.
>>>>>>
>>> --
>>> DISCLAIMER
>>> This email contains information that is confidential and which
>>> may be
>>> legally privileged. If you have received this email in error please
>>>
>>> notify the sender immediately and delete the email.
>>> This email is intended
>>> solely for the use of the intended recipient and you may not use or
>>> disclose this email in any way.
>>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: Enabling Auth between Zookeeper Servers
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
Hi Mate,
that's what I also tried. I copied it to the
/opt/zookeeper-cluster/-folder and got the same exception just with the
new path.
So, if that config works on your side it might be my environment then!?
Maybe it's a problem with the base-image openjdk:11-jre-stretch which I
use for the container... I'll try using the openjdk:8u222-jre you're using.
Best regards
Sebastian
On 17-Feb-20 9:19 AM, Szalay-Bekő Máté wrote:
> Hi Sebastian,
>
> It's strange indeed... I also see the owner is root. That should work in
> docker usually, given that you run the zookeeper process with the root
> user. Maybe copying it to a different folder? I see that the conf folder
> has different owner, maybe the java security library doesn't like that?
>
> But honestly, I don't have any useful explanation.
>
> Good luck!
> Mate
>
> On Sun, Feb 16, 2020, 20:06 Sebastian Schmitz <
> sebastian.schmitz@propellerhead.co.nz> wrote:
>
>> Hey Mate,
>>
>> now it gets really weird. I get the file not found exception:
>>
>> '.20-02-16 18:27:50,530 [myid:1] - ERROR [main:ServerCnxnFactory@246] -
>> No JAAS configuration section named 'Server' was found in
>> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>> java.lang.SecurityException: java.io.IOException:
>> /opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>> (No such file or directory)
>> at
>> java.base/sun.security.provider.ConfigFile$Spi.<init>(Unknown Source)
>> at java.base/sun.security.provider.ConfigFile.<init>(Unknown
>> Source)
>> at
>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>
>> Method)
>> at
>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>>
>> Source)
>> at
>> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>>
>> Source)
>> at java.base/java.lang.reflect.Constructor.newInstance(Unknown
>> Source)
>> at java.base/java.lang.Class.newInstance(Unknown Source)
>> at
>> java.base/javax.security.auth.login.Configuration$2.run(Unknown Source)
>> at
>> java.base/javax.security.auth.login.Configuration$2.run(Unknown Source)
>> at java.base/java.security.AccessController.doPrivileged(Native
>> Method)
>> at
>> java.base/javax.security.auth.login.Configuration.getConfiguration(Unknown
>> Source)
>> at
>>
>> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
>> at
>>
>> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
>> at
>>
>> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
>> at
>>
>> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
>> at
>>
>> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
>> at
>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
>> at
>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
>> Caused by: java.io.IOException:
>> /opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>> (No such file or directory)
>> at
>> java.base/sun.security.provider.ConfigFile$Spi.ioException(Unknown Source)
>> at java.base/sun.security.provider.ConfigFile$Spi.init(Unknown
>> Source)
>> ... 18 more
>> 2020-02-16 18:27:50,566 [myid:1] - ERROR [main:ZooKeeperServerMain@83] -
>> Unexpected exception, exiting abnormally
>> java.io.IOException: No JAAS configuration section named 'Server' was
>> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>> '.
>> at
>>
>> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:247)
>> at
>>
>> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
>> at
>>
>> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
>> at
>>
>> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
>> at
>>
>> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
>> at
>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
>> at
>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
>>
>> So I checked the image:
>>
>> root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# ls -la
>> total 32
>> drwxr-xr-x 2 1010 1011 4096 Feb 16 18:27 .
>> drwxr-xr-x 12 root root 4096 Feb 16 18:27 ..
>> -rw-r--r-- 1 1010 1011 535 Jan 30 12:18 configuration.xsl
>> -rw-r--r-- 1 root root 600 Feb 13 18:32 jaas.conf
>> -rw-r--r-- 1 root root 101 Feb 11 00:05 java.env
>> -rw-r--r-- 1 1010 1011 2712 Feb 14 05:49 log4j.properties
>> -rw-r--r-- 1 root root 1255 Feb 16 18:27 zoo.cfg
>> -rw-r--r-- 1 1010 1011 922 Feb 14 05:49 zoo_sample.cfg
>>
>> And tried to output the file it states in the error:
>>
>> root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# cat
>> /opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>> QuorumServer {
>> org.apache.zookeeper.server.auth.DigestLoginModule required
>> user_zookeeper="test";
>> };
>> QuorumClient {
>> org.apache.zookeeper.server.auth.DigestLoginModule required
>> username="zookeeper"
>> password="test";
>> };
>> Server {
>> org.apache.zookeeper.server.auth.DigestLoginModule required
>> user_zookeeper="test";
>> };
>> Client {
>> org.apache.zookeeper.server.auth.DigestLoginModule required
>> username="zookeeper"
>> password="test";
>> };
>>
>> The weird part now is that the access is set exactly the same as the
>> zoo.cfg which it can read without problems.
>>
>> Also changing the access to 666 doesn't change anything. And using your
>> config doesn't help either:
>>
>> jaas.conf:
>> QuorumServer {
>> org.apache.zookeeper.server.auth.DigestLoginModule required
>> user_zookeeper="test";
>> };
>> QuorumLearner {
>> org.apache.zookeeper.server.auth.DigestLoginModule required
>> username="zookeeper"
>> password="test";
>> };
>> Server {
>> org.apache.zookeeper.server.auth.DigestLoginModule required
>> user_zookeeper="test";
>> };
>>
>> zoo.cfg:
>> tickTime=2000
>> initLimit=10
>> syncLimit=5
>>
>> dataDir=/mnt/zk_data
>>
>> clientPort=2181
>>
>> standaloneEnabled=true
>> admin.enableServer=true
>> localSessionsEnabled=true
>> localSessionsUpgradingEnabled=true
>>
>> 4lw.commands.whitelist=stat, ruok, conf, isro, wchc, wchp, srvr, mntr, cons
>>
>> clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
>> serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
>>
>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>> quorum.auth.enableSasl=true
>> quorum.auth.learnerRequireSasl=false
>> quorum.auth.serverRequireSasl=false
>> quorum.auth.learner.saslLoginContext=QuorumLearner
>> quorum.auth.server.saslLoginContext=QuorumServer
>> dataLogDir=/mnt/zk_data_log
>> autopurge.snapRetainCount=3
>> autopurge.purgeInterval=24
>> quorum.cnxn.threads.size=20
>> server.1=0.0.0.0:2888:3888
>>
>> I have no idea what's different now. I'll try to run the stuff from your
>> repo and see if that works.
>>
>> Best regards
>>
>> Sebastian
>>
>> On 14-Feb-20 8:11 PM, Szalay-Bekő Máté wrote:
>>> Hi Sebastian!
>>>
>>> I was able to setup digest authentication, uploaded my results here:
>>> https://github.com/symat/zookeeper-docker-test
>>> You can see my docker compose file:
>>>
>> https://github.com/symat/zookeeper-docker-test/blob/master/3_nodes_digest_quorum_auth.yml
>>> also the zoo.cfg template:
>>>
>> https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_zoo.cfg
>>> and the jaas.cfg file:
>>>
>> https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_jaas.conf
>>> It works for me, using ZooKeeper 3.5.6. Although I haven't follow your
>>> config everywhere.
>>>
>>> Still, I wasn't able to reproduce your exception, only when I actually
>>> deleted the jaas config file. Are you sure that the ZooKeeper process in
>>> docker can see / open that file?
>>>
>>> I created a patched ZooKeeper 3.5.6 for you (you can download from here:
>>> https://drive.google.com/open?id=1KEPjNkiKf937jMJHAicwW9WATEuyRZIo),
>> where
>>> more details are printed in case of errors. E.g. in my case when I
>> deleted
>>> the jaas config file, I get:
>>>
>>> zoo1_1 | 2020-02-14 07:04:33,288 [myid:1] - ERROR
>>> [main:ServerCnxnFactory@246] - No JAAS configuration section named
>> 'Server'
>>> was found in '/scripts/conf/digest_jaas.conf'.
>>> zoo1_1 | java.lang.SecurityException: java.io.IOException:
>>> /scripts/conf/digest_jaas.conf (No such file or directory)
>>> zoo1_1 | at
>>> sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:137)
>>> zoo1_1 | at
>>> sun.security.provider.ConfigFile.<init>(ConfigFile.java:102)
>>> zoo1_1 | at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>> zoo1_1 | at
>>>
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>>> zoo1_1 | at
>>>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>> zoo1_1 | at
>>> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>>> zoo1_1 | at java.lang.Class.newInstance(Class.java:442)
>>> zoo1_1 | at
>>> javax.security.auth.login.Configuration$2.run(Configuration.java:255)
>>> zoo1_1 | at
>>> javax.security.auth.login.Configuration$2.run(Configuration.java:247)
>>> zoo1_1 | at java.security.AccessController.doPrivileged(Native
>>> Method)
>>> zoo1_1 | at
>>>
>> javax.security.auth.login.Configuration.getConfiguration(Configuration.java:246)
>>> zoo1_1 | at
>>>
>> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
>>> zoo1_1 | at
>>>
>> org.apache.zookeeper.server.NettyServerCnxnFactory.configure(NettyServerCnxnFactory.java:383)
>>> zoo1_1 | at
>>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:148)
>>> zoo1_1 | at
>>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
>>> zoo1_1 | at
>>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
>>> zoo1_1 | Caused by: java.io.IOException: /scripts/conf/digest_jaas.conf
>>> (No such file or directory)
>>> zoo1_1 | at
>>> sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:666)
>>> zoo1_1 | at
>>> sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:262)
>>> zoo1_1 | at
>>> sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:135)
>>> zoo1_1 | ... 15 more
>>> z
>>>
>>> Kind regards,
>>> Mate
>>>
>>> On Fri, Feb 14, 2020 at 7:12 AM sagar shukla <sa_shukla@yahoo.com.invalid
>>>
>>> wrote:
>>>
>>>> O
>>>> Sent from Yahoo Mail on Android
>>>>
>>>> On Fri, Feb 14, 2020 at 11:02 AM, Szalay-Bekő Máté<
>>>> szalay.beko.mate@gmail.com> wrote: Hi Sebastian,
>>>>
>>>>> But I still get the same exception.
>>>> at this point I don't know why this happen... Adding the Server section
>> to
>>>> the jaas config should have helped. Unfortunately the exact exception is
>>>> not printed out into the logs, just the error message, so it is hard to
>>>> find out more details.
>>>>
>>>> I will try to reproduce your case with 3.5.6 locally and see if it
>> works. I
>>>> never actually used digest authentication before... we always use
>> kerberos
>>>> in production. If it works, I will share my configs / dockerfiles and
>> send
>>>> you a patched version with more debug info printed out.
>>>>
>>>>> Why would configuring quorum-auth also enable client-server-auth?
>>>> it is not very logical indeed... if I see it right, based on the code
>> once
>>>> you set the java.security.auth.login.config property, then ZooKeeper
>>>> assumes you want to use server-client sasl authentication. I guess the
>>>> quorum-auth feature was added later and they introduced 'enable' config
>>>> property for this, but forget to introduce the same config for the
>> client
>>>> authentication. I also guess most of the people are interested in the
>>>> client authentication and it is rare that someone does't need that but
>>>> needs quorum auth. Still, the current behaviour is not good I think. I
>> will
>>>> submit a jira ticket requesting an improvement here when I will have
>> time,
>>>> but feel free to submit it yourself if you wish.
>>>>
>>>> Kind regards,
>>>> Mate
>>>>
>>>> On Thu, Feb 13, 2020 at 7:41 PM Sebastian Schmitz <
>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>>
>>>>> Hey Mate,
>>>>>
>>>>> I checked the java.env-file and it contains:
>>>>>
>>>>>
>>>>>
>> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"
>>>>> which is exactly the place where the pasted jaas.conf is placed.
>>>>>
>>>>> I also just changed the config to be saslLoginContext and added the
>>>>> missing semicolon.
>>>>>
>>>>> But I still get the same exception.
>>>>>
>>>>> Why would configuring quorum-auth also enable client-server-auth?
>>>>>
>>>>> Thanks
>>>>>
>>>>> Sebastian
>>>>>
>>>>>
>>>>> On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:
>>>>>> Hi Sebastian,
>>>>>>
>>>>>> thanks for the more details!
>>>>>>
>>>>>> One thing I found in your config is that you should use:
>>>>>> quorum.auth.learner.saslLoginContext=QuorumLearner
>>>>>> quorum.auth.server.saslLoginContext=QuorumServer
>>>>>>
>>>>>> so instead of loginContext, use saslLoginContext in both lines. I
>>>>> found
>>>>>> this in the source code, I think the wiki is wrong (I will fix it
>>>> later).
>>>>>> However, actually this didn't really change anything, as the default
>>>>> values
>>>>>> are anyway
>>>>>> QuorumLearner and QuorumServer, so you can even skip these lines from
>>>>> the
>>>>>> config.
>>>>>>
>>>>>> I think Rakesh is right, you are seeing exceptions related to not the
>>>>>> QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find
>> the
>>>>>> 'Server' section (what is configuring the server during the
>>>> client-server
>>>>>> authentication). The name of this section can be overwritten by the
>>>>>> "zookeeper.sasl.serverconfig" system property.
>>>>>>
>>>>>> Based on the exception, ZooKeeper can not find the 'Server' section in
>>>>>> the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are you sure
>>>>> this
>>>>>> is the correct jaas.conf? Does the ZooKeeper process have the
>>>> permissions
>>>>>> to open this file? You can specify the jaas config file path for
>>>>> ZooKeeper
>>>>>> by providing custom system property e.g. by exporting
>>>>>> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
>>>>>> before starting zkServer.sh
>>>>>>
>>>>>> Also in the jaas.conf you copied here, you are missing a semicolon
>> from
>>>>> the
>>>>>> end of the last line in the Server block. I am not sure if it is
>>>> causing
>>>>>> any parsing error, but I always add the semicolon to the end of the
>>>> last
>>>>>> line in the block.
>>>>>>
>>>>>> Mate
>>>>>>
>>>>>> On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
>>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>>>>
>>>>>>> Hello Rakesh,
>>>>>>>
>>>>>>> as mentioned in the other mail adding the "Server"to jaas.conf didn't
>>>>> help.
>>>>>>> Here are the Configs and Logs (with the Server-part included):
>>>>>>>
>>>>>>> jaas.conf:
>>>>>>> QuorumServer {
>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>>>> user_zookeeper="test";
>>>>>>> };
>>>>>>>
>>>>>>> QuorumClient {
>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>>>> username="zookeeper"
>>>>>>> password="test";
>>>>>>> };
>>>>>>>
>>>>>>> Server {
>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>>>> user_zookeeper="test"
>>>>>>> };
>>>>>>>
>>>>>>> Client {
>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>>>> username="zookeeper"
>>>>>>> password="test";
>>>>>>> };
>>>>>>>
>>>>>>> zoo.cfg:
>>>>>>> # The number of milliseconds of each tick
>>>>>>> tickTime=2000
>>>>>>> # The number of ticks that the initial
>>>>>>> # synchronization phase can take
>>>>>>> initLimit=10
>>>>>>> # The number of ticks that can pass between
>>>>>>> # sending a request and getting an acknowledgement
>>>>>>> syncLimit=5
>>>>>>> # the directory where the snapshot is stored.
>>>>>>> # do not use /tmp for storage, /tmp here is just
>>>>>>> # example sakes.
>>>>>>> dataDir=/mnt/zk_data
>>>>>>> # the port at which the clients will connect
>>>>>>> clientPort=2181
>>>>>>> # the maximum number of client connections.
>>>>>>> # increase this if you need to handle more clients
>>>>>>> #maxClientCnxns=60
>>>>>>> #
>>>>>>> # Be sure to read the maintenance section of the
>>>>>>> # administrator guide before turning on autopurge.
>>>>>>> #
>>>>>>> #
>>>>>>>
>> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>>>>>>> #
>>>>>>> # The number of snapshots to retain in dataDir
>>>>>>> #autopurge.snapRetainCount=3
>>>>>>> # Purge task interval in hours
>>>>>>> # Set to "0" to disable auto purge feature
>>>>>>> #autopurge.purgeInterval=1
>>>>>>> dataLogDir=/mnt/zk_data_log
>>>>>>> autopurge.snapRetainCount=3
>>>>>>> autopurge.purgeInterval=24
>>>>>>> quorum.auth.enableSasl=true
>>>>>>> quorum.auth.learnerRequireSasl=false
>>>>>>> quorum.auth.serverRequireSasl=false
>>>>>>> quorum.auth.learner.loginContext=QuorumLearner
>>>>>>> quorum.auth.server.loginContext=QuorumServer
>>>>>>> quorum.cnxn.threads.size=20
>>>>>>>
>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>>>>>>> secureClientPort=2281
>>>>>>> server.1=0.0.0.0:2888:3888
>>>>>>> server.2=kafkad02.x.azure.com:2888:3888
>>>>>>> server.3=kafkad03.x.azure.com:2888:3888
>>>>>>>
>>>>>>> Server-Log:
>>>>>>> Using config: /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
>>>>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
>>>>>>> Loading configuration file
>>>>> "/opt/zookeeper-cluster/newrelic/./newrelic.yml"
>>>>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: Using default
>>>>>>> collector host: collector.newrelic.com
>>>>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
>>>>>>> Writing to log file:
>>>>>>> /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
>>>>>>> WARNING: An illegal reflective access operation has occurred
>>>>>>> WARNING: Illegal reflective access by
>>>>>>> com.newrelic.weave.weavepackage.NewClassAppender
>>>>>>> (file:/opt/zookeeper-cluster/newrelic/newrelic.jar) to method
>>>>>>> java.net.URLClassLoader.addURL(java.net.URL)
>>>>>>> WARNING: Please consider reporting this to the maintainers of
>>>>>>> com.newrelic.weave.weavepackage.NewClassAppender
>>>>>>> WARNING: Use --illegal-access=warn to enable warnings of further
>>>> illegal
>>>>>>> reflective access operations
>>>>>>> WARNING: All illegal access operations will be denied in a future
>>>>> release
>>>>>>> 2020-02-11 18:43:59,257 [myid:] - INFO [main:QuorumPeerConfig@136] -
>>>>>>> Reading configuration from:
>>>>>>> /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
>>>>>>> 2020-02-11 18:43:59,477 [myid:] - INFO
>>>>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
>>>>>>> kafkad02.x.azure.com to address: kafkad02.x.azure.com/1.2.3.4
>>>>>>> 2020-02-11 18:43:59,477 [myid:] - INFO
>>>>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname: 0.0.0.0 to
>>>>>>> address: /0.0.0.0
>>>>>>> 2020-02-11 18:43:59,666 [myid:] - INFO
>>>>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
>>>>>>> kafkad03.x.azure.com to address: kafkad03.x.azure.com/1.2.3.5
>>>>>>> 2020-02-11 18:43:59,666 [myid:] - INFO [main:QuorumPeerConfig@398] -
>>>>>>> Defaulting to majority quorums
>>>>>>> 2020-02-11 18:43:59,677 [myid:1] - INFO
>> [main:DatadirCleanupManager@78
>>>> ]
>>>>>>> - autopurge.snapRetainCount set to 3
>>>>>>> 2020-02-11 18:43:59,677 [myid:1] - INFO
>> [main:DatadirCleanupManager@79
>>>> ]
>>>>>>> - autopurge.purgeInterval set to 24
>>>>>>> 2020-02-11 18:43:59,732 [myid:1] - INFO
>>>>>>> [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task
>> started.
>>>>>>> 2020-02-11 18:43:59,749 [myid:1] - INFO [main:QuorumPeerMain@130] -
>>>>>>> Starting quorum peer
>>>>>>> 2020-02-11 18:43:59,788 [myid:1] - INFO [main:ServerCnxnFactory@117]
>>>> -
>>>>>>> Using org.apache.zookeeper.server.NIOServerCnxnFactory as server
>>>>>>> connection factory
>>>>>>> 2020-02-11 18:43:59,804 [myid:1] - INFO
>>>>>>> [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task
>>>> completed.
>>>>>>> '.20-02-11 18:43:59,826 [myid:1] - ERROR [main:ServerCnxnFactory@210
>> ]
>>>> -
>>>>>>> No JAAS configuration section named 'Server' was foundin
>>>>>>> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>>>>> 2020-02-11 18:43:59,827 [myid:1] - ERROR [main:QuorumPeerMain@92] -
>>>>>>> Unexpected exception, exiting abnormally
>>>>>>> java.io.IOException: No JAAS configuration section named 'Server' was
>>>>>>> foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>>>>> '.
>>>>>>> at
>>>>>>>
>>>>>>>
>> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
>>>>>>> at
>>>>>>>
>>>>>>>
>> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
>>>>>>> at
>>>>>>>
>>>>>>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
>>>>>>> at
>>>>>>>
>>>>>>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
>>>>>>> at
>>>>>>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
>>>>>>>
>>>>>>> Best regards
>>>>>>>
>>>>>>> Sebastian
>>>>>>>
>>>>>>>
>>>>>>> On 12-Feb-20 2:36 AM, Rakesh Radhakrishnan wrote:
>>>>>>>>>>>>> java.io.IOException: No JAAS configuration section named
>>>> 'Server'
>>>>>>>> I could see you have enabled client-server authentication as well.
>> It
>>>>>>>> looks to me that the error is coming from that. Please share the
>>>>>>>> complete error logs to trace it.
>>>>>>>> Have you configured "*Server*" section along with the
>>>> "*QuorumServer*"
>>>>>>>> and "*QuorumClient*" sections? If not, please configure "*Server*"
>>>>>>>> section along with others and try it out.
>>>>>>>>
>>>>>>>> Reference:
>>>>>>>>
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
>>>>>>>> image.png
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Rakesh
>>>>>>>>
>>>>>>>> On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz
>>>>>>>> <sebastian.schmitz@propellerhead.co.nz
>>>>>>>> <ma...@propellerhead.co.nz>> wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I'm currently looking into enabling the Auth between
>>>>>>>> Zookeeper-Servers
>>>>>>>> and found this documentation:
>>>>>>>>
>>>>>>>>
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
>>>>>>>> However, when I use the config from the document (for
>>>> Digest-MD5)
>>>>>>>> I get
>>>>>>>> this exception in Zookeeper 3.4.14 and also 3.5.6, which I
>> tried
>>>>>>>> because
>>>>>>>> I thought using latest version could help:
>>>>>>>> java.io.IOException: No JAAS configuration section named
>>>> 'Server'
>>>>> was
>>>>>>>> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>>>>>>
>>>>>>>> And of course that's right, because there's only QuorumServer
>>>> and
>>>>>>>> QuorumClient in the jaas.conf:
>>>>>>>>
>>>>>>>> jaas.conf:
>>>>>>>> QuorumServer {
>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule
>>>>> required
>>>>>>>> user_zookeeper="test";
>>>>>>>> };
>>>>>>>>
>>>>>>>> QuorumClient {
>>>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule
>>>>> required
>>>>>>>> username="zookeeper"
>>>>>>>> password="test";
>>>>>>>> };
>>>>>>>>
>>>>>>>> I also tried renaming the QuorumServer to just "Server". No
>>>>> change.
>>>>>>>> My zoo.cfg:
>>>>>>>> tickTime=2000
>>>>>>>> initLimit=10
>>>>>>>> syncLimit=5
>>>>>>>> dataDir=/mnt/zk_data
>>>>>>>> clientPort=2181
>>>>>>>> dataLogDir=/mnt/zk_data_log
>>>>>>>> autopurge.snapRetainCount=3
>>>>>>>> autopurge.purgeInterval=24
>>>>>>>> quorum.auth.enableSasl=true
>>>>>>>> quorum.auth.learnerRequireSasl=false
>>>>>>>> quorum.auth.serverRequireSasl=false
>>>>>>>> quorum.auth.learner.loginContext=QuorumLearner
>>>>>>>> quorum.auth.server.loginContext=QuorumServer
>>>>>>>> quorum.cnxn.threads.size=20
>>>>>>>>
>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>>>>>>>> secureClientPort=2281
>>>>>>>> server.1=0.0.0.0:2888:3888
>>>>>>>>
>>>>>>>> Any idea what I could try? Or maybe there's some better
>> document
>>>>>>>> on how
>>>>>>>> to achieve this?
>>>>>>>>
>>>>>>>> Thank you
>>>>>>>>
>>>>>>>> Sebastian
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> DISCLAIMER
>>>>>>>> This email contains information that is confidential and which
>>>>>>>> may be
>>>>>>>> legally privileged. If you have received this email in error
>>>>> please
>>>>>>>> notify the sender immediately and delete the email.
>>>>>>>> This email is intended
>>>>>>>> solely for the use of the intended recipient and you may not
>> use
>>>>> or
>>>>>>>> disclose this email in any way.
>>>>>>>>
>>>>>>> --
>>>>>>> DISCLAIMER
>>>>>>> This email contains information that is confidential and which
>>>>>>> may be
>>>>>>> legally privileged. If you have received this email in error please
>>>>>>>
>>>>>>> notify the sender immediately and delete the email.
>>>>>>> This email is intended
>>>>>>> solely for the use of the intended recipient and you may not use or
>>>>>>> disclose this email in any way.
>>>>>>>
>>>>> --
>>>>> DISCLAIMER
>>>>> This email contains information that is confidential and which
>>>>> may be
>>>>> legally privileged. If you have received this email in error please
>>>>>
>>>>> notify the sender immediately and delete the email.
>>>>> This email is intended
>>>>> solely for the use of the intended recipient and you may not use or
>>>>> disclose this email in any way.
>>>>>
>> --
>> DISCLAIMER
>> This email contains information that is confidential and which
>> may be
>> legally privileged. If you have received this email in error please
>>
>> notify the sender immediately and delete the email.
>> This email is intended
>> solely for the use of the intended recipient and you may not use or
>> disclose this email in any way.
>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: Enabling Auth between Zookeeper Servers
Posted by Szalay-Bekő Máté <sz...@gmail.com>.
Hi Sebastian,
It's strange indeed... I also see the owner is root. That should work in
docker usually, given that you run the zookeeper process with the root
user. Maybe copying it to a different folder? I see that the conf folder
has different owner, maybe the java security library doesn't like that?
But honestly, I don't have any useful explanation.
Good luck!
Mate
On Sun, Feb 16, 2020, 20:06 Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> Hey Mate,
>
> now it gets really weird. I get the file not found exception:
>
> '.20-02-16 18:27:50,530 [myid:1] - ERROR [main:ServerCnxnFactory@246] -
> No JAAS configuration section named 'Server' was found in
> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> java.lang.SecurityException: java.io.IOException:
> /opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> (No such file or directory)
> at
> java.base/sun.security.provider.ConfigFile$Spi.<init>(Unknown Source)
> at java.base/sun.security.provider.ConfigFile.<init>(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>
> Method)
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>
> Source)
> at java.base/java.lang.reflect.Constructor.newInstance(Unknown
> Source)
> at java.base/java.lang.Class.newInstance(Unknown Source)
> at
> java.base/javax.security.auth.login.Configuration$2.run(Unknown Source)
> at
> java.base/javax.security.auth.login.Configuration$2.run(Unknown Source)
> at java.base/java.security.AccessController.doPrivileged(Native
> Method)
> at
> java.base/javax.security.auth.login.Configuration.getConfiguration(Unknown
> Source)
> at
>
> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
> at
>
> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
> at
>
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
> at
>
> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
> at
>
> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
> Caused by: java.io.IOException:
> /opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> (No such file or directory)
> at
> java.base/sun.security.provider.ConfigFile$Spi.ioException(Unknown Source)
> at java.base/sun.security.provider.ConfigFile$Spi.init(Unknown
> Source)
> ... 18 more
> 2020-02-16 18:27:50,566 [myid:1] - ERROR [main:ZooKeeperServerMain@83] -
> Unexpected exception, exiting abnormally
> java.io.IOException: No JAAS configuration section named 'Server' was
> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> '.
> at
>
> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:247)
> at
>
> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
> at
>
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
> at
>
> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
> at
>
> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
>
> So I checked the image:
>
> root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# ls -la
> total 32
> drwxr-xr-x 2 1010 1011 4096 Feb 16 18:27 .
> drwxr-xr-x 12 root root 4096 Feb 16 18:27 ..
> -rw-r--r-- 1 1010 1011 535 Jan 30 12:18 configuration.xsl
> -rw-r--r-- 1 root root 600 Feb 13 18:32 jaas.conf
> -rw-r--r-- 1 root root 101 Feb 11 00:05 java.env
> -rw-r--r-- 1 1010 1011 2712 Feb 14 05:49 log4j.properties
> -rw-r--r-- 1 root root 1255 Feb 16 18:27 zoo.cfg
> -rw-r--r-- 1 1010 1011 922 Feb 14 05:49 zoo_sample.cfg
>
> And tried to output the file it states in the error:
>
> root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# cat
> /opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> QuorumServer {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> user_zookeeper="test";
> };
> QuorumClient {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> username="zookeeper"
> password="test";
> };
> Server {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> user_zookeeper="test";
> };
> Client {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> username="zookeeper"
> password="test";
> };
>
> The weird part now is that the access is set exactly the same as the
> zoo.cfg which it can read without problems.
>
> Also changing the access to 666 doesn't change anything. And using your
> config doesn't help either:
>
> jaas.conf:
> QuorumServer {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> user_zookeeper="test";
> };
> QuorumLearner {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> username="zookeeper"
> password="test";
> };
> Server {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> user_zookeeper="test";
> };
>
> zoo.cfg:
> tickTime=2000
> initLimit=10
> syncLimit=5
>
> dataDir=/mnt/zk_data
>
> clientPort=2181
>
> standaloneEnabled=true
> admin.enableServer=true
> localSessionsEnabled=true
> localSessionsUpgradingEnabled=true
>
> 4lw.commands.whitelist=stat, ruok, conf, isro, wchc, wchp, srvr, mntr, cons
>
> clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
> serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
>
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> quorum.auth.enableSasl=true
> quorum.auth.learnerRequireSasl=false
> quorum.auth.serverRequireSasl=false
> quorum.auth.learner.saslLoginContext=QuorumLearner
> quorum.auth.server.saslLoginContext=QuorumServer
> dataLogDir=/mnt/zk_data_log
> autopurge.snapRetainCount=3
> autopurge.purgeInterval=24
> quorum.cnxn.threads.size=20
> server.1=0.0.0.0:2888:3888
>
> I have no idea what's different now. I'll try to run the stuff from your
> repo and see if that works.
>
> Best regards
>
> Sebastian
>
> On 14-Feb-20 8:11 PM, Szalay-Bekő Máté wrote:
> > Hi Sebastian!
> >
> > I was able to setup digest authentication, uploaded my results here:
> > https://github.com/symat/zookeeper-docker-test
> > You can see my docker compose file:
> >
> https://github.com/symat/zookeeper-docker-test/blob/master/3_nodes_digest_quorum_auth.yml
> > also the zoo.cfg template:
> >
> https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_zoo.cfg
> > and the jaas.cfg file:
> >
> https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_jaas.conf
> >
> > It works for me, using ZooKeeper 3.5.6. Although I haven't follow your
> > config everywhere.
> >
> > Still, I wasn't able to reproduce your exception, only when I actually
> > deleted the jaas config file. Are you sure that the ZooKeeper process in
> > docker can see / open that file?
> >
> > I created a patched ZooKeeper 3.5.6 for you (you can download from here:
> > https://drive.google.com/open?id=1KEPjNkiKf937jMJHAicwW9WATEuyRZIo),
> where
> > more details are printed in case of errors. E.g. in my case when I
> deleted
> > the jaas config file, I get:
> >
> > zoo1_1 | 2020-02-14 07:04:33,288 [myid:1] - ERROR
> > [main:ServerCnxnFactory@246] - No JAAS configuration section named
> 'Server'
> > was found in '/scripts/conf/digest_jaas.conf'.
> > zoo1_1 | java.lang.SecurityException: java.io.IOException:
> > /scripts/conf/digest_jaas.conf (No such file or directory)
> > zoo1_1 | at
> > sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:137)
> > zoo1_1 | at
> > sun.security.provider.ConfigFile.<init>(ConfigFile.java:102)
> > zoo1_1 | at
> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> > zoo1_1 | at
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> > zoo1_1 | at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > zoo1_1 | at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> > zoo1_1 | at java.lang.Class.newInstance(Class.java:442)
> > zoo1_1 | at
> > javax.security.auth.login.Configuration$2.run(Configuration.java:255)
> > zoo1_1 | at
> > javax.security.auth.login.Configuration$2.run(Configuration.java:247)
> > zoo1_1 | at java.security.AccessController.doPrivileged(Native
> > Method)
> > zoo1_1 | at
> >
> javax.security.auth.login.Configuration.getConfiguration(Configuration.java:246)
> > zoo1_1 | at
> >
> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
> > zoo1_1 | at
> >
> org.apache.zookeeper.server.NettyServerCnxnFactory.configure(NettyServerCnxnFactory.java:383)
> > zoo1_1 | at
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:148)
> > zoo1_1 | at
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
> > zoo1_1 | at
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
> > zoo1_1 | Caused by: java.io.IOException: /scripts/conf/digest_jaas.conf
> > (No such file or directory)
> > zoo1_1 | at
> > sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:666)
> > zoo1_1 | at
> > sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:262)
> > zoo1_1 | at
> > sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:135)
> > zoo1_1 | ... 15 more
> > z
> >
> > Kind regards,
> > Mate
> >
> > On Fri, Feb 14, 2020 at 7:12 AM sagar shukla <sa_shukla@yahoo.com.invalid
> >
> > wrote:
> >
> >> O
> >> Sent from Yahoo Mail on Android
> >>
> >> On Fri, Feb 14, 2020 at 11:02 AM, Szalay-Bekő Máté<
> >> szalay.beko.mate@gmail.com> wrote: Hi Sebastian,
> >>
> >>> But I still get the same exception.
> >> at this point I don't know why this happen... Adding the Server section
> to
> >> the jaas config should have helped. Unfortunately the exact exception is
> >> not printed out into the logs, just the error message, so it is hard to
> >> find out more details.
> >>
> >> I will try to reproduce your case with 3.5.6 locally and see if it
> works. I
> >> never actually used digest authentication before... we always use
> kerberos
> >> in production. If it works, I will share my configs / dockerfiles and
> send
> >> you a patched version with more debug info printed out.
> >>
> >>> Why would configuring quorum-auth also enable client-server-auth?
> >> it is not very logical indeed... if I see it right, based on the code
> once
> >> you set the java.security.auth.login.config property, then ZooKeeper
> >> assumes you want to use server-client sasl authentication. I guess the
> >> quorum-auth feature was added later and they introduced 'enable' config
> >> property for this, but forget to introduce the same config for the
> client
> >> authentication. I also guess most of the people are interested in the
> >> client authentication and it is rare that someone does't need that but
> >> needs quorum auth. Still, the current behaviour is not good I think. I
> will
> >> submit a jira ticket requesting an improvement here when I will have
> time,
> >> but feel free to submit it yourself if you wish.
> >>
> >> Kind regards,
> >> Mate
> >>
> >> On Thu, Feb 13, 2020 at 7:41 PM Sebastian Schmitz <
> >> sebastian.schmitz@propellerhead.co.nz> wrote:
> >>
> >>> Hey Mate,
> >>>
> >>> I checked the java.env-file and it contains:
> >>>
> >>>
> >>>
> >>
> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"
> >>> which is exactly the place where the pasted jaas.conf is placed.
> >>>
> >>> I also just changed the config to be saslLoginContext and added the
> >>> missing semicolon.
> >>>
> >>> But I still get the same exception.
> >>>
> >>> Why would configuring quorum-auth also enable client-server-auth?
> >>>
> >>> Thanks
> >>>
> >>> Sebastian
> >>>
> >>>
> >>> On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:
> >>>> Hi Sebastian,
> >>>>
> >>>> thanks for the more details!
> >>>>
> >>>> One thing I found in your config is that you should use:
> >>>> quorum.auth.learner.saslLoginContext=QuorumLearner
> >>>> quorum.auth.server.saslLoginContext=QuorumServer
> >>>>
> >>>> so instead of loginContext, use saslLoginContext in both lines. I
> >>> found
> >>>> this in the source code, I think the wiki is wrong (I will fix it
> >> later).
> >>>> However, actually this didn't really change anything, as the default
> >>> values
> >>>> are anyway
> >>>> QuorumLearner and QuorumServer, so you can even skip these lines from
> >>> the
> >>>> config.
> >>>>
> >>>> I think Rakesh is right, you are seeing exceptions related to not the
> >>>> QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find
> the
> >>>> 'Server' section (what is configuring the server during the
> >> client-server
> >>>> authentication). The name of this section can be overwritten by the
> >>>> "zookeeper.sasl.serverconfig" system property.
> >>>>
> >>>> Based on the exception, ZooKeeper can not find the 'Server' section in
> >>>> the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are you sure
> >>> this
> >>>> is the correct jaas.conf? Does the ZooKeeper process have the
> >> permissions
> >>>> to open this file? You can specify the jaas config file path for
> >>> ZooKeeper
> >>>> by providing custom system property e.g. by exporting
> >>>> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
> >>>> before starting zkServer.sh
> >>>>
> >>>> Also in the jaas.conf you copied here, you are missing a semicolon
> from
> >>> the
> >>>> end of the last line in the Server block. I am not sure if it is
> >> causing
> >>>> any parsing error, but I always add the semicolon to the end of the
> >> last
> >>>> line in the block.
> >>>>
> >>>> Mate
> >>>>
> >>>> On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
> >>>> sebastian.schmitz@propellerhead.co.nz> wrote:
> >>>>
> >>>>> Hello Rakesh,
> >>>>>
> >>>>> as mentioned in the other mail adding the "Server"to jaas.conf didn't
> >>> help.
> >>>>> Here are the Configs and Logs (with the Server-part included):
> >>>>>
> >>>>> jaas.conf:
> >>>>> QuorumServer {
> >>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
> >>>>> user_zookeeper="test";
> >>>>> };
> >>>>>
> >>>>> QuorumClient {
> >>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
> >>>>> username="zookeeper"
> >>>>> password="test";
> >>>>> };
> >>>>>
> >>>>> Server {
> >>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
> >>>>> user_zookeeper="test"
> >>>>> };
> >>>>>
> >>>>> Client {
> >>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
> >>>>> username="zookeeper"
> >>>>> password="test";
> >>>>> };
> >>>>>
> >>>>> zoo.cfg:
> >>>>> # The number of milliseconds of each tick
> >>>>> tickTime=2000
> >>>>> # The number of ticks that the initial
> >>>>> # synchronization phase can take
> >>>>> initLimit=10
> >>>>> # The number of ticks that can pass between
> >>>>> # sending a request and getting an acknowledgement
> >>>>> syncLimit=5
> >>>>> # the directory where the snapshot is stored.
> >>>>> # do not use /tmp for storage, /tmp here is just
> >>>>> # example sakes.
> >>>>> dataDir=/mnt/zk_data
> >>>>> # the port at which the clients will connect
> >>>>> clientPort=2181
> >>>>> # the maximum number of client connections.
> >>>>> # increase this if you need to handle more clients
> >>>>> #maxClientCnxns=60
> >>>>> #
> >>>>> # Be sure to read the maintenance section of the
> >>>>> # administrator guide before turning on autopurge.
> >>>>> #
> >>>>> #
> >>>>>
> >>
> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
> >>>>> #
> >>>>> # The number of snapshots to retain in dataDir
> >>>>> #autopurge.snapRetainCount=3
> >>>>> # Purge task interval in hours
> >>>>> # Set to "0" to disable auto purge feature
> >>>>> #autopurge.purgeInterval=1
> >>>>> dataLogDir=/mnt/zk_data_log
> >>>>> autopurge.snapRetainCount=3
> >>>>> autopurge.purgeInterval=24
> >>>>> quorum.auth.enableSasl=true
> >>>>> quorum.auth.learnerRequireSasl=false
> >>>>> quorum.auth.serverRequireSasl=false
> >>>>> quorum.auth.learner.loginContext=QuorumLearner
> >>>>> quorum.auth.server.loginContext=QuorumServer
> >>>>> quorum.cnxn.threads.size=20
> >>>>>
> >>
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> >>>>> secureClientPort=2281
> >>>>> server.1=0.0.0.0:2888:3888
> >>>>> server.2=kafkad02.x.azure.com:2888:3888
> >>>>> server.3=kafkad03.x.azure.com:2888:3888
> >>>>>
> >>>>> Server-Log:
> >>>>> Using config: /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
> >>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
> >>>>> Loading configuration file
> >>> "/opt/zookeeper-cluster/newrelic/./newrelic.yml"
> >>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: Using default
> >>>>> collector host: collector.newrelic.com
> >>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
> >>>>> Writing to log file:
> >>>>> /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
> >>>>> WARNING: An illegal reflective access operation has occurred
> >>>>> WARNING: Illegal reflective access by
> >>>>> com.newrelic.weave.weavepackage.NewClassAppender
> >>>>> (file:/opt/zookeeper-cluster/newrelic/newrelic.jar) to method
> >>>>> java.net.URLClassLoader.addURL(java.net.URL)
> >>>>> WARNING: Please consider reporting this to the maintainers of
> >>>>> com.newrelic.weave.weavepackage.NewClassAppender
> >>>>> WARNING: Use --illegal-access=warn to enable warnings of further
> >> illegal
> >>>>> reflective access operations
> >>>>> WARNING: All illegal access operations will be denied in a future
> >>> release
> >>>>> 2020-02-11 18:43:59,257 [myid:] - INFO [main:QuorumPeerConfig@136] -
> >>>>> Reading configuration from:
> >>>>> /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
> >>>>> 2020-02-11 18:43:59,477 [myid:] - INFO
> >>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
> >>>>> kafkad02.x.azure.com to address: kafkad02.x.azure.com/1.2.3.4
> >>>>> 2020-02-11 18:43:59,477 [myid:] - INFO
> >>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname: 0.0.0.0 to
> >>>>> address: /0.0.0.0
> >>>>> 2020-02-11 18:43:59,666 [myid:] - INFO
> >>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
> >>>>> kafkad03.x.azure.com to address: kafkad03.x.azure.com/1.2.3.5
> >>>>> 2020-02-11 18:43:59,666 [myid:] - INFO [main:QuorumPeerConfig@398] -
> >>>>> Defaulting to majority quorums
> >>>>> 2020-02-11 18:43:59,677 [myid:1] - INFO
> [main:DatadirCleanupManager@78
> >> ]
> >>>>> - autopurge.snapRetainCount set to 3
> >>>>> 2020-02-11 18:43:59,677 [myid:1] - INFO
> [main:DatadirCleanupManager@79
> >> ]
> >>>>> - autopurge.purgeInterval set to 24
> >>>>> 2020-02-11 18:43:59,732 [myid:1] - INFO
> >>>>> [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task
> started.
> >>>>> 2020-02-11 18:43:59,749 [myid:1] - INFO [main:QuorumPeerMain@130] -
> >>>>> Starting quorum peer
> >>>>> 2020-02-11 18:43:59,788 [myid:1] - INFO [main:ServerCnxnFactory@117]
> >> -
> >>>>> Using org.apache.zookeeper.server.NIOServerCnxnFactory as server
> >>>>> connection factory
> >>>>> 2020-02-11 18:43:59,804 [myid:1] - INFO
> >>>>> [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task
> >> completed.
> >>>>> '.20-02-11 18:43:59,826 [myid:1] - ERROR [main:ServerCnxnFactory@210
> ]
> >> -
> >>>>> No JAAS configuration section named 'Server' was foundin
> >>>>> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >>>>> 2020-02-11 18:43:59,827 [myid:1] - ERROR [main:QuorumPeerMain@92] -
> >>>>> Unexpected exception, exiting abnormally
> >>>>> java.io.IOException: No JAAS configuration section named 'Server' was
> >>>>> foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >>>>> '.
> >>>>> at
> >>>>>
> >>>>>
> >>
> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
> >>>>> at
> >>>>>
> >>>>>
> >>
> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
> >>>>> at
> >>>>>
> >>>>>
> >>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
> >>>>> at
> >>>>>
> >>>>>
> >>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
> >>>>> at
> >>>>>
> >>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
> >>>>>
> >>>>>
> >>>>> Best regards
> >>>>>
> >>>>> Sebastian
> >>>>>
> >>>>>
> >>>>> On 12-Feb-20 2:36 AM, Rakesh Radhakrishnan wrote:
> >>>>>>>>>>> java.io.IOException: No JAAS configuration section named
> >> 'Server'
> >>>>>> I could see you have enabled client-server authentication as well.
> It
> >>>>>> looks to me that the error is coming from that. Please share the
> >>>>>> complete error logs to trace it.
> >>>>>> Have you configured "*Server*" section along with the
> >> "*QuorumServer*"
> >>>>>> and "*QuorumClient*" sections? If not, please configure "*Server*"
> >>>>>> section along with others and try it out.
> >>>>>>
> >>>>>> Reference:
> >>>>>>
> >>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
> >>>>>> image.png
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Rakesh
> >>>>>>
> >>>>>> On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz
> >>>>>> <sebastian.schmitz@propellerhead.co.nz
> >>>>>> <ma...@propellerhead.co.nz>> wrote:
> >>>>>>
> >>>>>> Hello,
> >>>>>>
> >>>>>> I'm currently looking into enabling the Auth between
> >>>>>> Zookeeper-Servers
> >>>>>> and found this documentation:
> >>>>>>
> >>>>>>
> >>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
> >>>>>> However, when I use the config from the document (for
> >> Digest-MD5)
> >>>>>> I get
> >>>>>> this exception in Zookeeper 3.4.14 and also 3.5.6, which I
> tried
> >>>>>> because
> >>>>>> I thought using latest version could help:
> >>>>>> java.io.IOException: No JAAS configuration section named
> >> 'Server'
> >>> was
> >>>>>> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >>>>>>
> >>>>>> And of course that's right, because there's only QuorumServer
> >> and
> >>>>>> QuorumClient in the jaas.conf:
> >>>>>>
> >>>>>> jaas.conf:
> >>>>>> QuorumServer {
> >>>>>> org.apache.zookeeper.server.auth.DigestLoginModule
> >>> required
> >>>>>> user_zookeeper="test";
> >>>>>> };
> >>>>>>
> >>>>>> QuorumClient {
> >>>>>> org.apache.zookeeper.server.auth.DigestLoginModule
> >>> required
> >>>>>> username="zookeeper"
> >>>>>> password="test";
> >>>>>> };
> >>>>>>
> >>>>>> I also tried renaming the QuorumServer to just "Server". No
> >>> change.
> >>>>>> My zoo.cfg:
> >>>>>> tickTime=2000
> >>>>>> initLimit=10
> >>>>>> syncLimit=5
> >>>>>> dataDir=/mnt/zk_data
> >>>>>> clientPort=2181
> >>>>>> dataLogDir=/mnt/zk_data_log
> >>>>>> autopurge.snapRetainCount=3
> >>>>>> autopurge.purgeInterval=24
> >>>>>> quorum.auth.enableSasl=true
> >>>>>> quorum.auth.learnerRequireSasl=false
> >>>>>> quorum.auth.serverRequireSasl=false
> >>>>>> quorum.auth.learner.loginContext=QuorumLearner
> >>>>>> quorum.auth.server.loginContext=QuorumServer
> >>>>>> quorum.cnxn.threads.size=20
> >>>>>>
> >>
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> >>>>>> secureClientPort=2281
> >>>>>> server.1=0.0.0.0:2888:3888
> >>>>>>
> >>>>>> Any idea what I could try? Or maybe there's some better
> document
> >>>>>> on how
> >>>>>> to achieve this?
> >>>>>>
> >>>>>> Thank you
> >>>>>>
> >>>>>> Sebastian
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> DISCLAIMER
> >>>>>> This email contains information that is confidential and which
> >>>>>> may be
> >>>>>> legally privileged. If you have received this email in error
> >>> please
> >>>>>> notify the sender immediately and delete the email.
> >>>>>> This email is intended
> >>>>>> solely for the use of the intended recipient and you may not
> use
> >>> or
> >>>>>> disclose this email in any way.
> >>>>>>
> >>>>> --
> >>>>> DISCLAIMER
> >>>>> This email contains information that is confidential and which
> >>>>> may be
> >>>>> legally privileged. If you have received this email in error please
> >>>>>
> >>>>> notify the sender immediately and delete the email.
> >>>>> This email is intended
> >>>>> solely for the use of the intended recipient and you may not use or
> >>>>> disclose this email in any way.
> >>>>>
> >>> --
> >>> DISCLAIMER
> >>> This email contains information that is confidential and which
> >>> may be
> >>> legally privileged. If you have received this email in error please
> >>>
> >>> notify the sender immediately and delete the email.
> >>> This email is intended
> >>> solely for the use of the intended recipient and you may not use or
> >>> disclose this email in any way.
> >>>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>
Re: Enabling Auth between Zookeeper Servers
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
Hey Mate,
now it gets really weird. I get the file not found exception:
'.20-02-16 18:27:50,530 [myid:1] - ERROR [main:ServerCnxnFactory@246] -
No JAAS configuration section named 'Server' was found in
'/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
java.lang.SecurityException: java.io.IOException:
/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
(No such file or directory)
at
java.base/sun.security.provider.ConfigFile$Spi.<init>(Unknown Source)
at java.base/sun.security.provider.ConfigFile.<init>(Unknown
Source)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.base/java.lang.reflect.Constructor.newInstance(Unknown
Source)
at java.base/java.lang.Class.newInstance(Unknown Source)
at
java.base/javax.security.auth.login.Configuration$2.run(Unknown Source)
at
java.base/javax.security.auth.login.Configuration$2.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Native
Method)
at
java.base/javax.security.auth.login.Configuration.getConfiguration(Unknown
Source)
at
org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
at
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
at
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
Caused by: java.io.IOException:
/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
(No such file or directory)
at
java.base/sun.security.provider.ConfigFile$Spi.ioException(Unknown Source)
at java.base/sun.security.provider.ConfigFile$Spi.init(Unknown
Source)
... 18 more
2020-02-16 18:27:50,566 [myid:1] - ERROR [main:ZooKeeperServerMain@83] -
Unexpected exception, exiting abnormally
java.io.IOException: No JAAS configuration section named 'Server' was
found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
'.
at
org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:247)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
at
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
at
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
So I checked the image:
root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# ls -la
total 32
drwxr-xr-x 2 1010 1011 4096 Feb 16 18:27 .
drwxr-xr-x 12 root root 4096 Feb 16 18:27 ..
-rw-r--r-- 1 1010 1011 535 Jan 30 12:18 configuration.xsl
-rw-r--r-- 1 root root 600 Feb 13 18:32 jaas.conf
-rw-r--r-- 1 root root 101 Feb 11 00:05 java.env
-rw-r--r-- 1 1010 1011 2712 Feb 14 05:49 log4j.properties
-rw-r--r-- 1 root root 1255 Feb 16 18:27 zoo.cfg
-rw-r--r-- 1 1010 1011 922 Feb 14 05:49 zoo_sample.cfg
And tried to output the file it states in the error:
root@2eeeb625500c:/opt/zookeeper-cluster/zookeeper/conf# cat
/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
QuorumServer {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test";
};
QuorumClient {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zookeeper"
password="test";
};
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test";
};
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zookeeper"
password="test";
};
The weird part now is that the access is set exactly the same as the
zoo.cfg which it can read without problems.
Also changing the access to 666 doesn't change anything. And using your
config doesn't help either:
jaas.conf:
QuorumServer {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test";
};
QuorumLearner {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zookeeper"
password="test";
};
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test";
};
zoo.cfg:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/mnt/zk_data
clientPort=2181
standaloneEnabled=true
admin.enableServer=true
localSessionsEnabled=true
localSessionsUpgradingEnabled=true
4lw.commands.whitelist=stat, ruok, conf, isro, wchc, wchp, srvr, mntr, cons
clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=false
quorum.auth.serverRequireSasl=false
quorum.auth.learner.saslLoginContext=QuorumLearner
quorum.auth.server.saslLoginContext=QuorumServer
dataLogDir=/mnt/zk_data_log
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
quorum.cnxn.threads.size=20
server.1=0.0.0.0:2888:3888
I have no idea what's different now. I'll try to run the stuff from your
repo and see if that works.
Best regards
Sebastian
On 14-Feb-20 8:11 PM, Szalay-Bekő Máté wrote:
> Hi Sebastian!
>
> I was able to setup digest authentication, uploaded my results here:
> https://github.com/symat/zookeeper-docker-test
> You can see my docker compose file:
> https://github.com/symat/zookeeper-docker-test/blob/master/3_nodes_digest_quorum_auth.yml
> also the zoo.cfg template:
> https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_zoo.cfg
> and the jaas.cfg file:
> https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_jaas.conf
>
> It works for me, using ZooKeeper 3.5.6. Although I haven't follow your
> config everywhere.
>
> Still, I wasn't able to reproduce your exception, only when I actually
> deleted the jaas config file. Are you sure that the ZooKeeper process in
> docker can see / open that file?
>
> I created a patched ZooKeeper 3.5.6 for you (you can download from here:
> https://drive.google.com/open?id=1KEPjNkiKf937jMJHAicwW9WATEuyRZIo), where
> more details are printed in case of errors. E.g. in my case when I deleted
> the jaas config file, I get:
>
> zoo1_1 | 2020-02-14 07:04:33,288 [myid:1] - ERROR
> [main:ServerCnxnFactory@246] - No JAAS configuration section named 'Server'
> was found in '/scripts/conf/digest_jaas.conf'.
> zoo1_1 | java.lang.SecurityException: java.io.IOException:
> /scripts/conf/digest_jaas.conf (No such file or directory)
> zoo1_1 | at
> sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:137)
> zoo1_1 | at
> sun.security.provider.ConfigFile.<init>(ConfigFile.java:102)
> zoo1_1 | at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> zoo1_1 | at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> zoo1_1 | at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> zoo1_1 | at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> zoo1_1 | at java.lang.Class.newInstance(Class.java:442)
> zoo1_1 | at
> javax.security.auth.login.Configuration$2.run(Configuration.java:255)
> zoo1_1 | at
> javax.security.auth.login.Configuration$2.run(Configuration.java:247)
> zoo1_1 | at java.security.AccessController.doPrivileged(Native
> Method)
> zoo1_1 | at
> javax.security.auth.login.Configuration.getConfiguration(Configuration.java:246)
> zoo1_1 | at
> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
> zoo1_1 | at
> org.apache.zookeeper.server.NettyServerCnxnFactory.configure(NettyServerCnxnFactory.java:383)
> zoo1_1 | at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:148)
> zoo1_1 | at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
> zoo1_1 | at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
> zoo1_1 | Caused by: java.io.IOException: /scripts/conf/digest_jaas.conf
> (No such file or directory)
> zoo1_1 | at
> sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:666)
> zoo1_1 | at
> sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:262)
> zoo1_1 | at
> sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:135)
> zoo1_1 | ... 15 more
> z
>
> Kind regards,
> Mate
>
> On Fri, Feb 14, 2020 at 7:12 AM sagar shukla <sa...@yahoo.com.invalid>
> wrote:
>
>> O
>> Sent from Yahoo Mail on Android
>>
>> On Fri, Feb 14, 2020 at 11:02 AM, Szalay-Bekő Máté<
>> szalay.beko.mate@gmail.com> wrote: Hi Sebastian,
>>
>>> But I still get the same exception.
>> at this point I don't know why this happen... Adding the Server section to
>> the jaas config should have helped. Unfortunately the exact exception is
>> not printed out into the logs, just the error message, so it is hard to
>> find out more details.
>>
>> I will try to reproduce your case with 3.5.6 locally and see if it works. I
>> never actually used digest authentication before... we always use kerberos
>> in production. If it works, I will share my configs / dockerfiles and send
>> you a patched version with more debug info printed out.
>>
>>> Why would configuring quorum-auth also enable client-server-auth?
>> it is not very logical indeed... if I see it right, based on the code once
>> you set the java.security.auth.login.config property, then ZooKeeper
>> assumes you want to use server-client sasl authentication. I guess the
>> quorum-auth feature was added later and they introduced 'enable' config
>> property for this, but forget to introduce the same config for the client
>> authentication. I also guess most of the people are interested in the
>> client authentication and it is rare that someone does't need that but
>> needs quorum auth. Still, the current behaviour is not good I think. I will
>> submit a jira ticket requesting an improvement here when I will have time,
>> but feel free to submit it yourself if you wish.
>>
>> Kind regards,
>> Mate
>>
>> On Thu, Feb 13, 2020 at 7:41 PM Sebastian Schmitz <
>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>
>>> Hey Mate,
>>>
>>> I checked the java.env-file and it contains:
>>>
>>>
>>>
>> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"
>>> which is exactly the place where the pasted jaas.conf is placed.
>>>
>>> I also just changed the config to be saslLoginContext and added the
>>> missing semicolon.
>>>
>>> But I still get the same exception.
>>>
>>> Why would configuring quorum-auth also enable client-server-auth?
>>>
>>> Thanks
>>>
>>> Sebastian
>>>
>>>
>>> On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:
>>>> Hi Sebastian,
>>>>
>>>> thanks for the more details!
>>>>
>>>> One thing I found in your config is that you should use:
>>>> quorum.auth.learner.saslLoginContext=QuorumLearner
>>>> quorum.auth.server.saslLoginContext=QuorumServer
>>>>
>>>> so instead of loginContext, use saslLoginContext in both lines. I
>>> found
>>>> this in the source code, I think the wiki is wrong (I will fix it
>> later).
>>>> However, actually this didn't really change anything, as the default
>>> values
>>>> are anyway
>>>> QuorumLearner and QuorumServer, so you can even skip these lines from
>>> the
>>>> config.
>>>>
>>>> I think Rakesh is right, you are seeing exceptions related to not the
>>>> QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find the
>>>> 'Server' section (what is configuring the server during the
>> client-server
>>>> authentication). The name of this section can be overwritten by the
>>>> "zookeeper.sasl.serverconfig" system property.
>>>>
>>>> Based on the exception, ZooKeeper can not find the 'Server' section in
>>>> the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are you sure
>>> this
>>>> is the correct jaas.conf? Does the ZooKeeper process have the
>> permissions
>>>> to open this file? You can specify the jaas config file path for
>>> ZooKeeper
>>>> by providing custom system property e.g. by exporting
>>>> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
>>>> before starting zkServer.sh
>>>>
>>>> Also in the jaas.conf you copied here, you are missing a semicolon from
>>> the
>>>> end of the last line in the Server block. I am not sure if it is
>> causing
>>>> any parsing error, but I always add the semicolon to the end of the
>> last
>>>> line in the block.
>>>>
>>>> Mate
>>>>
>>>> On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>>
>>>>> Hello Rakesh,
>>>>>
>>>>> as mentioned in the other mail adding the "Server"to jaas.conf didn't
>>> help.
>>>>> Here are the Configs and Logs (with the Server-part included):
>>>>>
>>>>> jaas.conf:
>>>>> QuorumServer {
>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>> user_zookeeper="test";
>>>>> };
>>>>>
>>>>> QuorumClient {
>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>> username="zookeeper"
>>>>> password="test";
>>>>> };
>>>>>
>>>>> Server {
>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>> user_zookeeper="test"
>>>>> };
>>>>>
>>>>> Client {
>>>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>>>> username="zookeeper"
>>>>> password="test";
>>>>> };
>>>>>
>>>>> zoo.cfg:
>>>>> # The number of milliseconds of each tick
>>>>> tickTime=2000
>>>>> # The number of ticks that the initial
>>>>> # synchronization phase can take
>>>>> initLimit=10
>>>>> # The number of ticks that can pass between
>>>>> # sending a request and getting an acknowledgement
>>>>> syncLimit=5
>>>>> # the directory where the snapshot is stored.
>>>>> # do not use /tmp for storage, /tmp here is just
>>>>> # example sakes.
>>>>> dataDir=/mnt/zk_data
>>>>> # the port at which the clients will connect
>>>>> clientPort=2181
>>>>> # the maximum number of client connections.
>>>>> # increase this if you need to handle more clients
>>>>> #maxClientCnxns=60
>>>>> #
>>>>> # Be sure to read the maintenance section of the
>>>>> # administrator guide before turning on autopurge.
>>>>> #
>>>>> #
>>>>>
>> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>>>>> #
>>>>> # The number of snapshots to retain in dataDir
>>>>> #autopurge.snapRetainCount=3
>>>>> # Purge task interval in hours
>>>>> # Set to "0" to disable auto purge feature
>>>>> #autopurge.purgeInterval=1
>>>>> dataLogDir=/mnt/zk_data_log
>>>>> autopurge.snapRetainCount=3
>>>>> autopurge.purgeInterval=24
>>>>> quorum.auth.enableSasl=true
>>>>> quorum.auth.learnerRequireSasl=false
>>>>> quorum.auth.serverRequireSasl=false
>>>>> quorum.auth.learner.loginContext=QuorumLearner
>>>>> quorum.auth.server.loginContext=QuorumServer
>>>>> quorum.cnxn.threads.size=20
>>>>>
>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>>>>> secureClientPort=2281
>>>>> server.1=0.0.0.0:2888:3888
>>>>> server.2=kafkad02.x.azure.com:2888:3888
>>>>> server.3=kafkad03.x.azure.com:2888:3888
>>>>>
>>>>> Server-Log:
>>>>> Using config: /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
>>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
>>>>> Loading configuration file
>>> "/opt/zookeeper-cluster/newrelic/./newrelic.yml"
>>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: Using default
>>>>> collector host: collector.newrelic.com
>>>>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
>>>>> Writing to log file:
>>>>> /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
>>>>> WARNING: An illegal reflective access operation has occurred
>>>>> WARNING: Illegal reflective access by
>>>>> com.newrelic.weave.weavepackage.NewClassAppender
>>>>> (file:/opt/zookeeper-cluster/newrelic/newrelic.jar) to method
>>>>> java.net.URLClassLoader.addURL(java.net.URL)
>>>>> WARNING: Please consider reporting this to the maintainers of
>>>>> com.newrelic.weave.weavepackage.NewClassAppender
>>>>> WARNING: Use --illegal-access=warn to enable warnings of further
>> illegal
>>>>> reflective access operations
>>>>> WARNING: All illegal access operations will be denied in a future
>>> release
>>>>> 2020-02-11 18:43:59,257 [myid:] - INFO [main:QuorumPeerConfig@136] -
>>>>> Reading configuration from:
>>>>> /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
>>>>> 2020-02-11 18:43:59,477 [myid:] - INFO
>>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
>>>>> kafkad02.x.azure.com to address: kafkad02.x.azure.com/1.2.3.4
>>>>> 2020-02-11 18:43:59,477 [myid:] - INFO
>>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname: 0.0.0.0 to
>>>>> address: /0.0.0.0
>>>>> 2020-02-11 18:43:59,666 [myid:] - INFO
>>>>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
>>>>> kafkad03.x.azure.com to address: kafkad03.x.azure.com/1.2.3.5
>>>>> 2020-02-11 18:43:59,666 [myid:] - INFO [main:QuorumPeerConfig@398] -
>>>>> Defaulting to majority quorums
>>>>> 2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@78
>> ]
>>>>> - autopurge.snapRetainCount set to 3
>>>>> 2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@79
>> ]
>>>>> - autopurge.purgeInterval set to 24
>>>>> 2020-02-11 18:43:59,732 [myid:1] - INFO
>>>>> [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
>>>>> 2020-02-11 18:43:59,749 [myid:1] - INFO [main:QuorumPeerMain@130] -
>>>>> Starting quorum peer
>>>>> 2020-02-11 18:43:59,788 [myid:1] - INFO [main:ServerCnxnFactory@117]
>> -
>>>>> Using org.apache.zookeeper.server.NIOServerCnxnFactory as server
>>>>> connection factory
>>>>> 2020-02-11 18:43:59,804 [myid:1] - INFO
>>>>> [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task
>> completed.
>>>>> '.20-02-11 18:43:59,826 [myid:1] - ERROR [main:ServerCnxnFactory@210]
>> -
>>>>> No JAAS configuration section named 'Server' was foundin
>>>>> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>>> 2020-02-11 18:43:59,827 [myid:1] - ERROR [main:QuorumPeerMain@92] -
>>>>> Unexpected exception, exiting abnormally
>>>>> java.io.IOException: No JAAS configuration section named 'Server' was
>>>>> foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>>> '.
>>>>> at
>>>>>
>>>>>
>> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
>>>>> at
>>>>>
>>>>>
>> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
>>>>> at
>>>>>
>>>>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
>>>>> at
>>>>>
>>>>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
>>>>> at
>>>>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
>>>>>
>>>>>
>>>>> Best regards
>>>>>
>>>>> Sebastian
>>>>>
>>>>>
>>>>> On 12-Feb-20 2:36 AM, Rakesh Radhakrishnan wrote:
>>>>>>>>>>> java.io.IOException: No JAAS configuration section named
>> 'Server'
>>>>>> I could see you have enabled client-server authentication as well. It
>>>>>> looks to me that the error is coming from that. Please share the
>>>>>> complete error logs to trace it.
>>>>>> Have you configured "*Server*" section along with the
>> "*QuorumServer*"
>>>>>> and "*QuorumClient*" sections? If not, please configure "*Server*"
>>>>>> section along with others and try it out.
>>>>>>
>>>>>> Reference:
>>>>>>
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
>>>>>> image.png
>>>>>>
>>>>>> Thanks,
>>>>>> Rakesh
>>>>>>
>>>>>> On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz
>>>>>> <sebastian.schmitz@propellerhead.co.nz
>>>>>> <ma...@propellerhead.co.nz>> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I'm currently looking into enabling the Auth between
>>>>>> Zookeeper-Servers
>>>>>> and found this documentation:
>>>>>>
>>>>>>
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
>>>>>> However, when I use the config from the document (for
>> Digest-MD5)
>>>>>> I get
>>>>>> this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried
>>>>>> because
>>>>>> I thought using latest version could help:
>>>>>> java.io.IOException: No JAAS configuration section named
>> 'Server'
>>> was
>>>>>> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>>>>
>>>>>> And of course that's right, because there's only QuorumServer
>> and
>>>>>> QuorumClient in the jaas.conf:
>>>>>>
>>>>>> jaas.conf:
>>>>>> QuorumServer {
>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule
>>> required
>>>>>> user_zookeeper="test";
>>>>>> };
>>>>>>
>>>>>> QuorumClient {
>>>>>> org.apache.zookeeper.server.auth.DigestLoginModule
>>> required
>>>>>> username="zookeeper"
>>>>>> password="test";
>>>>>> };
>>>>>>
>>>>>> I also tried renaming the QuorumServer to just "Server". No
>>> change.
>>>>>> My zoo.cfg:
>>>>>> tickTime=2000
>>>>>> initLimit=10
>>>>>> syncLimit=5
>>>>>> dataDir=/mnt/zk_data
>>>>>> clientPort=2181
>>>>>> dataLogDir=/mnt/zk_data_log
>>>>>> autopurge.snapRetainCount=3
>>>>>> autopurge.purgeInterval=24
>>>>>> quorum.auth.enableSasl=true
>>>>>> quorum.auth.learnerRequireSasl=false
>>>>>> quorum.auth.serverRequireSasl=false
>>>>>> quorum.auth.learner.loginContext=QuorumLearner
>>>>>> quorum.auth.server.loginContext=QuorumServer
>>>>>> quorum.cnxn.threads.size=20
>>>>>>
>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>>>>>> secureClientPort=2281
>>>>>> server.1=0.0.0.0:2888:3888
>>>>>>
>>>>>> Any idea what I could try? Or maybe there's some better document
>>>>>> on how
>>>>>> to achieve this?
>>>>>>
>>>>>> Thank you
>>>>>>
>>>>>> Sebastian
>>>>>>
>>>>>>
>>>>>> --
>>>>>> DISCLAIMER
>>>>>> This email contains information that is confidential and which
>>>>>> may be
>>>>>> legally privileged. If you have received this email in error
>>> please
>>>>>> notify the sender immediately and delete the email.
>>>>>> This email is intended
>>>>>> solely for the use of the intended recipient and you may not use
>>> or
>>>>>> disclose this email in any way.
>>>>>>
>>>>> --
>>>>> DISCLAIMER
>>>>> This email contains information that is confidential and which
>>>>> may be
>>>>> legally privileged. If you have received this email in error please
>>>>>
>>>>> notify the sender immediately and delete the email.
>>>>> This email is intended
>>>>> solely for the use of the intended recipient and you may not use or
>>>>> disclose this email in any way.
>>>>>
>>> --
>>> DISCLAIMER
>>> This email contains information that is confidential and which
>>> may be
>>> legally privileged. If you have received this email in error please
>>>
>>> notify the sender immediately and delete the email.
>>> This email is intended
>>> solely for the use of the intended recipient and you may not use or
>>> disclose this email in any way.
>>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: Enabling Auth between Zookeeper Servers
Posted by Szalay-Bekő Máté <sz...@gmail.com>.
Hi Sebastian!
I was able to setup digest authentication, uploaded my results here:
https://github.com/symat/zookeeper-docker-test
You can see my docker compose file:
https://github.com/symat/zookeeper-docker-test/blob/master/3_nodes_digest_quorum_auth.yml
also the zoo.cfg template:
https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_zoo.cfg
and the jaas.cfg file:
https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_jaas.conf
It works for me, using ZooKeeper 3.5.6. Although I haven't follow your
config everywhere.
Still, I wasn't able to reproduce your exception, only when I actually
deleted the jaas config file. Are you sure that the ZooKeeper process in
docker can see / open that file?
I created a patched ZooKeeper 3.5.6 for you (you can download from here:
https://drive.google.com/open?id=1KEPjNkiKf937jMJHAicwW9WATEuyRZIo), where
more details are printed in case of errors. E.g. in my case when I deleted
the jaas config file, I get:
zoo1_1 | 2020-02-14 07:04:33,288 [myid:1] - ERROR
[main:ServerCnxnFactory@246] - No JAAS configuration section named 'Server'
was found in '/scripts/conf/digest_jaas.conf'.
zoo1_1 | java.lang.SecurityException: java.io.IOException:
/scripts/conf/digest_jaas.conf (No such file or directory)
zoo1_1 | at
sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:137)
zoo1_1 | at
sun.security.provider.ConfigFile.<init>(ConfigFile.java:102)
zoo1_1 | at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
zoo1_1 | at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
zoo1_1 | at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
zoo1_1 | at
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
zoo1_1 | at java.lang.Class.newInstance(Class.java:442)
zoo1_1 | at
javax.security.auth.login.Configuration$2.run(Configuration.java:255)
zoo1_1 | at
javax.security.auth.login.Configuration$2.run(Configuration.java:247)
zoo1_1 | at java.security.AccessController.doPrivileged(Native
Method)
zoo1_1 | at
javax.security.auth.login.Configuration.getConfiguration(Configuration.java:246)
zoo1_1 | at
org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:210)
zoo1_1 | at
org.apache.zookeeper.server.NettyServerCnxnFactory.configure(NettyServerCnxnFactory.java:383)
zoo1_1 | at
org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:148)
zoo1_1 | at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
zoo1_1 | at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
zoo1_1 | Caused by: java.io.IOException: /scripts/conf/digest_jaas.conf
(No such file or directory)
zoo1_1 | at
sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:666)
zoo1_1 | at
sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:262)
zoo1_1 | at
sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:135)
zoo1_1 | ... 15 more
z
Kind regards,
Mate
On Fri, Feb 14, 2020 at 7:12 AM sagar shukla <sa...@yahoo.com.invalid>
wrote:
> O
> Sent from Yahoo Mail on Android
>
> On Fri, Feb 14, 2020 at 11:02 AM, Szalay-Bekő Máté<
> szalay.beko.mate@gmail.com> wrote: Hi Sebastian,
>
> > But I still get the same exception.
> at this point I don't know why this happen... Adding the Server section to
> the jaas config should have helped. Unfortunately the exact exception is
> not printed out into the logs, just the error message, so it is hard to
> find out more details.
>
> I will try to reproduce your case with 3.5.6 locally and see if it works. I
> never actually used digest authentication before... we always use kerberos
> in production. If it works, I will share my configs / dockerfiles and send
> you a patched version with more debug info printed out.
>
> > Why would configuring quorum-auth also enable client-server-auth?
> it is not very logical indeed... if I see it right, based on the code once
> you set the java.security.auth.login.config property, then ZooKeeper
> assumes you want to use server-client sasl authentication. I guess the
> quorum-auth feature was added later and they introduced 'enable' config
> property for this, but forget to introduce the same config for the client
> authentication. I also guess most of the people are interested in the
> client authentication and it is rare that someone does't need that but
> needs quorum auth. Still, the current behaviour is not good I think. I will
> submit a jira ticket requesting an improvement here when I will have time,
> but feel free to submit it yourself if you wish.
>
> Kind regards,
> Mate
>
> On Thu, Feb 13, 2020 at 7:41 PM Sebastian Schmitz <
> sebastian.schmitz@propellerhead.co.nz> wrote:
>
> > Hey Mate,
> >
> > I checked the java.env-file and it contains:
> >
> >
> >
> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"
> >
> > which is exactly the place where the pasted jaas.conf is placed.
> >
> > I also just changed the config to be saslLoginContext and added the
> > missing semicolon.
> >
> > But I still get the same exception.
> >
> > Why would configuring quorum-auth also enable client-server-auth?
> >
> > Thanks
> >
> > Sebastian
> >
> >
> > On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:
> > > Hi Sebastian,
> > >
> > > thanks for the more details!
> > >
> > > One thing I found in your config is that you should use:
> > > quorum.auth.learner.saslLoginContext=QuorumLearner
> > > quorum.auth.server.saslLoginContext=QuorumServer
> > >
> > > so instead of loginContext, use saslLoginContext in both lines. I
> > found
> > > this in the source code, I think the wiki is wrong (I will fix it
> later).
> > > However, actually this didn't really change anything, as the default
> > values
> > > are anyway
> > > QuorumLearner and QuorumServer, so you can even skip these lines from
> > the
> > > config.
> > >
> > > I think Rakesh is right, you are seeing exceptions related to not the
> > > QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find the
> > > 'Server' section (what is configuring the server during the
> client-server
> > > authentication). The name of this section can be overwritten by the
> > > "zookeeper.sasl.serverconfig" system property.
> > >
> > > Based on the exception, ZooKeeper can not find the 'Server' section in
> > > the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are you sure
> > this
> > > is the correct jaas.conf? Does the ZooKeeper process have the
> permissions
> > > to open this file? You can specify the jaas config file path for
> > ZooKeeper
> > > by providing custom system property e.g. by exporting
> > > SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
> > > before starting zkServer.sh
> > >
> > > Also in the jaas.conf you copied here, you are missing a semicolon from
> > the
> > > end of the last line in the Server block. I am not sure if it is
> causing
> > > any parsing error, but I always add the semicolon to the end of the
> last
> > > line in the block.
> > >
> > > Mate
> > >
> > > On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
> > > sebastian.schmitz@propellerhead.co.nz> wrote:
> > >
> > >> Hello Rakesh,
> > >>
> > >> as mentioned in the other mail adding the "Server"to jaas.conf didn't
> > help.
> > >>
> > >> Here are the Configs and Logs (with the Server-part included):
> > >>
> > >> jaas.conf:
> > >> QuorumServer {
> > >> org.apache.zookeeper.server.auth.DigestLoginModule required
> > >> user_zookeeper="test";
> > >> };
> > >>
> > >> QuorumClient {
> > >> org.apache.zookeeper.server.auth.DigestLoginModule required
> > >> username="zookeeper"
> > >> password="test";
> > >> };
> > >>
> > >> Server {
> > >> org.apache.zookeeper.server.auth.DigestLoginModule required
> > >> user_zookeeper="test"
> > >> };
> > >>
> > >> Client {
> > >> org.apache.zookeeper.server.auth.DigestLoginModule required
> > >> username="zookeeper"
> > >> password="test";
> > >> };
> > >>
> > >> zoo.cfg:
> > >> # The number of milliseconds of each tick
> > >> tickTime=2000
> > >> # The number of ticks that the initial
> > >> # synchronization phase can take
> > >> initLimit=10
> > >> # The number of ticks that can pass between
> > >> # sending a request and getting an acknowledgement
> > >> syncLimit=5
> > >> # the directory where the snapshot is stored.
> > >> # do not use /tmp for storage, /tmp here is just
> > >> # example sakes.
> > >> dataDir=/mnt/zk_data
> > >> # the port at which the clients will connect
> > >> clientPort=2181
> > >> # the maximum number of client connections.
> > >> # increase this if you need to handle more clients
> > >> #maxClientCnxns=60
> > >> #
> > >> # Be sure to read the maintenance section of the
> > >> # administrator guide before turning on autopurge.
> > >> #
> > >> #
> > >>
> >
> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
> > >> #
> > >> # The number of snapshots to retain in dataDir
> > >> #autopurge.snapRetainCount=3
> > >> # Purge task interval in hours
> > >> # Set to "0" to disable auto purge feature
> > >> #autopurge.purgeInterval=1
> > >> dataLogDir=/mnt/zk_data_log
> > >> autopurge.snapRetainCount=3
> > >> autopurge.purgeInterval=24
> > >> quorum.auth.enableSasl=true
> > >> quorum.auth.learnerRequireSasl=false
> > >> quorum.auth.serverRequireSasl=false
> > >> quorum.auth.learner.loginContext=QuorumLearner
> > >> quorum.auth.server.loginContext=QuorumServer
> > >> quorum.cnxn.threads.size=20
> > >>
> >
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> > >> secureClientPort=2281
> > >> server.1=0.0.0.0:2888:3888
> > >> server.2=kafkad02.x.azure.com:2888:3888
> > >> server.3=kafkad03.x.azure.com:2888:3888
> > >>
> > >> Server-Log:
> > >> Using config: /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
> > >> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
> > >> Loading configuration file
> > "/opt/zookeeper-cluster/newrelic/./newrelic.yml"
> > >> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: Using default
> > >> collector host: collector.newrelic.com
> > >> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
> > >> Writing to log file:
> > >> /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
> > >> WARNING: An illegal reflective access operation has occurred
> > >> WARNING: Illegal reflective access by
> > >> com.newrelic.weave.weavepackage.NewClassAppender
> > >> (file:/opt/zookeeper-cluster/newrelic/newrelic.jar) to method
> > >> java.net.URLClassLoader.addURL(java.net.URL)
> > >> WARNING: Please consider reporting this to the maintainers of
> > >> com.newrelic.weave.weavepackage.NewClassAppender
> > >> WARNING: Use --illegal-access=warn to enable warnings of further
> illegal
> > >> reflective access operations
> > >> WARNING: All illegal access operations will be denied in a future
> > release
> > >> 2020-02-11 18:43:59,257 [myid:] - INFO [main:QuorumPeerConfig@136] -
> > >> Reading configuration from:
> > >> /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
> > >> 2020-02-11 18:43:59,477 [myid:] - INFO
> > >> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
> > >> kafkad02.x.azure.com to address: kafkad02.x.azure.com/1.2.3.4
> > >> 2020-02-11 18:43:59,477 [myid:] - INFO
> > >> [main:QuorumPeer$QuorumServer@185] - Resolved hostname: 0.0.0.0 to
> > >> address: /0.0.0.0
> > >> 2020-02-11 18:43:59,666 [myid:] - INFO
> > >> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
> > >> kafkad03.x.azure.com to address: kafkad03.x.azure.com/1.2.3.5
> > >> 2020-02-11 18:43:59,666 [myid:] - INFO [main:QuorumPeerConfig@398] -
> > >> Defaulting to majority quorums
> > >> 2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@78
> ]
> > >> - autopurge.snapRetainCount set to 3
> > >> 2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@79
> ]
> > >> - autopurge.purgeInterval set to 24
> > >> 2020-02-11 18:43:59,732 [myid:1] - INFO
> > >> [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
> > >> 2020-02-11 18:43:59,749 [myid:1] - INFO [main:QuorumPeerMain@130] -
> > >> Starting quorum peer
> > >> 2020-02-11 18:43:59,788 [myid:1] - INFO [main:ServerCnxnFactory@117]
> -
> > >> Using org.apache.zookeeper.server.NIOServerCnxnFactory as server
> > >> connection factory
> > >> 2020-02-11 18:43:59,804 [myid:1] - INFO
> > >> [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task
> completed.
> > >> '.20-02-11 18:43:59,826 [myid:1] - ERROR [main:ServerCnxnFactory@210]
> -
> > >> No JAAS configuration section named 'Server' was foundin
> > >> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> > >> 2020-02-11 18:43:59,827 [myid:1] - ERROR [main:QuorumPeerMain@92] -
> > >> Unexpected exception, exiting abnormally
> > >> java.io.IOException: No JAAS configuration section named 'Server' was
> > >> foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> > >> '.
> > >> at
> > >>
> > >>
> >
> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
> > >> at
> > >>
> > >>
> >
> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
> > >> at
> > >>
> > >>
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
> > >> at
> > >>
> > >>
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
> > >> at
> > >>
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
> > >>
> > >>
> > >>
> > >> Best regards
> > >>
> > >> Sebastian
> > >>
> > >>
> > >> On 12-Feb-20 2:36 AM, Rakesh Radhakrishnan wrote:
> > >>>>>>>> java.io.IOException: No JAAS configuration section named
> 'Server'
> > >>> I could see you have enabled client-server authentication as well. It
> > >>> looks to me that the error is coming from that. Please share the
> > >>> complete error logs to trace it.
> > >>> Have you configured "*Server*" section along with the
> "*QuorumServer*"
> > >>> and "*QuorumClient*" sections? If not, please configure "*Server*"
> > >>> section along with others and try it out.
> > >>>
> > >>> Reference:
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
> > >>> image.png
> > >>>
> > >>> Thanks,
> > >>> Rakesh
> > >>>
> > >>> On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz
> > >>> <sebastian.schmitz@propellerhead.co.nz
> > >>> <ma...@propellerhead.co.nz>> wrote:
> > >>>
> > >>> Hello,
> > >>>
> > >>> I'm currently looking into enabling the Auth between
> > >>> Zookeeper-Servers
> > >>> and found this documentation:
> > >>>
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
> > >>> However, when I use the config from the document (for
> Digest-MD5)
> > >>> I get
> > >>> this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried
> > >>> because
> > >>> I thought using latest version could help:
> > >>> java.io.IOException: No JAAS configuration section named
> 'Server'
> > was
> > >>> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> > >>>
> > >>> And of course that's right, because there's only QuorumServer
> and
> > >>> QuorumClient in the jaas.conf:
> > >>>
> > >>> jaas.conf:
> > >>> QuorumServer {
> > >>> org.apache.zookeeper.server.auth.DigestLoginModule
> > required
> > >>> user_zookeeper="test";
> > >>> };
> > >>>
> > >>> QuorumClient {
> > >>> org.apache.zookeeper.server.auth.DigestLoginModule
> > required
> > >>> username="zookeeper"
> > >>> password="test";
> > >>> };
> > >>>
> > >>> I also tried renaming the QuorumServer to just "Server". No
> > change.
> > >>>
> > >>> My zoo.cfg:
> > >>> tickTime=2000
> > >>> initLimit=10
> > >>> syncLimit=5
> > >>> dataDir=/mnt/zk_data
> > >>> clientPort=2181
> > >>> dataLogDir=/mnt/zk_data_log
> > >>> autopurge.snapRetainCount=3
> > >>> autopurge.purgeInterval=24
> > >>> quorum.auth.enableSasl=true
> > >>> quorum.auth.learnerRequireSasl=false
> > >>> quorum.auth.serverRequireSasl=false
> > >>> quorum.auth.learner.loginContext=QuorumLearner
> > >>> quorum.auth.server.loginContext=QuorumServer
> > >>> quorum.cnxn.threads.size=20
> > >>>
> > >>
> >
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> > >>> secureClientPort=2281
> > >>> server.1=0.0.0.0:2888:3888
> > >>>
> > >>> Any idea what I could try? Or maybe there's some better document
> > >>> on how
> > >>> to achieve this?
> > >>>
> > >>> Thank you
> > >>>
> > >>> Sebastian
> > >>>
> > >>>
> > >>> --
> > >>> DISCLAIMER
> > >>> This email contains information that is confidential and which
> > >>> may be
> > >>> legally privileged. If you have received this email in error
> > please
> > >>>
> > >>> notify the sender immediately and delete the email.
> > >>> This email is intended
> > >>> solely for the use of the intended recipient and you may not use
> > or
> > >>> disclose this email in any way.
> > >>>
> > >> --
> > >> DISCLAIMER
> > >> This email contains information that is confidential and which
> > >> may be
> > >> legally privileged. If you have received this email in error please
> > >>
> > >> notify the sender immediately and delete the email.
> > >> This email is intended
> > >> solely for the use of the intended recipient and you may not use or
> > >> disclose this email in any way.
> > >>
> >
> > --
> > DISCLAIMER
> > This email contains information that is confidential and which
> > may be
> > legally privileged. If you have received this email in error please
> >
> > notify the sender immediately and delete the email.
> > This email is intended
> > solely for the use of the intended recipient and you may not use or
> > disclose this email in any way.
> >
>
Re: Enabling Auth between Zookeeper Servers
Posted by sagar shukla <sa...@yahoo.com.INVALID>.
O
Sent from Yahoo Mail on Android
On Fri, Feb 14, 2020 at 11:02 AM, Szalay-Bekő Máté<sz...@gmail.com> wrote: Hi Sebastian,
> But I still get the same exception.
at this point I don't know why this happen... Adding the Server section to
the jaas config should have helped. Unfortunately the exact exception is
not printed out into the logs, just the error message, so it is hard to
find out more details.
I will try to reproduce your case with 3.5.6 locally and see if it works. I
never actually used digest authentication before... we always use kerberos
in production. If it works, I will share my configs / dockerfiles and send
you a patched version with more debug info printed out.
> Why would configuring quorum-auth also enable client-server-auth?
it is not very logical indeed... if I see it right, based on the code once
you set the java.security.auth.login.config property, then ZooKeeper
assumes you want to use server-client sasl authentication. I guess the
quorum-auth feature was added later and they introduced 'enable' config
property for this, but forget to introduce the same config for the client
authentication. I also guess most of the people are interested in the
client authentication and it is rare that someone does't need that but
needs quorum auth. Still, the current behaviour is not good I think. I will
submit a jira ticket requesting an improvement here when I will have time,
but feel free to submit it yourself if you wish.
Kind regards,
Mate
On Thu, Feb 13, 2020 at 7:41 PM Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> Hey Mate,
>
> I checked the java.env-file and it contains:
>
>
> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"
>
> which is exactly the place where the pasted jaas.conf is placed.
>
> I also just changed the config to be saslLoginContext and added the
> missing semicolon.
>
> But I still get the same exception.
>
> Why would configuring quorum-auth also enable client-server-auth?
>
> Thanks
>
> Sebastian
>
>
> On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:
> > Hi Sebastian,
> >
> > thanks for the more details!
> >
> > One thing I found in your config is that you should use:
> > quorum.auth.learner.saslLoginContext=QuorumLearner
> > quorum.auth.server.saslLoginContext=QuorumServer
> >
> > so instead of loginContext, use saslLoginContext in both lines. I
> found
> > this in the source code, I think the wiki is wrong (I will fix it later).
> > However, actually this didn't really change anything, as the default
> values
> > are anyway
> > QuorumLearner and QuorumServer, so you can even skip these lines from
> the
> > config.
> >
> > I think Rakesh is right, you are seeing exceptions related to not the
> > QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find the
> > 'Server' section (what is configuring the server during the client-server
> > authentication). The name of this section can be overwritten by the
> > "zookeeper.sasl.serverconfig" system property.
> >
> > Based on the exception, ZooKeeper can not find the 'Server' section in
> > the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are you sure
> this
> > is the correct jaas.conf? Does the ZooKeeper process have the permissions
> > to open this file? You can specify the jaas config file path for
> ZooKeeper
> > by providing custom system property e.g. by exporting
> > SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
> > before starting zkServer.sh
> >
> > Also in the jaas.conf you copied here, you are missing a semicolon from
> the
> > end of the last line in the Server block. I am not sure if it is causing
> > any parsing error, but I always add the semicolon to the end of the last
> > line in the block.
> >
> > Mate
> >
> > On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
> > sebastian.schmitz@propellerhead.co.nz> wrote:
> >
> >> Hello Rakesh,
> >>
> >> as mentioned in the other mail adding the "Server"to jaas.conf didn't
> help.
> >>
> >> Here are the Configs and Logs (with the Server-part included):
> >>
> >> jaas.conf:
> >> QuorumServer {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> user_zookeeper="test";
> >> };
> >>
> >> QuorumClient {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> username="zookeeper"
> >> password="test";
> >> };
> >>
> >> Server {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> user_zookeeper="test"
> >> };
> >>
> >> Client {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> username="zookeeper"
> >> password="test";
> >> };
> >>
> >> zoo.cfg:
> >> # The number of milliseconds of each tick
> >> tickTime=2000
> >> # The number of ticks that the initial
> >> # synchronization phase can take
> >> initLimit=10
> >> # The number of ticks that can pass between
> >> # sending a request and getting an acknowledgement
> >> syncLimit=5
> >> # the directory where the snapshot is stored.
> >> # do not use /tmp for storage, /tmp here is just
> >> # example sakes.
> >> dataDir=/mnt/zk_data
> >> # the port at which the clients will connect
> >> clientPort=2181
> >> # the maximum number of client connections.
> >> # increase this if you need to handle more clients
> >> #maxClientCnxns=60
> >> #
> >> # Be sure to read the maintenance section of the
> >> # administrator guide before turning on autopurge.
> >> #
> >> #
> >>
> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
> >> #
> >> # The number of snapshots to retain in dataDir
> >> #autopurge.snapRetainCount=3
> >> # Purge task interval in hours
> >> # Set to "0" to disable auto purge feature
> >> #autopurge.purgeInterval=1
> >> dataLogDir=/mnt/zk_data_log
> >> autopurge.snapRetainCount=3
> >> autopurge.purgeInterval=24
> >> quorum.auth.enableSasl=true
> >> quorum.auth.learnerRequireSasl=false
> >> quorum.auth.serverRequireSasl=false
> >> quorum.auth.learner.loginContext=QuorumLearner
> >> quorum.auth.server.loginContext=QuorumServer
> >> quorum.cnxn.threads.size=20
> >>
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> >> secureClientPort=2281
> >> server.1=0.0.0.0:2888:3888
> >> server.2=kafkad02.x.azure.com:2888:3888
> >> server.3=kafkad03.x.azure.com:2888:3888
> >>
> >> Server-Log:
> >> Using config: /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
> >> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
> >> Loading configuration file
> "/opt/zookeeper-cluster/newrelic/./newrelic.yml"
> >> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: Using default
> >> collector host: collector.newrelic.com
> >> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
> >> Writing to log file:
> >> /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
> >> WARNING: An illegal reflective access operation has occurred
> >> WARNING: Illegal reflective access by
> >> com.newrelic.weave.weavepackage.NewClassAppender
> >> (file:/opt/zookeeper-cluster/newrelic/newrelic.jar) to method
> >> java.net.URLClassLoader.addURL(java.net.URL)
> >> WARNING: Please consider reporting this to the maintainers of
> >> com.newrelic.weave.weavepackage.NewClassAppender
> >> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> >> reflective access operations
> >> WARNING: All illegal access operations will be denied in a future
> release
> >> 2020-02-11 18:43:59,257 [myid:] - INFO [main:QuorumPeerConfig@136] -
> >> Reading configuration from:
> >> /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
> >> 2020-02-11 18:43:59,477 [myid:] - INFO
> >> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
> >> kafkad02.x.azure.com to address: kafkad02.x.azure.com/1.2.3.4
> >> 2020-02-11 18:43:59,477 [myid:] - INFO
> >> [main:QuorumPeer$QuorumServer@185] - Resolved hostname: 0.0.0.0 to
> >> address: /0.0.0.0
> >> 2020-02-11 18:43:59,666 [myid:] - INFO
> >> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
> >> kafkad03.x.azure.com to address: kafkad03.x.azure.com/1.2.3.5
> >> 2020-02-11 18:43:59,666 [myid:] - INFO [main:QuorumPeerConfig@398] -
> >> Defaulting to majority quorums
> >> 2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@78]
> >> - autopurge.snapRetainCount set to 3
> >> 2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@79]
> >> - autopurge.purgeInterval set to 24
> >> 2020-02-11 18:43:59,732 [myid:1] - INFO
> >> [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
> >> 2020-02-11 18:43:59,749 [myid:1] - INFO [main:QuorumPeerMain@130] -
> >> Starting quorum peer
> >> 2020-02-11 18:43:59,788 [myid:1] - INFO [main:ServerCnxnFactory@117] -
> >> Using org.apache.zookeeper.server.NIOServerCnxnFactory as server
> >> connection factory
> >> 2020-02-11 18:43:59,804 [myid:1] - INFO
> >> [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
> >> '.20-02-11 18:43:59,826 [myid:1] - ERROR [main:ServerCnxnFactory@210] -
> >> No JAAS configuration section named 'Server' was foundin
> >> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >> 2020-02-11 18:43:59,827 [myid:1] - ERROR [main:QuorumPeerMain@92] -
> >> Unexpected exception, exiting abnormally
> >> java.io.IOException: No JAAS configuration section named 'Server' was
> >> foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >> '.
> >> at
> >>
> >>
> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
> >> at
> >>
> >>
> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
> >> at
> >>
> >>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
> >> at
> >>
> >>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
> >> at
> >>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
> >>
> >>
> >>
> >> Best regards
> >>
> >> Sebastian
> >>
> >>
> >> On 12-Feb-20 2:36 AM, Rakesh Radhakrishnan wrote:
> >>>>>>>> java.io.IOException: No JAAS configuration section named 'Server'
> >>> I could see you have enabled client-server authentication as well. It
> >>> looks to me that the error is coming from that. Please share the
> >>> complete error logs to trace it.
> >>> Have you configured "*Server*" section along with the "*QuorumServer*"
> >>> and "*QuorumClient*" sections? If not, please configure "*Server*"
> >>> section along with others and try it out.
> >>>
> >>> Reference:
> >>>
> >>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
> >>> image.png
> >>>
> >>> Thanks,
> >>> Rakesh
> >>>
> >>> On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz
> >>> <sebastian.schmitz@propellerhead.co.nz
> >>> <ma...@propellerhead.co.nz>> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I'm currently looking into enabling the Auth between
> >>> Zookeeper-Servers
> >>> and found this documentation:
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
> >>> However, when I use the config from the document (for Digest-MD5)
> >>> I get
> >>> this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried
> >>> because
> >>> I thought using latest version could help:
> >>> java.io.IOException: No JAAS configuration section named 'Server'
> was
> >>> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >>>
> >>> And of course that's right, because there's only QuorumServer and
> >>> QuorumClient in the jaas.conf:
> >>>
> >>> jaas.conf:
> >>> QuorumServer {
> >>> org.apache.zookeeper.server.auth.DigestLoginModule
> required
> >>> user_zookeeper="test";
> >>> };
> >>>
> >>> QuorumClient {
> >>> org.apache.zookeeper.server.auth.DigestLoginModule
> required
> >>> username="zookeeper"
> >>> password="test";
> >>> };
> >>>
> >>> I also tried renaming the QuorumServer to just "Server". No
> change.
> >>>
> >>> My zoo.cfg:
> >>> tickTime=2000
> >>> initLimit=10
> >>> syncLimit=5
> >>> dataDir=/mnt/zk_data
> >>> clientPort=2181
> >>> dataLogDir=/mnt/zk_data_log
> >>> autopurge.snapRetainCount=3
> >>> autopurge.purgeInterval=24
> >>> quorum.auth.enableSasl=true
> >>> quorum.auth.learnerRequireSasl=false
> >>> quorum.auth.serverRequireSasl=false
> >>> quorum.auth.learner.loginContext=QuorumLearner
> >>> quorum.auth.server.loginContext=QuorumServer
> >>> quorum.cnxn.threads.size=20
> >>>
> >>
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> >>> secureClientPort=2281
> >>> server.1=0.0.0.0:2888:3888
> >>>
> >>> Any idea what I could try? Or maybe there's some better document
> >>> on how
> >>> to achieve this?
> >>>
> >>> Thank you
> >>>
> >>> Sebastian
> >>>
> >>>
> >>> --
> >>> DISCLAIMER
> >>> This email contains information that is confidential and which
> >>> may be
> >>> legally privileged. If you have received this email in error
> please
> >>>
> >>> notify the sender immediately and delete the email.
> >>> This email is intended
> >>> solely for the use of the intended recipient and you may not use
> or
> >>> disclose this email in any way.
> >>>
> >> --
> >> DISCLAIMER
> >> This email contains information that is confidential and which
> >> may be
> >> legally privileged. If you have received this email in error please
> >>
> >> notify the sender immediately and delete the email.
> >> This email is intended
> >> solely for the use of the intended recipient and you may not use or
> >> disclose this email in any way.
> >>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>
Re: Enabling Auth between Zookeeper Servers
Posted by Szalay-Bekő Máté <sz...@gmail.com>.
Hi Sebastian,
> But I still get the same exception.
at this point I don't know why this happen... Adding the Server section to
the jaas config should have helped. Unfortunately the exact exception is
not printed out into the logs, just the error message, so it is hard to
find out more details.
I will try to reproduce your case with 3.5.6 locally and see if it works. I
never actually used digest authentication before... we always use kerberos
in production. If it works, I will share my configs / dockerfiles and send
you a patched version with more debug info printed out.
> Why would configuring quorum-auth also enable client-server-auth?
it is not very logical indeed... if I see it right, based on the code once
you set the java.security.auth.login.config property, then ZooKeeper
assumes you want to use server-client sasl authentication. I guess the
quorum-auth feature was added later and they introduced 'enable' config
property for this, but forget to introduce the same config for the client
authentication. I also guess most of the people are interested in the
client authentication and it is rare that someone does't need that but
needs quorum auth. Still, the current behaviour is not good I think. I will
submit a jira ticket requesting an improvement here when I will have time,
but feel free to submit it yourself if you wish.
Kind regards,
Mate
On Thu, Feb 13, 2020 at 7:41 PM Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> Hey Mate,
>
> I checked the java.env-file and it contains:
>
>
> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"
>
> which is exactly the place where the pasted jaas.conf is placed.
>
> I also just changed the config to be saslLoginContext and added the
> missing semicolon.
>
> But I still get the same exception.
>
> Why would configuring quorum-auth also enable client-server-auth?
>
> Thanks
>
> Sebastian
>
>
> On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:
> > Hi Sebastian,
> >
> > thanks for the more details!
> >
> > One thing I found in your config is that you should use:
> > quorum.auth.learner.saslLoginContext=QuorumLearner
> > quorum.auth.server.saslLoginContext=QuorumServer
> >
> > so instead of loginContext, use saslLoginContext in both lines. I
> found
> > this in the source code, I think the wiki is wrong (I will fix it later).
> > However, actually this didn't really change anything, as the default
> values
> > are anyway
> > QuorumLearner and QuorumServer, so you can even skip these lines from
> the
> > config.
> >
> > I think Rakesh is right, you are seeing exceptions related to not the
> > QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find the
> > 'Server' section (what is configuring the server during the client-server
> > authentication). The name of this section can be overwritten by the
> > "zookeeper.sasl.serverconfig" system property.
> >
> > Based on the exception, ZooKeeper can not find the 'Server' section in
> > the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are you sure
> this
> > is the correct jaas.conf? Does the ZooKeeper process have the permissions
> > to open this file? You can specify the jaas config file path for
> ZooKeeper
> > by providing custom system property e.g. by exporting
> > SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
> > before starting zkServer.sh
> >
> > Also in the jaas.conf you copied here, you are missing a semicolon from
> the
> > end of the last line in the Server block. I am not sure if it is causing
> > any parsing error, but I always add the semicolon to the end of the last
> > line in the block.
> >
> > Mate
> >
> > On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
> > sebastian.schmitz@propellerhead.co.nz> wrote:
> >
> >> Hello Rakesh,
> >>
> >> as mentioned in the other mail adding the "Server"to jaas.conf didn't
> help.
> >>
> >> Here are the Configs and Logs (with the Server-part included):
> >>
> >> jaas.conf:
> >> QuorumServer {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> user_zookeeper="test";
> >> };
> >>
> >> QuorumClient {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> username="zookeeper"
> >> password="test";
> >> };
> >>
> >> Server {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> user_zookeeper="test"
> >> };
> >>
> >> Client {
> >> org.apache.zookeeper.server.auth.DigestLoginModule required
> >> username="zookeeper"
> >> password="test";
> >> };
> >>
> >> zoo.cfg:
> >> # The number of milliseconds of each tick
> >> tickTime=2000
> >> # The number of ticks that the initial
> >> # synchronization phase can take
> >> initLimit=10
> >> # The number of ticks that can pass between
> >> # sending a request and getting an acknowledgement
> >> syncLimit=5
> >> # the directory where the snapshot is stored.
> >> # do not use /tmp for storage, /tmp here is just
> >> # example sakes.
> >> dataDir=/mnt/zk_data
> >> # the port at which the clients will connect
> >> clientPort=2181
> >> # the maximum number of client connections.
> >> # increase this if you need to handle more clients
> >> #maxClientCnxns=60
> >> #
> >> # Be sure to read the maintenance section of the
> >> # administrator guide before turning on autopurge.
> >> #
> >> #
> >>
> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
> >> #
> >> # The number of snapshots to retain in dataDir
> >> #autopurge.snapRetainCount=3
> >> # Purge task interval in hours
> >> # Set to "0" to disable auto purge feature
> >> #autopurge.purgeInterval=1
> >> dataLogDir=/mnt/zk_data_log
> >> autopurge.snapRetainCount=3
> >> autopurge.purgeInterval=24
> >> quorum.auth.enableSasl=true
> >> quorum.auth.learnerRequireSasl=false
> >> quorum.auth.serverRequireSasl=false
> >> quorum.auth.learner.loginContext=QuorumLearner
> >> quorum.auth.server.loginContext=QuorumServer
> >> quorum.cnxn.threads.size=20
> >>
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> >> secureClientPort=2281
> >> server.1=0.0.0.0:2888:3888
> >> server.2=kafkad02.x.azure.com:2888:3888
> >> server.3=kafkad03.x.azure.com:2888:3888
> >>
> >> Server-Log:
> >> Using config: /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
> >> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
> >> Loading configuration file
> "/opt/zookeeper-cluster/newrelic/./newrelic.yml"
> >> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: Using default
> >> collector host: collector.newrelic.com
> >> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
> >> Writing to log file:
> >> /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
> >> WARNING: An illegal reflective access operation has occurred
> >> WARNING: Illegal reflective access by
> >> com.newrelic.weave.weavepackage.NewClassAppender
> >> (file:/opt/zookeeper-cluster/newrelic/newrelic.jar) to method
> >> java.net.URLClassLoader.addURL(java.net.URL)
> >> WARNING: Please consider reporting this to the maintainers of
> >> com.newrelic.weave.weavepackage.NewClassAppender
> >> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> >> reflective access operations
> >> WARNING: All illegal access operations will be denied in a future
> release
> >> 2020-02-11 18:43:59,257 [myid:] - INFO [main:QuorumPeerConfig@136] -
> >> Reading configuration from:
> >> /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
> >> 2020-02-11 18:43:59,477 [myid:] - INFO
> >> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
> >> kafkad02.x.azure.com to address: kafkad02.x.azure.com/1.2.3.4
> >> 2020-02-11 18:43:59,477 [myid:] - INFO
> >> [main:QuorumPeer$QuorumServer@185] - Resolved hostname: 0.0.0.0 to
> >> address: /0.0.0.0
> >> 2020-02-11 18:43:59,666 [myid:] - INFO
> >> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
> >> kafkad03.x.azure.com to address: kafkad03.x.azure.com/1.2.3.5
> >> 2020-02-11 18:43:59,666 [myid:] - INFO [main:QuorumPeerConfig@398] -
> >> Defaulting to majority quorums
> >> 2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@78]
> >> - autopurge.snapRetainCount set to 3
> >> 2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@79]
> >> - autopurge.purgeInterval set to 24
> >> 2020-02-11 18:43:59,732 [myid:1] - INFO
> >> [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
> >> 2020-02-11 18:43:59,749 [myid:1] - INFO [main:QuorumPeerMain@130] -
> >> Starting quorum peer
> >> 2020-02-11 18:43:59,788 [myid:1] - INFO [main:ServerCnxnFactory@117] -
> >> Using org.apache.zookeeper.server.NIOServerCnxnFactory as server
> >> connection factory
> >> 2020-02-11 18:43:59,804 [myid:1] - INFO
> >> [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
> >> '.20-02-11 18:43:59,826 [myid:1] - ERROR [main:ServerCnxnFactory@210] -
> >> No JAAS configuration section named 'Server' was foundin
> >> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >> 2020-02-11 18:43:59,827 [myid:1] - ERROR [main:QuorumPeerMain@92] -
> >> Unexpected exception, exiting abnormally
> >> java.io.IOException: No JAAS configuration section named 'Server' was
> >> foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >> '.
> >> at
> >>
> >>
> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
> >> at
> >>
> >>
> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
> >> at
> >>
> >>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
> >> at
> >>
> >>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
> >> at
> >>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
> >>
> >>
> >>
> >> Best regards
> >>
> >> Sebastian
> >>
> >>
> >> On 12-Feb-20 2:36 AM, Rakesh Radhakrishnan wrote:
> >>>>>>>> java.io.IOException: No JAAS configuration section named 'Server'
> >>> I could see you have enabled client-server authentication as well. It
> >>> looks to me that the error is coming from that. Please share the
> >>> complete error logs to trace it.
> >>> Have you configured "*Server*" section along with the "*QuorumServer*"
> >>> and "*QuorumClient*" sections? If not, please configure "*Server*"
> >>> section along with others and try it out.
> >>>
> >>> Reference:
> >>>
> >>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
> >>> image.png
> >>>
> >>> Thanks,
> >>> Rakesh
> >>>
> >>> On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz
> >>> <sebastian.schmitz@propellerhead.co.nz
> >>> <ma...@propellerhead.co.nz>> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I'm currently looking into enabling the Auth between
> >>> Zookeeper-Servers
> >>> and found this documentation:
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
> >>> However, when I use the config from the document (for Digest-MD5)
> >>> I get
> >>> this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried
> >>> because
> >>> I thought using latest version could help:
> >>> java.io.IOException: No JAAS configuration section named 'Server'
> was
> >>> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >>>
> >>> And of course that's right, because there's only QuorumServer and
> >>> QuorumClient in the jaas.conf:
> >>>
> >>> jaas.conf:
> >>> QuorumServer {
> >>> org.apache.zookeeper.server.auth.DigestLoginModule
> required
> >>> user_zookeeper="test";
> >>> };
> >>>
> >>> QuorumClient {
> >>> org.apache.zookeeper.server.auth.DigestLoginModule
> required
> >>> username="zookeeper"
> >>> password="test";
> >>> };
> >>>
> >>> I also tried renaming the QuorumServer to just "Server". No
> change.
> >>>
> >>> My zoo.cfg:
> >>> tickTime=2000
> >>> initLimit=10
> >>> syncLimit=5
> >>> dataDir=/mnt/zk_data
> >>> clientPort=2181
> >>> dataLogDir=/mnt/zk_data_log
> >>> autopurge.snapRetainCount=3
> >>> autopurge.purgeInterval=24
> >>> quorum.auth.enableSasl=true
> >>> quorum.auth.learnerRequireSasl=false
> >>> quorum.auth.serverRequireSasl=false
> >>> quorum.auth.learner.loginContext=QuorumLearner
> >>> quorum.auth.server.loginContext=QuorumServer
> >>> quorum.cnxn.threads.size=20
> >>>
> >>
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> >>> secureClientPort=2281
> >>> server.1=0.0.0.0:2888:3888
> >>>
> >>> Any idea what I could try? Or maybe there's some better document
> >>> on how
> >>> to achieve this?
> >>>
> >>> Thank you
> >>>
> >>> Sebastian
> >>>
> >>>
> >>> --
> >>> DISCLAIMER
> >>> This email contains information that is confidential and which
> >>> may be
> >>> legally privileged. If you have received this email in error
> please
> >>>
> >>> notify the sender immediately and delete the email.
> >>> This email is intended
> >>> solely for the use of the intended recipient and you may not use
> or
> >>> disclose this email in any way.
> >>>
> >> --
> >> DISCLAIMER
> >> This email contains information that is confidential and which
> >> may be
> >> legally privileged. If you have received this email in error please
> >>
> >> notify the sender immediately and delete the email.
> >> This email is intended
> >> solely for the use of the intended recipient and you may not use or
> >> disclose this email in any way.
> >>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>
Re: Enabling Auth between Zookeeper Servers
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
Hey Mate,
I checked the java.env-file and it contains:
SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-cluster/zookeeper/conf/jaas.conf"
which is exactly the place where the pasted jaas.conf is placed.
I also just changed the config to be saslLoginContext and added the
missing semicolon.
But I still get the same exception.
Why would configuring quorum-auth also enable client-server-auth?
Thanks
Sebastian
On 13-Feb-20 5:50 AM, Szalay-Bekő Máté wrote:
> Hi Sebastian,
>
> thanks for the more details!
>
> One thing I found in your config is that you should use:
> quorum.auth.learner.saslLoginContext=QuorumLearner
> quorum.auth.server.saslLoginContext=QuorumServer
>
> so instead of loginContext, use saslLoginContext in both lines. I found
> this in the source code, I think the wiki is wrong (I will fix it later).
> However, actually this didn't really change anything, as the default values
> are anyway
> QuorumLearner and QuorumServer, so you can even skip these lines from the
> config.
>
> I think Rakesh is right, you are seeing exceptions related to not the
> QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find the
> 'Server' section (what is configuring the server during the client-server
> authentication). The name of this section can be overwritten by the
> "zookeeper.sasl.serverconfig" system property.
>
> Based on the exception, ZooKeeper can not find the 'Server' section in
> the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are you sure this
> is the correct jaas.conf? Does the ZooKeeper process have the permissions
> to open this file? You can specify the jaas config file path for ZooKeeper
> by providing custom system property e.g. by exporting
> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
> before starting zkServer.sh
>
> Also in the jaas.conf you copied here, you are missing a semicolon from the
> end of the last line in the Server block. I am not sure if it is causing
> any parsing error, but I always add the semicolon to the end of the last
> line in the block.
>
> Mate
>
> On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
> sebastian.schmitz@propellerhead.co.nz> wrote:
>
>> Hello Rakesh,
>>
>> as mentioned in the other mail adding the "Server"to jaas.conf didn't help.
>>
>> Here are the Configs and Logs (with the Server-part included):
>>
>> jaas.conf:
>> QuorumServer {
>> org.apache.zookeeper.server.auth.DigestLoginModule required
>> user_zookeeper="test";
>> };
>>
>> QuorumClient {
>> org.apache.zookeeper.server.auth.DigestLoginModule required
>> username="zookeeper"
>> password="test";
>> };
>>
>> Server {
>> org.apache.zookeeper.server.auth.DigestLoginModule required
>> user_zookeeper="test"
>> };
>>
>> Client {
>> org.apache.zookeeper.server.auth.DigestLoginModule required
>> username="zookeeper"
>> password="test";
>> };
>>
>> zoo.cfg:
>> # The number of milliseconds of each tick
>> tickTime=2000
>> # The number of ticks that the initial
>> # synchronization phase can take
>> initLimit=10
>> # The number of ticks that can pass between
>> # sending a request and getting an acknowledgement
>> syncLimit=5
>> # the directory where the snapshot is stored.
>> # do not use /tmp for storage, /tmp here is just
>> # example sakes.
>> dataDir=/mnt/zk_data
>> # the port at which the clients will connect
>> clientPort=2181
>> # the maximum number of client connections.
>> # increase this if you need to handle more clients
>> #maxClientCnxns=60
>> #
>> # Be sure to read the maintenance section of the
>> # administrator guide before turning on autopurge.
>> #
>> #
>> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>> #
>> # The number of snapshots to retain in dataDir
>> #autopurge.snapRetainCount=3
>> # Purge task interval in hours
>> # Set to "0" to disable auto purge feature
>> #autopurge.purgeInterval=1
>> dataLogDir=/mnt/zk_data_log
>> autopurge.snapRetainCount=3
>> autopurge.purgeInterval=24
>> quorum.auth.enableSasl=true
>> quorum.auth.learnerRequireSasl=false
>> quorum.auth.serverRequireSasl=false
>> quorum.auth.learner.loginContext=QuorumLearner
>> quorum.auth.server.loginContext=QuorumServer
>> quorum.cnxn.threads.size=20
>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>> secureClientPort=2281
>> server.1=0.0.0.0:2888:3888
>> server.2=kafkad02.x.azure.com:2888:3888
>> server.3=kafkad03.x.azure.com:2888:3888
>>
>> Server-Log:
>> Using config: /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
>> Loading configuration file "/opt/zookeeper-cluster/newrelic/./newrelic.yml"
>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: Using default
>> collector host: collector.newrelic.com
>> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
>> Writing to log file:
>> /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
>> WARNING: An illegal reflective access operation has occurred
>> WARNING: Illegal reflective access by
>> com.newrelic.weave.weavepackage.NewClassAppender
>> (file:/opt/zookeeper-cluster/newrelic/newrelic.jar) to method
>> java.net.URLClassLoader.addURL(java.net.URL)
>> WARNING: Please consider reporting this to the maintainers of
>> com.newrelic.weave.weavepackage.NewClassAppender
>> WARNING: Use --illegal-access=warn to enable warnings of further illegal
>> reflective access operations
>> WARNING: All illegal access operations will be denied in a future release
>> 2020-02-11 18:43:59,257 [myid:] - INFO [main:QuorumPeerConfig@136] -
>> Reading configuration from:
>> /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
>> 2020-02-11 18:43:59,477 [myid:] - INFO
>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
>> kafkad02.x.azure.com to address: kafkad02.x.azure.com/1.2.3.4
>> 2020-02-11 18:43:59,477 [myid:] - INFO
>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname: 0.0.0.0 to
>> address: /0.0.0.0
>> 2020-02-11 18:43:59,666 [myid:] - INFO
>> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
>> kafkad03.x.azure.com to address: kafkad03.x.azure.com/1.2.3.5
>> 2020-02-11 18:43:59,666 [myid:] - INFO [main:QuorumPeerConfig@398] -
>> Defaulting to majority quorums
>> 2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@78]
>> - autopurge.snapRetainCount set to 3
>> 2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@79]
>> - autopurge.purgeInterval set to 24
>> 2020-02-11 18:43:59,732 [myid:1] - INFO
>> [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
>> 2020-02-11 18:43:59,749 [myid:1] - INFO [main:QuorumPeerMain@130] -
>> Starting quorum peer
>> 2020-02-11 18:43:59,788 [myid:1] - INFO [main:ServerCnxnFactory@117] -
>> Using org.apache.zookeeper.server.NIOServerCnxnFactory as server
>> connection factory
>> 2020-02-11 18:43:59,804 [myid:1] - INFO
>> [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
>> '.20-02-11 18:43:59,826 [myid:1] - ERROR [main:ServerCnxnFactory@210] -
>> No JAAS configuration section named 'Server' was foundin
>> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>> 2020-02-11 18:43:59,827 [myid:1] - ERROR [main:QuorumPeerMain@92] -
>> Unexpected exception, exiting abnormally
>> java.io.IOException: No JAAS configuration section named 'Server' was
>> foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>> '.
>> at
>>
>> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
>> at
>>
>> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
>> at
>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
>> at
>>
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
>> at
>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
>>
>>
>>
>> Best regards
>>
>> Sebastian
>>
>>
>> On 12-Feb-20 2:36 AM, Rakesh Radhakrishnan wrote:
>>>>>>>> java.io.IOException: No JAAS configuration section named 'Server'
>>> I could see you have enabled client-server authentication as well. It
>>> looks to me that the error is coming from that. Please share the
>>> complete error logs to trace it.
>>> Have you configured "*Server*" section along with the "*QuorumServer*"
>>> and "*QuorumClient*" sections? If not, please configure "*Server*"
>>> section along with others and try it out.
>>>
>>> Reference:
>>>
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
>>> image.png
>>>
>>> Thanks,
>>> Rakesh
>>>
>>> On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz
>>> <sebastian.schmitz@propellerhead.co.nz
>>> <ma...@propellerhead.co.nz>> wrote:
>>>
>>> Hello,
>>>
>>> I'm currently looking into enabling the Auth between
>>> Zookeeper-Servers
>>> and found this documentation:
>>>
>>>
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
>>> However, when I use the config from the document (for Digest-MD5)
>>> I get
>>> this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried
>>> because
>>> I thought using latest version could help:
>>> java.io.IOException: No JAAS configuration section named 'Server' was
>>> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>>
>>> And of course that's right, because there's only QuorumServer and
>>> QuorumClient in the jaas.conf:
>>>
>>> jaas.conf:
>>> QuorumServer {
>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>> user_zookeeper="test";
>>> };
>>>
>>> QuorumClient {
>>> org.apache.zookeeper.server.auth.DigestLoginModule required
>>> username="zookeeper"
>>> password="test";
>>> };
>>>
>>> I also tried renaming the QuorumServer to just "Server". No change.
>>>
>>> My zoo.cfg:
>>> tickTime=2000
>>> initLimit=10
>>> syncLimit=5
>>> dataDir=/mnt/zk_data
>>> clientPort=2181
>>> dataLogDir=/mnt/zk_data_log
>>> autopurge.snapRetainCount=3
>>> autopurge.purgeInterval=24
>>> quorum.auth.enableSasl=true
>>> quorum.auth.learnerRequireSasl=false
>>> quorum.auth.serverRequireSasl=false
>>> quorum.auth.learner.loginContext=QuorumLearner
>>> quorum.auth.server.loginContext=QuorumServer
>>> quorum.cnxn.threads.size=20
>>>
>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>>> secureClientPort=2281
>>> server.1=0.0.0.0:2888:3888
>>>
>>> Any idea what I could try? Or maybe there's some better document
>>> on how
>>> to achieve this?
>>>
>>> Thank you
>>>
>>> Sebastian
>>>
>>>
>>> --
>>> DISCLAIMER
>>> This email contains information that is confidential and which
>>> may be
>>> legally privileged. If you have received this email in error please
>>>
>>> notify the sender immediately and delete the email.
>>> This email is intended
>>> solely for the use of the intended recipient and you may not use or
>>> disclose this email in any way.
>>>
>> --
>> DISCLAIMER
>> This email contains information that is confidential and which
>> may be
>> legally privileged. If you have received this email in error please
>>
>> notify the sender immediately and delete the email.
>> This email is intended
>> solely for the use of the intended recipient and you may not use or
>> disclose this email in any way.
>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: Enabling Auth between Zookeeper Servers
Posted by Szalay-Bekő Máté <sz...@gmail.com>.
Hi Sebastian,
thanks for the more details!
One thing I found in your config is that you should use:
quorum.auth.learner.saslLoginContext=QuorumLearner
quorum.auth.server.saslLoginContext=QuorumServer
so instead of loginContext, use saslLoginContext in both lines. I found
this in the source code, I think the wiki is wrong (I will fix it later).
However, actually this didn't really change anything, as the default values
are anyway
QuorumLearner and QuorumServer, so you can even skip these lines from the
config.
I think Rakesh is right, you are seeing exceptions related to not the
QuorumSasl, but the ClientSasl. This is why ZooKeeper tries to find the
'Server' section (what is configuring the server during the client-server
authentication). The name of this section can be overwritten by the
"zookeeper.sasl.serverconfig" system property.
Based on the exception, ZooKeeper can not find the 'Server' section in
the /opt/zookeeper-cluster/zookeeper/conf/jaas.conf file. Are you sure this
is the correct jaas.conf? Does the ZooKeeper process have the permissions
to open this file? You can specify the jaas config file path for ZooKeeper
by providing custom system property e.g. by exporting
SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/jaas.conf"
before starting zkServer.sh
Also in the jaas.conf you copied here, you are missing a semicolon from the
end of the last line in the Server block. I am not sure if it is causing
any parsing error, but I always add the semicolon to the end of the last
line in the block.
Mate
On Tue, Feb 11, 2020 at 7:53 PM Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> Hello Rakesh,
>
> as mentioned in the other mail adding the "Server"to jaas.conf didn't help.
>
> Here are the Configs and Logs (with the Server-part included):
>
> jaas.conf:
> QuorumServer {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> user_zookeeper="test";
> };
>
> QuorumClient {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> username="zookeeper"
> password="test";
> };
>
> Server {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> user_zookeeper="test"
> };
>
> Client {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> username="zookeeper"
> password="test";
> };
>
> zoo.cfg:
> # The number of milliseconds of each tick
> tickTime=2000
> # The number of ticks that the initial
> # synchronization phase can take
> initLimit=10
> # The number of ticks that can pass between
> # sending a request and getting an acknowledgement
> syncLimit=5
> # the directory where the snapshot is stored.
> # do not use /tmp for storage, /tmp here is just
> # example sakes.
> dataDir=/mnt/zk_data
> # the port at which the clients will connect
> clientPort=2181
> # the maximum number of client connections.
> # increase this if you need to handle more clients
> #maxClientCnxns=60
> #
> # Be sure to read the maintenance section of the
> # administrator guide before turning on autopurge.
> #
> #
> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
> #
> # The number of snapshots to retain in dataDir
> #autopurge.snapRetainCount=3
> # Purge task interval in hours
> # Set to "0" to disable auto purge feature
> #autopurge.purgeInterval=1
> dataLogDir=/mnt/zk_data_log
> autopurge.snapRetainCount=3
> autopurge.purgeInterval=24
> quorum.auth.enableSasl=true
> quorum.auth.learnerRequireSasl=false
> quorum.auth.serverRequireSasl=false
> quorum.auth.learner.loginContext=QuorumLearner
> quorum.auth.server.loginContext=QuorumServer
> quorum.cnxn.threads.size=20
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> secureClientPort=2281
> server.1=0.0.0.0:2888:3888
> server.2=kafkad02.x.azure.com:2888:3888
> server.3=kafkad03.x.azure.com:2888:3888
>
> Server-Log:
> Using config: /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
> Loading configuration file "/opt/zookeeper-cluster/newrelic/./newrelic.yml"
> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: Using default
> collector host: collector.newrelic.com
> Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
> Writing to log file:
> /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> com.newrelic.weave.weavepackage.NewClassAppender
> (file:/opt/zookeeper-cluster/newrelic/newrelic.jar) to method
> java.net.URLClassLoader.addURL(java.net.URL)
> WARNING: Please consider reporting this to the maintainers of
> com.newrelic.weave.weavepackage.NewClassAppender
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> 2020-02-11 18:43:59,257 [myid:] - INFO [main:QuorumPeerConfig@136] -
> Reading configuration from:
> /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
> 2020-02-11 18:43:59,477 [myid:] - INFO
> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
> kafkad02.x.azure.com to address: kafkad02.x.azure.com/1.2.3.4
> 2020-02-11 18:43:59,477 [myid:] - INFO
> [main:QuorumPeer$QuorumServer@185] - Resolved hostname: 0.0.0.0 to
> address: /0.0.0.0
> 2020-02-11 18:43:59,666 [myid:] - INFO
> [main:QuorumPeer$QuorumServer@185] - Resolved hostname:
> kafkad03.x.azure.com to address: kafkad03.x.azure.com/1.2.3.5
> 2020-02-11 18:43:59,666 [myid:] - INFO [main:QuorumPeerConfig@398] -
> Defaulting to majority quorums
> 2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@78]
> - autopurge.snapRetainCount set to 3
> 2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@79]
> - autopurge.purgeInterval set to 24
> 2020-02-11 18:43:59,732 [myid:1] - INFO
> [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
> 2020-02-11 18:43:59,749 [myid:1] - INFO [main:QuorumPeerMain@130] -
> Starting quorum peer
> 2020-02-11 18:43:59,788 [myid:1] - INFO [main:ServerCnxnFactory@117] -
> Using org.apache.zookeeper.server.NIOServerCnxnFactory as server
> connection factory
> 2020-02-11 18:43:59,804 [myid:1] - INFO
> [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
> '.20-02-11 18:43:59,826 [myid:1] - ERROR [main:ServerCnxnFactory@210] -
> No JAAS configuration section named 'Server' was foundin
> '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> 2020-02-11 18:43:59,827 [myid:1] - ERROR [main:QuorumPeerMain@92] -
> Unexpected exception, exiting abnormally
> java.io.IOException: No JAAS configuration section named 'Server' was
> foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> '.
> at
>
> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
> at
>
> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
>
>
>
> Best regards
>
> Sebastian
>
>
> On 12-Feb-20 2:36 AM, Rakesh Radhakrishnan wrote:
> > >>>>>java.io.IOException: No JAAS configuration section named 'Server'
> >
> > I could see you have enabled client-server authentication as well. It
> > looks to me that the error is coming from that. Please share the
> > complete error logs to trace it.
> > Have you configured "*Server*" section along with the "*QuorumServer*"
> > and "*QuorumClient*" sections? If not, please configure "*Server*"
> > section along with others and try it out.
> >
> > Reference:
> >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
> > image.png
> >
> > Thanks,
> > Rakesh
> >
> > On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz
> > <sebastian.schmitz@propellerhead.co.nz
> > <ma...@propellerhead.co.nz>> wrote:
> >
> > Hello,
> >
> > I'm currently looking into enabling the Auth between
> > Zookeeper-Servers
> > and found this documentation:
> >
> >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
> >
> > However, when I use the config from the document (for Digest-MD5)
> > I get
> > this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried
> > because
> > I thought using latest version could help:
> > java.io.IOException: No JAAS configuration section named 'Server' was
> > found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
> >
> > And of course that's right, because there's only QuorumServer and
> > QuorumClient in the jaas.conf:
> >
> > jaas.conf:
> > QuorumServer {
> > org.apache.zookeeper.server.auth.DigestLoginModule required
> > user_zookeeper="test";
> > };
> >
> > QuorumClient {
> > org.apache.zookeeper.server.auth.DigestLoginModule required
> > username="zookeeper"
> > password="test";
> > };
> >
> > I also tried renaming the QuorumServer to just "Server". No change.
> >
> > My zoo.cfg:
> > tickTime=2000
> > initLimit=10
> > syncLimit=5
> > dataDir=/mnt/zk_data
> > clientPort=2181
> > dataLogDir=/mnt/zk_data_log
> > autopurge.snapRetainCount=3
> > autopurge.purgeInterval=24
> > quorum.auth.enableSasl=true
> > quorum.auth.learnerRequireSasl=false
> > quorum.auth.serverRequireSasl=false
> > quorum.auth.learner.loginContext=QuorumLearner
> > quorum.auth.server.loginContext=QuorumServer
> > quorum.cnxn.threads.size=20
> >
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> > secureClientPort=2281
> > server.1=0.0.0.0:2888:3888
> >
> > Any idea what I could try? Or maybe there's some better document
> > on how
> > to achieve this?
> >
> > Thank you
> >
> > Sebastian
> >
> >
> > --
> > DISCLAIMER
> > This email contains information that is confidential and which
> > may be
> > legally privileged. If you have received this email in error please
> >
> > notify the sender immediately and delete the email.
> > This email is intended
> > solely for the use of the intended recipient and you may not use or
> > disclose this email in any way.
> >
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>
Re: Enabling Auth between Zookeeper Servers
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
Hello Rakesh,
as mentioned in the other mail adding the "Server"to jaas.conf didn't help.
Here are the Configs and Logs (with the Server-part included):
jaas.conf:
QuorumServer {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test";
};
QuorumClient {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zookeeper"
password="test";
};
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test"
};
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zookeeper"
password="test";
};
zoo.cfg:
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/mnt/zk_data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
#
http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
dataLogDir=/mnt/zk_data_log
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=false
quorum.auth.serverRequireSasl=false
quorum.auth.learner.loginContext=QuorumLearner
quorum.auth.server.loginContext=QuorumServer
quorum.cnxn.threads.size=20
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
secureClientPort=2281
server.1=0.0.0.0:2888:3888
server.2=kafkad02.x.azure.com:2888:3888
server.3=kafkad03.x.azure.com:2888:3888
Server-Log:
Using config: /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
Loading configuration file "/opt/zookeeper-cluster/newrelic/./newrelic.yml"
Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: Using default
collector host: collector.newrelic.com
Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
Writing to log file:
/opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
com.newrelic.weave.weavepackage.NewClassAppender
(file:/opt/zookeeper-cluster/newrelic/newrelic.jar) to method
java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of
com.newrelic.weave.weavepackage.NewClassAppender
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
2020-02-11 18:43:59,257 [myid:] - INFO [main:QuorumPeerConfig@136] -
Reading configuration from:
/opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
2020-02-11 18:43:59,477 [myid:] - INFO
[main:QuorumPeer$QuorumServer@185] - Resolved hostname:
kafkad02.x.azure.com to address: kafkad02.x.azure.com/1.2.3.4
2020-02-11 18:43:59,477 [myid:] - INFO
[main:QuorumPeer$QuorumServer@185] - Resolved hostname: 0.0.0.0 to
address: /0.0.0.0
2020-02-11 18:43:59,666 [myid:] - INFO
[main:QuorumPeer$QuorumServer@185] - Resolved hostname:
kafkad03.x.azure.com to address: kafkad03.x.azure.com/1.2.3.5
2020-02-11 18:43:59,666 [myid:] - INFO [main:QuorumPeerConfig@398] -
Defaulting to majority quorums
2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@78]
- autopurge.snapRetainCount set to 3
2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@79]
- autopurge.purgeInterval set to 24
2020-02-11 18:43:59,732 [myid:1] - INFO
[PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
2020-02-11 18:43:59,749 [myid:1] - INFO [main:QuorumPeerMain@130] -
Starting quorum peer
2020-02-11 18:43:59,788 [myid:1] - INFO [main:ServerCnxnFactory@117] -
Using org.apache.zookeeper.server.NIOServerCnxnFactory as server
connection factory
2020-02-11 18:43:59,804 [myid:1] - INFO
[PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
'.20-02-11 18:43:59,826 [myid:1] - ERROR [main:ServerCnxnFactory@210] -
No JAAS configuration section named 'Server' was foundin
'/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
2020-02-11 18:43:59,827 [myid:1] - ERROR [main:QuorumPeerMain@92] -
Unexpected exception, exiting abnormally
java.io.IOException: No JAAS configuration section named 'Server' was
foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
'.
at
org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Best regards
Sebastian
On 12-Feb-20 2:36 AM, Rakesh Radhakrishnan wrote:
> >>>>>java.io.IOException: No JAAS configuration section named 'Server'
>
> I could see you have enabled client-server authentication as well. It
> looks to me that the error is coming from that. Please share the
> complete error logs to trace it.
> Have you configured "*Server*" section along with the "*QuorumServer*"
> and "*QuorumClient*" sections? If not, please configure "*Server*"
> section along with others and try it out.
>
> Reference:
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
> image.png
>
> Thanks,
> Rakesh
>
> On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz
> <sebastian.schmitz@propellerhead.co.nz
> <ma...@propellerhead.co.nz>> wrote:
>
> Hello,
>
> I'm currently looking into enabling the Auth between
> Zookeeper-Servers
> and found this documentation:
>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
>
> However, when I use the config from the document (for Digest-MD5)
> I get
> this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried
> because
> I thought using latest version could help:
> java.io.IOException: No JAAS configuration section named 'Server' was
> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>
> And of course that's right, because there's only QuorumServer and
> QuorumClient in the jaas.conf:
>
> jaas.conf:
> QuorumServer {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> user_zookeeper="test";
> };
>
> QuorumClient {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> username="zookeeper"
> password="test";
> };
>
> I also tried renaming the QuorumServer to just "Server". No change.
>
> My zoo.cfg:
> tickTime=2000
> initLimit=10
> syncLimit=5
> dataDir=/mnt/zk_data
> clientPort=2181
> dataLogDir=/mnt/zk_data_log
> autopurge.snapRetainCount=3
> autopurge.purgeInterval=24
> quorum.auth.enableSasl=true
> quorum.auth.learnerRequireSasl=false
> quorum.auth.serverRequireSasl=false
> quorum.auth.learner.loginContext=QuorumLearner
> quorum.auth.server.loginContext=QuorumServer
> quorum.cnxn.threads.size=20
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> secureClientPort=2281
> server.1=0.0.0.0:2888:3888
>
> Any idea what I could try? Or maybe there's some better document
> on how
> to achieve this?
>
> Thank you
>
> Sebastian
>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: Enabling Auth between Zookeeper Servers
Posted by Rakesh Radhakrishnan <ra...@apache.org>.
>>>>>java.io.IOException: No JAAS configuration section named 'Server'
I could see you have enabled client-server authentication as well. It looks
to me that the error is coming from that. Please share the complete error
logs to trace it.
Have you configured "*Server*" section along with the "*QuorumServer*" and "
*QuorumClient*" sections? If not, please configure "*Server*" section along
with others and try it out.
Reference:
https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
[image: image.png]
Thanks,
Rakesh
On Tue, Feb 11, 2020 at 7:26 AM Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> Hello,
>
> I'm currently looking into enabling the Auth between Zookeeper-Servers
> and found this documentation:
>
>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
>
> However, when I use the config from the document (for Digest-MD5) I get
> this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried because
> I thought using latest version could help:
> java.io.IOException: No JAAS configuration section named 'Server' was
> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>
> And of course that's right, because there's only QuorumServer and
> QuorumClient in the jaas.conf:
>
> jaas.conf:
> QuorumServer {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> user_zookeeper="test";
> };
>
> QuorumClient {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> username="zookeeper"
> password="test";
> };
>
> I also tried renaming the QuorumServer to just "Server". No change.
>
> My zoo.cfg:
> tickTime=2000
> initLimit=10
> syncLimit=5
> dataDir=/mnt/zk_data
> clientPort=2181
> dataLogDir=/mnt/zk_data_log
> autopurge.snapRetainCount=3
> autopurge.purgeInterval=24
> quorum.auth.enableSasl=true
> quorum.auth.learnerRequireSasl=false
> quorum.auth.serverRequireSasl=false
> quorum.auth.learner.loginContext=QuorumLearner
> quorum.auth.server.loginContext=QuorumServer
> quorum.cnxn.threads.size=20
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> secureClientPort=2281
> server.1=0.0.0.0:2888:3888
>
> Any idea what I could try? Or maybe there's some better document on how
> to achieve this?
>
> Thank you
>
> Sebastian
>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>
Re: Enabling Auth between Zookeeper Servers
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
Hello Mate,
thank you for your reply.
I just deployed it in our Dev-Environment, which has three nodes. But
when updating the first node it has the same problem.
The problem with not using 0.0.0.0 is that the configuration of Docker
only promotes the internal network to the containers and if I add the
internal IP to the config it just won't work as all three docker-hosts
have the same internal network and only port-mappings allow the services
to be reached from outside world. The FQDN doesn't match the internal IP
either, so the only solution for me is to use 0.0.0.0.
I set it to false as it's a rolling update of the three nodes. Will
change them to true on two more rolling restarts then.
That's what I don't understand either. Even adding the "Server" to the
jaas.conf didn't fix that exception.
Here are the Configs and Logs (with the Server-part included):
jaas.conf:
QuorumServer {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test";
};
QuorumClient {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zookeeper"
password="test";
};
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zookeeper="test"
};
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zookeeper"
password="test";
};
zoo.cfg:
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/mnt/zk_data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
dataLogDir=/mnt/zk_data_log
autopurge.snapRetainCount=3
autopurge.purgeInterval=24
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=false
quorum.auth.serverRequireSasl=false
quorum.auth.learner.loginContext=QuorumLearner
quorum.auth.server.loginContext=QuorumServer
quorum.cnxn.threads.size=20
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
secureClientPort=2281
server.1=0.0.0.0:2888:3888
server.2=kafkad02.x.azure.com:2888:3888
server.3=kafkad03.x.azure.com:2888:3888
Server-Log:
Using config: /opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
Loading configuration file "/opt/zookeeper-cluster/newrelic/./newrelic.yml"
Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: Using default
collector host: collector.newrelic.com
Feb 11, 2020 18:43:53 +0000 [1 1] com.newrelic INFO: New Relic Agent:
Writing to log file: /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.log
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
com.newrelic.weave.weavepackage.NewClassAppender
(file:/opt/zookeeper-cluster/newrelic/newrelic.jar) to method
java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of
com.newrelic.weave.weavepackage.NewClassAppender
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
2020-02-11 18:43:59,257 [myid:] - INFO [main:QuorumPeerConfig@136] -
Reading configuration from:
/opt/zookeeper-cluster/zookeeper/bin/../conf/zoo.cfg
2020-02-11 18:43:59,477 [myid:] - INFO
[main:QuorumPeer$QuorumServer@185] - Resolved hostname:
kafkad02.x.azure.com to address: kafkad02.x.azure.com/1.2.3.4
2020-02-11 18:43:59,477 [myid:] - INFO
[main:QuorumPeer$QuorumServer@185] - Resolved hostname: 0.0.0.0 to
address: /0.0.0.0
2020-02-11 18:43:59,666 [myid:] - INFO
[main:QuorumPeer$QuorumServer@185] - Resolved hostname:
kafkad03.x.azure.com to address: kafkad03.x.azure.com/1.2.3.5
2020-02-11 18:43:59,666 [myid:] - INFO [main:QuorumPeerConfig@398] -
Defaulting to majority quorums
2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@78]
- autopurge.snapRetainCount set to 3
2020-02-11 18:43:59,677 [myid:1] - INFO [main:DatadirCleanupManager@79]
- autopurge.purgeInterval set to 24
2020-02-11 18:43:59,732 [myid:1] - INFO
[PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
2020-02-11 18:43:59,749 [myid:1] - INFO [main:QuorumPeerMain@130] -
Starting quorum peer
2020-02-11 18:43:59,788 [myid:1] - INFO [main:ServerCnxnFactory@117] -
Using org.apache.zookeeper.server.NIOServerCnxnFactory as server
connection factory
2020-02-11 18:43:59,804 [myid:1] - INFO
[PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
'.20-02-11 18:43:59,826 [myid:1] - ERROR [main:ServerCnxnFactory@210] -
No JAAS configuration section named 'Server' was foundin
'/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
2020-02-11 18:43:59,827 [myid:1] - ERROR [main:QuorumPeerMain@92] -
Unexpected exception, exiting abnormally
java.io.IOException: No JAAS configuration section named 'Server' was
foundin '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
'.
at
org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:133)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Best regards
Sebastian
On 11-Feb-20 9:56 PM, Szalay-Bekő Máté wrote:
> Hello Sebastian,
>
> In general I think you did configure ZooKeeper just fine. A few remarks:
> - I am not sure how ZooKeeper server-server authentication is expected to
> work when you only use a single server. Would you mind trying to start a
> Quorum with e.g. 3 servers?
> - also, I think it is a good idea to avoid using 0.0.0.0 as hostname,
> especially if you would run the cluster on multiple different servers /
> docker containers. Try using the fully qualified domain name for a
> multi-server setup, or if you just test multiple ZooKeeper servers on the
> same machine, then just use 127.0.0.1. (maybe it has no effect in the
> current case, but for SSL or for dynamic reconfig it might be bad to use
> 0.0.0.0. Also I remember problems with rolling restarts when using 0.0.0.0
> in the config)
> - is there a reason why you set 'quorum.auth.learnerRequireSasl' and
> 'quorum.auth.serverRequireSasl' to false? Using false is usually good idea
> during rolling upgrade, but if you start a new cluster and want to use
> server-server authentication, then you can just set them to true.
>
> I don't understand why you got the exception " No JAAS configuration
> section named 'Server' was found" ... Setting the loginContext should have
> fixed that. If you still see the same issue with the 3 server setup, then
> can you please share the config files, the command how you start ZooKeeper
> and also the log files with us, so that we can look deeper?
>
> Kind regards,
> Mate
>
> On Tue, Feb 11, 2020 at 2:56 AM Sebastian Schmitz <
> sebastian.schmitz@propellerhead.co.nz> wrote:
>
>> Hello,
>>
>> I'm currently looking into enabling the Auth between Zookeeper-Servers
>> and found this documentation:
>>
>>
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
>>
>> However, when I use the config from the document (for Digest-MD5) I get
>> this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried because
>> I thought using latest version could help:
>> java.io.IOException: No JAAS configuration section named 'Server' was
>> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>>
>> And of course that's right, because there's only QuorumServer and
>> QuorumClient in the jaas.conf:
>>
>> jaas.conf:
>> QuorumServer {
>> org.apache.zookeeper.server.auth.DigestLoginModule required
>> user_zookeeper="test";
>> };
>>
>> QuorumClient {
>> org.apache.zookeeper.server.auth.DigestLoginModule required
>> username="zookeeper"
>> password="test";
>> };
>>
>> I also tried renaming the QuorumServer to just "Server". No change.
>>
>> My zoo.cfg:
>> tickTime=2000
>> initLimit=10
>> syncLimit=5
>> dataDir=/mnt/zk_data
>> clientPort=2181
>> dataLogDir=/mnt/zk_data_log
>> autopurge.snapRetainCount=3
>> autopurge.purgeInterval=24
>> quorum.auth.enableSasl=true
>> quorum.auth.learnerRequireSasl=false
>> quorum.auth.serverRequireSasl=false
>> quorum.auth.learner.loginContext=QuorumLearner
>> quorum.auth.server.loginContext=QuorumServer
>> quorum.cnxn.threads.size=20
>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>> secureClientPort=2281
>> server.1=0.0.0.0:2888:3888
>>
>> Any idea what I could try? Or maybe there's some better document on how
>> to achieve this?
>>
>> Thank you
>>
>> Sebastian
>>
>>
>> --
>> DISCLAIMER
>> This email contains information that is confidential and which
>> may be
>> legally privileged. If you have received this email in error please
>>
>> notify the sender immediately and delete the email.
>> This email is intended
>> solely for the use of the intended recipient and you may not use or
>> disclose this email in any way.
>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: Enabling Auth between Zookeeper Servers
Posted by Szalay-Bekő Máté <sz...@gmail.com>.
Hello Sebastian,
In general I think you did configure ZooKeeper just fine. A few remarks:
- I am not sure how ZooKeeper server-server authentication is expected to
work when you only use a single server. Would you mind trying to start a
Quorum with e.g. 3 servers?
- also, I think it is a good idea to avoid using 0.0.0.0 as hostname,
especially if you would run the cluster on multiple different servers /
docker containers. Try using the fully qualified domain name for a
multi-server setup, or if you just test multiple ZooKeeper servers on the
same machine, then just use 127.0.0.1. (maybe it has no effect in the
current case, but for SSL or for dynamic reconfig it might be bad to use
0.0.0.0. Also I remember problems with rolling restarts when using 0.0.0.0
in the config)
- is there a reason why you set 'quorum.auth.learnerRequireSasl' and
'quorum.auth.serverRequireSasl' to false? Using false is usually good idea
during rolling upgrade, but if you start a new cluster and want to use
server-server authentication, then you can just set them to true.
I don't understand why you got the exception " No JAAS configuration
section named 'Server' was found" ... Setting the loginContext should have
fixed that. If you still see the same issue with the 3 server setup, then
can you please share the config files, the command how you start ZooKeeper
and also the log files with us, so that we can look deeper?
Kind regards,
Mate
On Tue, Feb 11, 2020 at 2:56 AM Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> Hello,
>
> I'm currently looking into enabling the Auth between Zookeeper-Servers
> and found this documentation:
>
>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
>
> However, when I use the config from the document (for Digest-MD5) I get
> this exception in Zookeeper 3.4.14 and also 3.5.6, which I tried because
> I thought using latest version could help:
> java.io.IOException: No JAAS configuration section named 'Server' was
> found in '/opt/zookeeper-cluster/zookeeper/conf/jaas.conf
>
> And of course that's right, because there's only QuorumServer and
> QuorumClient in the jaas.conf:
>
> jaas.conf:
> QuorumServer {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> user_zookeeper="test";
> };
>
> QuorumClient {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> username="zookeeper"
> password="test";
> };
>
> I also tried renaming the QuorumServer to just "Server". No change.
>
> My zoo.cfg:
> tickTime=2000
> initLimit=10
> syncLimit=5
> dataDir=/mnt/zk_data
> clientPort=2181
> dataLogDir=/mnt/zk_data_log
> autopurge.snapRetainCount=3
> autopurge.purgeInterval=24
> quorum.auth.enableSasl=true
> quorum.auth.learnerRequireSasl=false
> quorum.auth.serverRequireSasl=false
> quorum.auth.learner.loginContext=QuorumLearner
> quorum.auth.server.loginContext=QuorumServer
> quorum.cnxn.threads.size=20
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> secureClientPort=2281
> server.1=0.0.0.0:2888:3888
>
> Any idea what I could try? Or maybe there's some better document on how
> to achieve this?
>
> Thank you
>
> Sebastian
>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>