You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Flavio Junqueira <fp...@apache.org> on 2016/10/21 14:34:12 UTC
Test plan for ZK-1045 - Call for volunteers
Michael Han posted an update to the test plan to the jira and I want to call the attention of the community to it. It is a big change that we need to be extra careful about because it is supposed to go to the 3.4 branch. It'd be great to have more folks in the community involved with the testing. If you have cycles and interest, please help with testing.
-Flavio
Re: Test plan for ZK-1045 - Call for volunteers
Posted by Flavio Junqueira <fp...@apache.org>.
Thanks a lot, Rakesh!
-Flavio
> On 25 Oct 2016, at 06:24, Rakesh Radhakrishnan <ra...@apache.org> wrote:
>
> I've uploaded feature document to the ZOOKEEPER-1045 jira, link =>
> https://goo.gl/cvzG8A. I hope this doc will help you to setup a secured zk
> cluster and start testing the feature. It would be really great to see the
> test/review/usability feedback. Thanks!
>
>
> Rakesh
>
> On Fri, Oct 21, 2016 at 8:04 PM, Flavio Junqueira <fp...@apache.org> wrote:
>
>> Michael Han posted an update to the test plan to the jira and I want to
>> call the attention of the community to it. It is a big change that we need
>> to be extra careful about because it is supposed to go to the 3.4 branch.
>> It'd be great to have more folks in the community involved with the
>> testing. If you have cycles and interest, please help with testing.
>>
>> -Flavio
Re: Test plan for ZK-1045 - Call for volunteers
Posted by Rakesh Radhakrishnan <ra...@apache.org>.
I've uploaded feature document to the ZOOKEEPER-1045 jira, link =>
https://goo.gl/cvzG8A. I hope this doc will help you to setup a secured zk
cluster and start testing the feature. It would be really great to see the
test/review/usability feedback. Thanks!
Rakesh
On Fri, Oct 21, 2016 at 8:04 PM, Flavio Junqueira <fp...@apache.org> wrote:
> Michael Han posted an update to the test plan to the jira and I want to
> call the attention of the community to it. It is a big change that we need
> to be extra careful about because it is supposed to go to the 3.4 branch.
> It'd be great to have more folks in the community involved with the
> testing. If you have cycles and interest, please help with testing.
>
> -Flavio
Re: Test plan for ZK-1045 - Call for volunteers
Posted by Patrick Hunt <ph...@apache.org>.
3.5.3-alpha. Sorry, mixing 3.4 and 3.5. :-)
Patrick
On Tue, Nov 29, 2016 at 11:53 AM, Raúl Gutiérrez Segalés <
rgs@itevenworks.net> wrote:
> On 18 November 2016 at 12:15, Patrick Hunt <ph...@apache.org> wrote:
>
> > As Flavio said originally on this thread this is a big change. Based on
> the
> > current status of the patch and the testing feedback it seems like we've
> > done significant work to ensure the quality of the change. Do folks feel
> > that there has been sufficient review/testing that we can commit this and
> > cut a 3.5.10 release? If not what specifically is left?
> >
>
> 3.5.10?
>
>
> -rgs
>
Re: Test plan for ZK-1045 - Call for volunteers
Posted by Raúl Gutiérrez Segalés <rg...@itevenworks.net>.
On 18 November 2016 at 12:15, Patrick Hunt <ph...@apache.org> wrote:
> As Flavio said originally on this thread this is a big change. Based on the
> current status of the patch and the testing feedback it seems like we've
> done significant work to ensure the quality of the change. Do folks feel
> that there has been sufficient review/testing that we can commit this and
> cut a 3.5.10 release? If not what specifically is left?
>
3.5.10?
-rgs
Re: Test plan for ZK-1045 - Call for volunteers
Posted by Rakesh Radhakrishnan <ra...@apache.org>.
Thank you very much for all your inputs, as well as the helpful discussions
in the community.
Sure, I will start 3.4.10 release plans. Since we have git repo in place,
it requires changes in the release procedure. Let me spend sometime trying
to understand this part.
Regards,
Rakesh
On Tue, Dec 6, 2016 at 4:52 AM, Patrick Hunt <ph...@apache.org> wrote:
> Shoot. Make that 3.4.10. Urg.
>
> Patrick
>
> On Mon, Dec 5, 2016 at 3:03 PM, Patrick Hunt <ph...@apache.org> wrote:
>
> > I like your second suggestion - let's cut an RC and give everyone the
> time
> > you are suggesting (2 weeks). That way folks can test and/or vote over
> the
> > period, rather than waiting the two weeks then starting the vote
> mechanics.
> > I know there are some people interested in getting access to 3.5.3-alpha
> > asap. Two birds.
> >
> > Patrick
> >
> > On Tue, Nov 29, 2016 at 3:11 PM, Flavio Junqueira <fp...@apache.org>
> wrote:
> >
> >> My suggestion is that we get this in and let it sit for a couple of
> weeks
> >> before cutting an RC. Alternatively, we could cut an RC and just give
> more
> >> time than the typical 2 weeks for people to test.
> >>
> >> -Flavio
> >>
> >> > On 29 Nov 2016, at 17:23, Patrick Hunt <ph...@apache.org> wrote:
> >> >
> >> > I did a bunch of manual testing last week. I tried running multiple
> >> cluster
> >> > sizes, tried running new server against old server, also went through
> >> the
> >> > rolling upgrade testing part of the document and that worked fine.
> >> afaict
> >> > at this point we're ready to commit. If there are any concerns please
> >> speak
> >> > up now as I intend to commit this soon.
> >> >
> >> > Patrick
> >> >
> >> > On Fri, Nov 18, 2016 at 12:15 PM, Patrick Hunt <ph...@apache.org>
> >> wrote:
> >> >
> >> >> As Flavio said originally on this thread this is a big change. Based
> on
> >> >> the current status of the patch and the testing feedback it seems
> like
> >> >> we've done significant work to ensure the quality of the change. Do
> >> folks
> >> >> feel that there has been sufficient review/testing that we can commit
> >> this
> >> >> and cut a 3.5.10 release? If not what specifically is left?
> >> >>
> >> >> Patrick
> >> >>
> >> >> On Wed, Nov 2, 2016 at 8:51 AM, Andrew Purtell <ap...@apache.org>
> >> >> wrote:
> >> >>
> >> >>>> My view on this-> Since ZK server
> >> >>>> already has "jaas.config" in place for client-server auth, IMHO to
> >> >>> continue
> >> >>>
> >> >>> Right, also build the client-server authentication context
> >> >>> programatically.
> >> >>>
> >> >>>> How about pushing the current
> >> >>>> approach as a first step and based on the community interests later
> >> we
> >> >>>> could enhance this feature by programmatically builds the JAAS
> >> context.
> >> >>>
> >> >>> Yes, this is what I was suggesting.
> >> >>>
> >> >>> Thanks for considering the idea.
> >> >>>
> >> >>>> 1) A bad host or bad Kerb principal is keep on trying to establish
> >> >>>> connection with the Quorum.
> >> >>>
> >> >>>> 2) Trigger FLE several times.
> >> >>>
> >> >>> Let me get back to you.
> >> >>>
> >> >>> I think it would also not be difficult to create a new unit test
> that
> >> >>> extends QuorumHammerTest (like ObserverQuorumHammerTest), sets up a
> >> >>> secure
> >> >>> configuration using the minikdc, and then uses QuorumBase methods to
> >> start
> >> >>> and stop quorum peers at random while the generated load is hitting
> >> the
> >> >>> quorum.
> >> >>>
> >> >>>
> >> >>> On Wed, Nov 2, 2016 at 7:15 AM, Rakesh Radhakrishnan <
> >> rakeshr@apache.org>
> >> >>> wrote:
> >> >>>
> >> >>>> Thanks a lot Andrew Purtell for your time and comments.
> >> >>>>
> >> >>>>>>>>> I like how Hadoop programmatically builds the JAAS context
> from
> >> its
> >> >>>>>>>>> configuration such that the JAAS configuration file and
> >> definition
> >> >>> of
> >> >>>>>>>>> system property pointing to same are not needed. Would be nice
> >> if
> >> >>> ZK
> >> >>>> could
> >> >>>>>>>>> optionally do the same, that would be one fewer config file to
> >> get
> >> >>>>>>>>> precisely right.
> >> >>>>
> >> >>>> Its really an interesting thought. My view on this-> Since ZK
> server
> >> >>>> already has "jaas.config" in place for client-server auth, IMHO to
> >> >>> continue
> >> >>>> with this jaas config and allows to configure the 'QuorumServer' &
> >> >>>> 'QuorumLearner' quorum auth sections. How about pushing the current
> >> >>>> approach as a first step and based on the community interests later
> >> we
> >> >>>> could enhance this feature by programmatically builds the JAAS
> >> context.
> >> >>>> Does this makes sense to you?
> >> >>>>
> >> >>>>
> >> >>>>>>>> Any suggestions on some kind of hammer test to apply now ?
> >> >>>> 1) A bad host or bad Kerb principal is keep on trying to establish
> >> >>>> connection with the Quorum. Mostly this will increase the load on
> >> >>> Leader ZK
> >> >>>> server and observe how the system behaves.
> >> >>>> 2) Trigger FLE several times. This can be done by finding out newly
> >> >>> elected
> >> >>>> Leader and kill it. Probably can do this many times.
> >> >>>>
> >> >>>>
> >> >>>> Dear Committers,
> >> >>>>
> >> >>>> Could you please help me pushing ZOOKEEPER-2479 this in. This would
> >> >>> help to
> >> >>>> evaluate the time taken for FLE(enable or disable auth) and used to
> >> >>> compare
> >> >>>> time taken in several runs programatically in scripts or so.
> Thanks!
> >> >>>>
> >> >>>> Thanks,
> >> >>>> Rakesh
> >> >>>>
> >> >>>> On Wed, Nov 2, 2016 at 6:59 AM, Andrew Purtell <
> apurtell@apache.org>
> >> >>>> wrote:
> >> >>>>
> >> >>>>> I would like to relay a working example of a patched 3.4.9 ZK
> >> cluster
> >> >>>>> running on one host (without containers). I can confirm mutual
> SASL
> >> >>> auth
> >> >>>>> among the quorum is required and enforced because when instances
> >> were
> >> >>>>> slightly misconfigured with incorrect principal strings they
> >> couldn't
> >> >>>>> participate in the quorum.
> >> >>>>>
> >> >>>>> 1. Assign extra loopback addresses. Example below is what you need
> >> to
> >> >>> do
> >> >>>>> for FreeBSD:
> >> >>>>>
> >> >>>>> $ sudo ifconfig lo0 127.0.1.1 netmask 255.255.255.0 alias
> >> >>>>> $ sudo ifconfig lo0 127.0.2.1 netmask 255.255.255.0 alias
> >> >>>>> $ sudo ifconfig lo0 127.0.3.1 netmask 255.255.255.0 alias
> >> >>>>>
> >> >>>>> 2. Create Kerberos principals. Example below is for Heimdal, MIT
> >> will
> >> >>> be
> >> >>>>> similar:
> >> >>>>>
> >> >>>>> $ sudo kadmin -l
> >> >>>>>> add --random-key zookeeper
> >> >>>>>> add --random-key zookeeper/127.0.1.1
> >> >>>>>> add --random-key zookeeper/127.0.2.1
> >> >>>>>> add --random-key zookeeper/127.0.3.1
> >> >>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper
> >> >>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
> >> >>> 127.0.1.1
> >> >>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
> >> >>> 127.0.2.1
> >> >>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
> >> >>> 127.0.3.1
> >> >>>>>> exit
> >> >>>>>
> >> >>>>> 3. Patch ZK with 1045, build a tarball, then unpack it into three
> >> >>> install
> >> >>>>> directories. Mine are /var/tmp/test/{1,2,3}.
> >> >>>>>
> >> >>>>> 4. Initialize 'myid' files:
> >> >>>>>
> >> >>>>> $ mkdir /var/tmp/test/1/data
> >> >>>>> $ echo 1 > /var/tmp/test/1/data/myid
> >> >>>>> $ mkdir /var/tmp/test/2/data
> >> >>>>> $ echo 2 > /var/tmp/test/2/data/myid
> >> >>>>> $ mkdir /var/tmp/test/3/data
> >> >>>>> $ echo 3 > /var/tmp/test/3/data/myid
> >> >>>>>
> >> >>>>> 5. Create configuration files for each instance. Below are
> examples
> >> >>> for
> >> >>>>> instance 1. You will need to make substitutions for your Kerberos
> >> >>> realm
> >> >>>>> (mine is LOCAL), and for instances 2 and 3 change the path to the
> >> JAAS
> >> >>>>> configuration file, path to data directory, and bind address for
> >> >>>>> clientPortAddress as appropriate.
> >> >>>>>
> >> >>>>> conf/java.env:
> >> >>>>>
> >> >>>>> export
> >> >>>>> JVMFLAGS="-Djava.security.auth.login.config=/var/tmp/
> >> >>>> test/1/conf/jaas.conf
> >> >>>>> -Djavax.security.auth.useSubjectCredsOnly=false"
> >> >>>>>
> >> >>>>> conf/jaas.conf:
> >> >>>>>
> >> >>>>> QuorumServer {
> >> >>>>> com.sun.security.auth.module.Krb5LoginModule required
> >> >>>>> useKeyTab=true
> >> >>>>> keyTab="/var/tmp/test/zookeeper.keytab"
> >> >>>>> storeKey=true
> >> >>>>> useTicketCache=false
> >> >>>>> debug=false
> >> >>>>> principal="zookeeper/127.0.1.1@LOCAL";
> >> >>>>> };
> >> >>>>>
> >> >>>>> QuorumLearner {
> >> >>>>> com.sun.security.auth.module.Krb5LoginModule required
> >> >>>>> useKeyTab=true
> >> >>>>> keyTab="/var/tmp/test/zookeeper.keytab"
> >> >>>>> storeKey=true
> >> >>>>> useTicketCache=false
> >> >>>>> debug=false
> >> >>>>> principal="zookeeper/127.0.1.1@LOCAL";
> >> >>>>> };
> >> >>>>>
> >> >>>>> conf/zoo.cfg:
> >> >>>>>
> >> >>>>> tickTime=2000
> >> >>>>> initLimit=10
> >> >>>>> syncLimit=5
> >> >>>>> dataDir=/var/tmp/test/zk/1/data
> >> >>>>> clientPort=2181
> >> >>>>> clientPortAddress=127.0.1.1
> >> >>>>>
> >> >>>>> quorum.auth.enableSasl=true
> >> >>>>> quorum.auth.learnerRequireSasl=true
> >> >>>>> quorum.auth.serverRequireSasl=true
> >> >>>>> quorum.auth.learner.loginContext=QuorumLearner
> >> >>>>> quorum.auth.server.loginContext=QuorumServer
> >> >>>>> quorum.auth.kerberos.servicePrincipal=zookeeper/_HOST
> >> >>>>>
> >> >>>>> server.1=127.0.1.1:2888:3888
> >> >>>>> server.2=127.0.2.1:2888:3888
> >> >>>>> server.3=127.0.3.1:2888:3888
> >> >>>>>
> >> >>>>> 5. Launch the three instances in the foreground in separate
> >> terminals.
> >> >>>>> Example for instance 1:
> >> >>>>>
> >> >>>>> $ cd /var/tmp/test/1
> >> >>>>> $ bash ./bin/zkServer.sh start-foreground
> >> >>>>>
> >> >>>>> Having done all this, I can see successful authentication and
> >> >>> bootstrap
> >> >>>> of
> >> >>>>> a quorum.
> >> >>>>>
> >> >>>>> This example shares a keytab. No need to do that if you'd like to
> be
> >> >>>>> pedantic. Clone the keytab file into the separate conf directories
> >> and
> >> >>>>> update the jaas.conf files.
> >> >>>>>
> >> >>>>> I like how Hadoop programmatically builds the JAAS context from
> its
> >> >>>>> configuration such that the JAAS configuration file and definition
> >> of
> >> >>>>> system property pointing to same are not needed. Would be nice if
> ZK
> >> >>>> could
> >> >>>>> optionally do the same, that would be one fewer config file to get
> >> >>>>> precisely right.
> >> >>>>>
> >> >>>>> Any suggestions on some kind of hammer test to apply now ?
> >> >>>>>
> >> >>>>>
> >> >>>>> On Fri, Oct 21, 2016 at 7:34 AM, Flavio Junqueira <fpj@apache.org
> >
> >> >>>> wrote:
> >> >>>>>
> >> >>>>>> Michael Han posted an update to the test plan to the jira and I
> >> >>> want to
> >> >>>>>> call the attention of the community to it. It is a big change
> that
> >> >>> we
> >> >>>>> need
> >> >>>>>> to be extra careful about because it is supposed to go to the 3.4
> >> >>>> branch.
> >> >>>>>> It'd be great to have more folks in the community involved with
> the
> >> >>>>>> testing. If you have cycles and interest, please help with
> testing.
> >> >>>>>>
> >> >>>>>> -Flavio
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> --
> >> >>>>> Best regards,
> >> >>>>>
> >> >>>>> - Andy
> >> >>>>>
> >> >>>>> Problems worthy of attack prove their worth by hitting back. -
> Piet
> >> >>> Hein
> >> >>>>> (via Tom White)
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Best regards,
> >> >>>
> >> >>> - Andy
> >> >>>
> >> >>> Problems worthy of attack prove their worth by hitting back. - Piet
> >> Hein
> >> >>> (via Tom White)
> >> >>>
> >> >>
> >> >>
> >>
> >>
> >
>
Re: Test plan for ZK-1045 - Call for volunteers
Posted by Patrick Hunt <ph...@apache.org>.
Shoot. Make that 3.4.10. Urg.
Patrick
On Mon, Dec 5, 2016 at 3:03 PM, Patrick Hunt <ph...@apache.org> wrote:
> I like your second suggestion - let's cut an RC and give everyone the time
> you are suggesting (2 weeks). That way folks can test and/or vote over the
> period, rather than waiting the two weeks then starting the vote mechanics.
> I know there are some people interested in getting access to 3.5.3-alpha
> asap. Two birds.
>
> Patrick
>
> On Tue, Nov 29, 2016 at 3:11 PM, Flavio Junqueira <fp...@apache.org> wrote:
>
>> My suggestion is that we get this in and let it sit for a couple of weeks
>> before cutting an RC. Alternatively, we could cut an RC and just give more
>> time than the typical 2 weeks for people to test.
>>
>> -Flavio
>>
>> > On 29 Nov 2016, at 17:23, Patrick Hunt <ph...@apache.org> wrote:
>> >
>> > I did a bunch of manual testing last week. I tried running multiple
>> cluster
>> > sizes, tried running new server against old server, also went through
>> the
>> > rolling upgrade testing part of the document and that worked fine.
>> afaict
>> > at this point we're ready to commit. If there are any concerns please
>> speak
>> > up now as I intend to commit this soon.
>> >
>> > Patrick
>> >
>> > On Fri, Nov 18, 2016 at 12:15 PM, Patrick Hunt <ph...@apache.org>
>> wrote:
>> >
>> >> As Flavio said originally on this thread this is a big change. Based on
>> >> the current status of the patch and the testing feedback it seems like
>> >> we've done significant work to ensure the quality of the change. Do
>> folks
>> >> feel that there has been sufficient review/testing that we can commit
>> this
>> >> and cut a 3.5.10 release? If not what specifically is left?
>> >>
>> >> Patrick
>> >>
>> >> On Wed, Nov 2, 2016 at 8:51 AM, Andrew Purtell <ap...@apache.org>
>> >> wrote:
>> >>
>> >>>> My view on this-> Since ZK server
>> >>>> already has "jaas.config" in place for client-server auth, IMHO to
>> >>> continue
>> >>>
>> >>> Right, also build the client-server authentication context
>> >>> programatically.
>> >>>
>> >>>> How about pushing the current
>> >>>> approach as a first step and based on the community interests later
>> we
>> >>>> could enhance this feature by programmatically builds the JAAS
>> context.
>> >>>
>> >>> Yes, this is what I was suggesting.
>> >>>
>> >>> Thanks for considering the idea.
>> >>>
>> >>>> 1) A bad host or bad Kerb principal is keep on trying to establish
>> >>>> connection with the Quorum.
>> >>>
>> >>>> 2) Trigger FLE several times.
>> >>>
>> >>> Let me get back to you.
>> >>>
>> >>> I think it would also not be difficult to create a new unit test that
>> >>> extends QuorumHammerTest (like ObserverQuorumHammerTest), sets up a
>> >>> secure
>> >>> configuration using the minikdc, and then uses QuorumBase methods to
>> start
>> >>> and stop quorum peers at random while the generated load is hitting
>> the
>> >>> quorum.
>> >>>
>> >>>
>> >>> On Wed, Nov 2, 2016 at 7:15 AM, Rakesh Radhakrishnan <
>> rakeshr@apache.org>
>> >>> wrote:
>> >>>
>> >>>> Thanks a lot Andrew Purtell for your time and comments.
>> >>>>
>> >>>>>>>>> I like how Hadoop programmatically builds the JAAS context from
>> its
>> >>>>>>>>> configuration such that the JAAS configuration file and
>> definition
>> >>> of
>> >>>>>>>>> system property pointing to same are not needed. Would be nice
>> if
>> >>> ZK
>> >>>> could
>> >>>>>>>>> optionally do the same, that would be one fewer config file to
>> get
>> >>>>>>>>> precisely right.
>> >>>>
>> >>>> Its really an interesting thought. My view on this-> Since ZK server
>> >>>> already has "jaas.config" in place for client-server auth, IMHO to
>> >>> continue
>> >>>> with this jaas config and allows to configure the 'QuorumServer' &
>> >>>> 'QuorumLearner' quorum auth sections. How about pushing the current
>> >>>> approach as a first step and based on the community interests later
>> we
>> >>>> could enhance this feature by programmatically builds the JAAS
>> context.
>> >>>> Does this makes sense to you?
>> >>>>
>> >>>>
>> >>>>>>>> Any suggestions on some kind of hammer test to apply now ?
>> >>>> 1) A bad host or bad Kerb principal is keep on trying to establish
>> >>>> connection with the Quorum. Mostly this will increase the load on
>> >>> Leader ZK
>> >>>> server and observe how the system behaves.
>> >>>> 2) Trigger FLE several times. This can be done by finding out newly
>> >>> elected
>> >>>> Leader and kill it. Probably can do this many times.
>> >>>>
>> >>>>
>> >>>> Dear Committers,
>> >>>>
>> >>>> Could you please help me pushing ZOOKEEPER-2479 this in. This would
>> >>> help to
>> >>>> evaluate the time taken for FLE(enable or disable auth) and used to
>> >>> compare
>> >>>> time taken in several runs programatically in scripts or so. Thanks!
>> >>>>
>> >>>> Thanks,
>> >>>> Rakesh
>> >>>>
>> >>>> On Wed, Nov 2, 2016 at 6:59 AM, Andrew Purtell <ap...@apache.org>
>> >>>> wrote:
>> >>>>
>> >>>>> I would like to relay a working example of a patched 3.4.9 ZK
>> cluster
>> >>>>> running on one host (without containers). I can confirm mutual SASL
>> >>> auth
>> >>>>> among the quorum is required and enforced because when instances
>> were
>> >>>>> slightly misconfigured with incorrect principal strings they
>> couldn't
>> >>>>> participate in the quorum.
>> >>>>>
>> >>>>> 1. Assign extra loopback addresses. Example below is what you need
>> to
>> >>> do
>> >>>>> for FreeBSD:
>> >>>>>
>> >>>>> $ sudo ifconfig lo0 127.0.1.1 netmask 255.255.255.0 alias
>> >>>>> $ sudo ifconfig lo0 127.0.2.1 netmask 255.255.255.0 alias
>> >>>>> $ sudo ifconfig lo0 127.0.3.1 netmask 255.255.255.0 alias
>> >>>>>
>> >>>>> 2. Create Kerberos principals. Example below is for Heimdal, MIT
>> will
>> >>> be
>> >>>>> similar:
>> >>>>>
>> >>>>> $ sudo kadmin -l
>> >>>>>> add --random-key zookeeper
>> >>>>>> add --random-key zookeeper/127.0.1.1
>> >>>>>> add --random-key zookeeper/127.0.2.1
>> >>>>>> add --random-key zookeeper/127.0.3.1
>> >>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper
>> >>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
>> >>> 127.0.1.1
>> >>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
>> >>> 127.0.2.1
>> >>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
>> >>> 127.0.3.1
>> >>>>>> exit
>> >>>>>
>> >>>>> 3. Patch ZK with 1045, build a tarball, then unpack it into three
>> >>> install
>> >>>>> directories. Mine are /var/tmp/test/{1,2,3}.
>> >>>>>
>> >>>>> 4. Initialize 'myid' files:
>> >>>>>
>> >>>>> $ mkdir /var/tmp/test/1/data
>> >>>>> $ echo 1 > /var/tmp/test/1/data/myid
>> >>>>> $ mkdir /var/tmp/test/2/data
>> >>>>> $ echo 2 > /var/tmp/test/2/data/myid
>> >>>>> $ mkdir /var/tmp/test/3/data
>> >>>>> $ echo 3 > /var/tmp/test/3/data/myid
>> >>>>>
>> >>>>> 5. Create configuration files for each instance. Below are examples
>> >>> for
>> >>>>> instance 1. You will need to make substitutions for your Kerberos
>> >>> realm
>> >>>>> (mine is LOCAL), and for instances 2 and 3 change the path to the
>> JAAS
>> >>>>> configuration file, path to data directory, and bind address for
>> >>>>> clientPortAddress as appropriate.
>> >>>>>
>> >>>>> conf/java.env:
>> >>>>>
>> >>>>> export
>> >>>>> JVMFLAGS="-Djava.security.auth.login.config=/var/tmp/
>> >>>> test/1/conf/jaas.conf
>> >>>>> -Djavax.security.auth.useSubjectCredsOnly=false"
>> >>>>>
>> >>>>> conf/jaas.conf:
>> >>>>>
>> >>>>> QuorumServer {
>> >>>>> com.sun.security.auth.module.Krb5LoginModule required
>> >>>>> useKeyTab=true
>> >>>>> keyTab="/var/tmp/test/zookeeper.keytab"
>> >>>>> storeKey=true
>> >>>>> useTicketCache=false
>> >>>>> debug=false
>> >>>>> principal="zookeeper/127.0.1.1@LOCAL";
>> >>>>> };
>> >>>>>
>> >>>>> QuorumLearner {
>> >>>>> com.sun.security.auth.module.Krb5LoginModule required
>> >>>>> useKeyTab=true
>> >>>>> keyTab="/var/tmp/test/zookeeper.keytab"
>> >>>>> storeKey=true
>> >>>>> useTicketCache=false
>> >>>>> debug=false
>> >>>>> principal="zookeeper/127.0.1.1@LOCAL";
>> >>>>> };
>> >>>>>
>> >>>>> conf/zoo.cfg:
>> >>>>>
>> >>>>> tickTime=2000
>> >>>>> initLimit=10
>> >>>>> syncLimit=5
>> >>>>> dataDir=/var/tmp/test/zk/1/data
>> >>>>> clientPort=2181
>> >>>>> clientPortAddress=127.0.1.1
>> >>>>>
>> >>>>> quorum.auth.enableSasl=true
>> >>>>> quorum.auth.learnerRequireSasl=true
>> >>>>> quorum.auth.serverRequireSasl=true
>> >>>>> quorum.auth.learner.loginContext=QuorumLearner
>> >>>>> quorum.auth.server.loginContext=QuorumServer
>> >>>>> quorum.auth.kerberos.servicePrincipal=zookeeper/_HOST
>> >>>>>
>> >>>>> server.1=127.0.1.1:2888:3888
>> >>>>> server.2=127.0.2.1:2888:3888
>> >>>>> server.3=127.0.3.1:2888:3888
>> >>>>>
>> >>>>> 5. Launch the three instances in the foreground in separate
>> terminals.
>> >>>>> Example for instance 1:
>> >>>>>
>> >>>>> $ cd /var/tmp/test/1
>> >>>>> $ bash ./bin/zkServer.sh start-foreground
>> >>>>>
>> >>>>> Having done all this, I can see successful authentication and
>> >>> bootstrap
>> >>>> of
>> >>>>> a quorum.
>> >>>>>
>> >>>>> This example shares a keytab. No need to do that if you'd like to be
>> >>>>> pedantic. Clone the keytab file into the separate conf directories
>> and
>> >>>>> update the jaas.conf files.
>> >>>>>
>> >>>>> I like how Hadoop programmatically builds the JAAS context from its
>> >>>>> configuration such that the JAAS configuration file and definition
>> of
>> >>>>> system property pointing to same are not needed. Would be nice if ZK
>> >>>> could
>> >>>>> optionally do the same, that would be one fewer config file to get
>> >>>>> precisely right.
>> >>>>>
>> >>>>> Any suggestions on some kind of hammer test to apply now ?
>> >>>>>
>> >>>>>
>> >>>>> On Fri, Oct 21, 2016 at 7:34 AM, Flavio Junqueira <fp...@apache.org>
>> >>>> wrote:
>> >>>>>
>> >>>>>> Michael Han posted an update to the test plan to the jira and I
>> >>> want to
>> >>>>>> call the attention of the community to it. It is a big change that
>> >>> we
>> >>>>> need
>> >>>>>> to be extra careful about because it is supposed to go to the 3.4
>> >>>> branch.
>> >>>>>> It'd be great to have more folks in the community involved with the
>> >>>>>> testing. If you have cycles and interest, please help with testing.
>> >>>>>>
>> >>>>>> -Flavio
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Best regards,
>> >>>>>
>> >>>>> - Andy
>> >>>>>
>> >>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>> >>> Hein
>> >>>>> (via Tom White)
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Best regards,
>> >>>
>> >>> - Andy
>> >>>
>> >>> Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein
>> >>> (via Tom White)
>> >>>
>> >>
>> >>
>>
>>
>
Re: Test plan for ZK-1045 - Call for volunteers
Posted by Patrick Hunt <ph...@apache.org>.
I like your second suggestion - let's cut an RC and give everyone the time
you are suggesting (2 weeks). That way folks can test and/or vote over the
period, rather than waiting the two weeks then starting the vote mechanics.
I know there are some people interested in getting access to 3.5.3-alpha
asap. Two birds.
Patrick
On Tue, Nov 29, 2016 at 3:11 PM, Flavio Junqueira <fp...@apache.org> wrote:
> My suggestion is that we get this in and let it sit for a couple of weeks
> before cutting an RC. Alternatively, we could cut an RC and just give more
> time than the typical 2 weeks for people to test.
>
> -Flavio
>
> > On 29 Nov 2016, at 17:23, Patrick Hunt <ph...@apache.org> wrote:
> >
> > I did a bunch of manual testing last week. I tried running multiple
> cluster
> > sizes, tried running new server against old server, also went through the
> > rolling upgrade testing part of the document and that worked fine. afaict
> > at this point we're ready to commit. If there are any concerns please
> speak
> > up now as I intend to commit this soon.
> >
> > Patrick
> >
> > On Fri, Nov 18, 2016 at 12:15 PM, Patrick Hunt <ph...@apache.org> wrote:
> >
> >> As Flavio said originally on this thread this is a big change. Based on
> >> the current status of the patch and the testing feedback it seems like
> >> we've done significant work to ensure the quality of the change. Do
> folks
> >> feel that there has been sufficient review/testing that we can commit
> this
> >> and cut a 3.5.10 release? If not what specifically is left?
> >>
> >> Patrick
> >>
> >> On Wed, Nov 2, 2016 at 8:51 AM, Andrew Purtell <ap...@apache.org>
> >> wrote:
> >>
> >>>> My view on this-> Since ZK server
> >>>> already has "jaas.config" in place for client-server auth, IMHO to
> >>> continue
> >>>
> >>> Right, also build the client-server authentication context
> >>> programatically.
> >>>
> >>>> How about pushing the current
> >>>> approach as a first step and based on the community interests later we
> >>>> could enhance this feature by programmatically builds the JAAS
> context.
> >>>
> >>> Yes, this is what I was suggesting.
> >>>
> >>> Thanks for considering the idea.
> >>>
> >>>> 1) A bad host or bad Kerb principal is keep on trying to establish
> >>>> connection with the Quorum.
> >>>
> >>>> 2) Trigger FLE several times.
> >>>
> >>> Let me get back to you.
> >>>
> >>> I think it would also not be difficult to create a new unit test that
> >>> extends QuorumHammerTest (like ObserverQuorumHammerTest), sets up a
> >>> secure
> >>> configuration using the minikdc, and then uses QuorumBase methods to
> start
> >>> and stop quorum peers at random while the generated load is hitting the
> >>> quorum.
> >>>
> >>>
> >>> On Wed, Nov 2, 2016 at 7:15 AM, Rakesh Radhakrishnan <
> rakeshr@apache.org>
> >>> wrote:
> >>>
> >>>> Thanks a lot Andrew Purtell for your time and comments.
> >>>>
> >>>>>>>>> I like how Hadoop programmatically builds the JAAS context from
> its
> >>>>>>>>> configuration such that the JAAS configuration file and
> definition
> >>> of
> >>>>>>>>> system property pointing to same are not needed. Would be nice if
> >>> ZK
> >>>> could
> >>>>>>>>> optionally do the same, that would be one fewer config file to
> get
> >>>>>>>>> precisely right.
> >>>>
> >>>> Its really an interesting thought. My view on this-> Since ZK server
> >>>> already has "jaas.config" in place for client-server auth, IMHO to
> >>> continue
> >>>> with this jaas config and allows to configure the 'QuorumServer' &
> >>>> 'QuorumLearner' quorum auth sections. How about pushing the current
> >>>> approach as a first step and based on the community interests later we
> >>>> could enhance this feature by programmatically builds the JAAS
> context.
> >>>> Does this makes sense to you?
> >>>>
> >>>>
> >>>>>>>> Any suggestions on some kind of hammer test to apply now ?
> >>>> 1) A bad host or bad Kerb principal is keep on trying to establish
> >>>> connection with the Quorum. Mostly this will increase the load on
> >>> Leader ZK
> >>>> server and observe how the system behaves.
> >>>> 2) Trigger FLE several times. This can be done by finding out newly
> >>> elected
> >>>> Leader and kill it. Probably can do this many times.
> >>>>
> >>>>
> >>>> Dear Committers,
> >>>>
> >>>> Could you please help me pushing ZOOKEEPER-2479 this in. This would
> >>> help to
> >>>> evaluate the time taken for FLE(enable or disable auth) and used to
> >>> compare
> >>>> time taken in several runs programatically in scripts or so. Thanks!
> >>>>
> >>>> Thanks,
> >>>> Rakesh
> >>>>
> >>>> On Wed, Nov 2, 2016 at 6:59 AM, Andrew Purtell <ap...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> I would like to relay a working example of a patched 3.4.9 ZK cluster
> >>>>> running on one host (without containers). I can confirm mutual SASL
> >>> auth
> >>>>> among the quorum is required and enforced because when instances were
> >>>>> slightly misconfigured with incorrect principal strings they couldn't
> >>>>> participate in the quorum.
> >>>>>
> >>>>> 1. Assign extra loopback addresses. Example below is what you need to
> >>> do
> >>>>> for FreeBSD:
> >>>>>
> >>>>> $ sudo ifconfig lo0 127.0.1.1 netmask 255.255.255.0 alias
> >>>>> $ sudo ifconfig lo0 127.0.2.1 netmask 255.255.255.0 alias
> >>>>> $ sudo ifconfig lo0 127.0.3.1 netmask 255.255.255.0 alias
> >>>>>
> >>>>> 2. Create Kerberos principals. Example below is for Heimdal, MIT will
> >>> be
> >>>>> similar:
> >>>>>
> >>>>> $ sudo kadmin -l
> >>>>>> add --random-key zookeeper
> >>>>>> add --random-key zookeeper/127.0.1.1
> >>>>>> add --random-key zookeeper/127.0.2.1
> >>>>>> add --random-key zookeeper/127.0.3.1
> >>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper
> >>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
> >>> 127.0.1.1
> >>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
> >>> 127.0.2.1
> >>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
> >>> 127.0.3.1
> >>>>>> exit
> >>>>>
> >>>>> 3. Patch ZK with 1045, build a tarball, then unpack it into three
> >>> install
> >>>>> directories. Mine are /var/tmp/test/{1,2,3}.
> >>>>>
> >>>>> 4. Initialize 'myid' files:
> >>>>>
> >>>>> $ mkdir /var/tmp/test/1/data
> >>>>> $ echo 1 > /var/tmp/test/1/data/myid
> >>>>> $ mkdir /var/tmp/test/2/data
> >>>>> $ echo 2 > /var/tmp/test/2/data/myid
> >>>>> $ mkdir /var/tmp/test/3/data
> >>>>> $ echo 3 > /var/tmp/test/3/data/myid
> >>>>>
> >>>>> 5. Create configuration files for each instance. Below are examples
> >>> for
> >>>>> instance 1. You will need to make substitutions for your Kerberos
> >>> realm
> >>>>> (mine is LOCAL), and for instances 2 and 3 change the path to the
> JAAS
> >>>>> configuration file, path to data directory, and bind address for
> >>>>> clientPortAddress as appropriate.
> >>>>>
> >>>>> conf/java.env:
> >>>>>
> >>>>> export
> >>>>> JVMFLAGS="-Djava.security.auth.login.config=/var/tmp/
> >>>> test/1/conf/jaas.conf
> >>>>> -Djavax.security.auth.useSubjectCredsOnly=false"
> >>>>>
> >>>>> conf/jaas.conf:
> >>>>>
> >>>>> QuorumServer {
> >>>>> com.sun.security.auth.module.Krb5LoginModule required
> >>>>> useKeyTab=true
> >>>>> keyTab="/var/tmp/test/zookeeper.keytab"
> >>>>> storeKey=true
> >>>>> useTicketCache=false
> >>>>> debug=false
> >>>>> principal="zookeeper/127.0.1.1@LOCAL";
> >>>>> };
> >>>>>
> >>>>> QuorumLearner {
> >>>>> com.sun.security.auth.module.Krb5LoginModule required
> >>>>> useKeyTab=true
> >>>>> keyTab="/var/tmp/test/zookeeper.keytab"
> >>>>> storeKey=true
> >>>>> useTicketCache=false
> >>>>> debug=false
> >>>>> principal="zookeeper/127.0.1.1@LOCAL";
> >>>>> };
> >>>>>
> >>>>> conf/zoo.cfg:
> >>>>>
> >>>>> tickTime=2000
> >>>>> initLimit=10
> >>>>> syncLimit=5
> >>>>> dataDir=/var/tmp/test/zk/1/data
> >>>>> clientPort=2181
> >>>>> clientPortAddress=127.0.1.1
> >>>>>
> >>>>> quorum.auth.enableSasl=true
> >>>>> quorum.auth.learnerRequireSasl=true
> >>>>> quorum.auth.serverRequireSasl=true
> >>>>> quorum.auth.learner.loginContext=QuorumLearner
> >>>>> quorum.auth.server.loginContext=QuorumServer
> >>>>> quorum.auth.kerberos.servicePrincipal=zookeeper/_HOST
> >>>>>
> >>>>> server.1=127.0.1.1:2888:3888
> >>>>> server.2=127.0.2.1:2888:3888
> >>>>> server.3=127.0.3.1:2888:3888
> >>>>>
> >>>>> 5. Launch the three instances in the foreground in separate
> terminals.
> >>>>> Example for instance 1:
> >>>>>
> >>>>> $ cd /var/tmp/test/1
> >>>>> $ bash ./bin/zkServer.sh start-foreground
> >>>>>
> >>>>> Having done all this, I can see successful authentication and
> >>> bootstrap
> >>>> of
> >>>>> a quorum.
> >>>>>
> >>>>> This example shares a keytab. No need to do that if you'd like to be
> >>>>> pedantic. Clone the keytab file into the separate conf directories
> and
> >>>>> update the jaas.conf files.
> >>>>>
> >>>>> I like how Hadoop programmatically builds the JAAS context from its
> >>>>> configuration such that the JAAS configuration file and definition of
> >>>>> system property pointing to same are not needed. Would be nice if ZK
> >>>> could
> >>>>> optionally do the same, that would be one fewer config file to get
> >>>>> precisely right.
> >>>>>
> >>>>> Any suggestions on some kind of hammer test to apply now ?
> >>>>>
> >>>>>
> >>>>> On Fri, Oct 21, 2016 at 7:34 AM, Flavio Junqueira <fp...@apache.org>
> >>>> wrote:
> >>>>>
> >>>>>> Michael Han posted an update to the test plan to the jira and I
> >>> want to
> >>>>>> call the attention of the community to it. It is a big change that
> >>> we
> >>>>> need
> >>>>>> to be extra careful about because it is supposed to go to the 3.4
> >>>> branch.
> >>>>>> It'd be great to have more folks in the community involved with the
> >>>>>> testing. If you have cycles and interest, please help with testing.
> >>>>>>
> >>>>>> -Flavio
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Best regards,
> >>>>>
> >>>>> - Andy
> >>>>>
> >>>>> Problems worthy of attack prove their worth by hitting back. - Piet
> >>> Hein
> >>>>> (via Tom White)
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Best regards,
> >>>
> >>> - Andy
> >>>
> >>> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> >>> (via Tom White)
> >>>
> >>
> >>
>
>
Re: Test plan for ZK-1045 - Call for volunteers
Posted by Flavio Junqueira <fp...@apache.org>.
My suggestion is that we get this in and let it sit for a couple of weeks before cutting an RC. Alternatively, we could cut an RC and just give more time than the typical 2 weeks for people to test.
-Flavio
> On 29 Nov 2016, at 17:23, Patrick Hunt <ph...@apache.org> wrote:
>
> I did a bunch of manual testing last week. I tried running multiple cluster
> sizes, tried running new server against old server, also went through the
> rolling upgrade testing part of the document and that worked fine. afaict
> at this point we're ready to commit. If there are any concerns please speak
> up now as I intend to commit this soon.
>
> Patrick
>
> On Fri, Nov 18, 2016 at 12:15 PM, Patrick Hunt <ph...@apache.org> wrote:
>
>> As Flavio said originally on this thread this is a big change. Based on
>> the current status of the patch and the testing feedback it seems like
>> we've done significant work to ensure the quality of the change. Do folks
>> feel that there has been sufficient review/testing that we can commit this
>> and cut a 3.5.10 release? If not what specifically is left?
>>
>> Patrick
>>
>> On Wed, Nov 2, 2016 at 8:51 AM, Andrew Purtell <ap...@apache.org>
>> wrote:
>>
>>>> My view on this-> Since ZK server
>>>> already has "jaas.config" in place for client-server auth, IMHO to
>>> continue
>>>
>>> Right, also build the client-server authentication context
>>> programatically.
>>>
>>>> How about pushing the current
>>>> approach as a first step and based on the community interests later we
>>>> could enhance this feature by programmatically builds the JAAS context.
>>>
>>> Yes, this is what I was suggesting.
>>>
>>> Thanks for considering the idea.
>>>
>>>> 1) A bad host or bad Kerb principal is keep on trying to establish
>>>> connection with the Quorum.
>>>
>>>> 2) Trigger FLE several times.
>>>
>>> Let me get back to you.
>>>
>>> I think it would also not be difficult to create a new unit test that
>>> extends QuorumHammerTest (like ObserverQuorumHammerTest), sets up a
>>> secure
>>> configuration using the minikdc, and then uses QuorumBase methods to start
>>> and stop quorum peers at random while the generated load is hitting the
>>> quorum.
>>>
>>>
>>> On Wed, Nov 2, 2016 at 7:15 AM, Rakesh Radhakrishnan <ra...@apache.org>
>>> wrote:
>>>
>>>> Thanks a lot Andrew Purtell for your time and comments.
>>>>
>>>>>>>>> I like how Hadoop programmatically builds the JAAS context from its
>>>>>>>>> configuration such that the JAAS configuration file and definition
>>> of
>>>>>>>>> system property pointing to same are not needed. Would be nice if
>>> ZK
>>>> could
>>>>>>>>> optionally do the same, that would be one fewer config file to get
>>>>>>>>> precisely right.
>>>>
>>>> Its really an interesting thought. My view on this-> Since ZK server
>>>> already has "jaas.config" in place for client-server auth, IMHO to
>>> continue
>>>> with this jaas config and allows to configure the 'QuorumServer' &
>>>> 'QuorumLearner' quorum auth sections. How about pushing the current
>>>> approach as a first step and based on the community interests later we
>>>> could enhance this feature by programmatically builds the JAAS context.
>>>> Does this makes sense to you?
>>>>
>>>>
>>>>>>>> Any suggestions on some kind of hammer test to apply now ?
>>>> 1) A bad host or bad Kerb principal is keep on trying to establish
>>>> connection with the Quorum. Mostly this will increase the load on
>>> Leader ZK
>>>> server and observe how the system behaves.
>>>> 2) Trigger FLE several times. This can be done by finding out newly
>>> elected
>>>> Leader and kill it. Probably can do this many times.
>>>>
>>>>
>>>> Dear Committers,
>>>>
>>>> Could you please help me pushing ZOOKEEPER-2479 this in. This would
>>> help to
>>>> evaluate the time taken for FLE(enable or disable auth) and used to
>>> compare
>>>> time taken in several runs programatically in scripts or so. Thanks!
>>>>
>>>> Thanks,
>>>> Rakesh
>>>>
>>>> On Wed, Nov 2, 2016 at 6:59 AM, Andrew Purtell <ap...@apache.org>
>>>> wrote:
>>>>
>>>>> I would like to relay a working example of a patched 3.4.9 ZK cluster
>>>>> running on one host (without containers). I can confirm mutual SASL
>>> auth
>>>>> among the quorum is required and enforced because when instances were
>>>>> slightly misconfigured with incorrect principal strings they couldn't
>>>>> participate in the quorum.
>>>>>
>>>>> 1. Assign extra loopback addresses. Example below is what you need to
>>> do
>>>>> for FreeBSD:
>>>>>
>>>>> $ sudo ifconfig lo0 127.0.1.1 netmask 255.255.255.0 alias
>>>>> $ sudo ifconfig lo0 127.0.2.1 netmask 255.255.255.0 alias
>>>>> $ sudo ifconfig lo0 127.0.3.1 netmask 255.255.255.0 alias
>>>>>
>>>>> 2. Create Kerberos principals. Example below is for Heimdal, MIT will
>>> be
>>>>> similar:
>>>>>
>>>>> $ sudo kadmin -l
>>>>>> add --random-key zookeeper
>>>>>> add --random-key zookeeper/127.0.1.1
>>>>>> add --random-key zookeeper/127.0.2.1
>>>>>> add --random-key zookeeper/127.0.3.1
>>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper
>>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
>>> 127.0.1.1
>>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
>>> 127.0.2.1
>>>>>> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
>>> 127.0.3.1
>>>>>> exit
>>>>>
>>>>> 3. Patch ZK with 1045, build a tarball, then unpack it into three
>>> install
>>>>> directories. Mine are /var/tmp/test/{1,2,3}.
>>>>>
>>>>> 4. Initialize 'myid' files:
>>>>>
>>>>> $ mkdir /var/tmp/test/1/data
>>>>> $ echo 1 > /var/tmp/test/1/data/myid
>>>>> $ mkdir /var/tmp/test/2/data
>>>>> $ echo 2 > /var/tmp/test/2/data/myid
>>>>> $ mkdir /var/tmp/test/3/data
>>>>> $ echo 3 > /var/tmp/test/3/data/myid
>>>>>
>>>>> 5. Create configuration files for each instance. Below are examples
>>> for
>>>>> instance 1. You will need to make substitutions for your Kerberos
>>> realm
>>>>> (mine is LOCAL), and for instances 2 and 3 change the path to the JAAS
>>>>> configuration file, path to data directory, and bind address for
>>>>> clientPortAddress as appropriate.
>>>>>
>>>>> conf/java.env:
>>>>>
>>>>> export
>>>>> JVMFLAGS="-Djava.security.auth.login.config=/var/tmp/
>>>> test/1/conf/jaas.conf
>>>>> -Djavax.security.auth.useSubjectCredsOnly=false"
>>>>>
>>>>> conf/jaas.conf:
>>>>>
>>>>> QuorumServer {
>>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>>> useKeyTab=true
>>>>> keyTab="/var/tmp/test/zookeeper.keytab"
>>>>> storeKey=true
>>>>> useTicketCache=false
>>>>> debug=false
>>>>> principal="zookeeper/127.0.1.1@LOCAL";
>>>>> };
>>>>>
>>>>> QuorumLearner {
>>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>>> useKeyTab=true
>>>>> keyTab="/var/tmp/test/zookeeper.keytab"
>>>>> storeKey=true
>>>>> useTicketCache=false
>>>>> debug=false
>>>>> principal="zookeeper/127.0.1.1@LOCAL";
>>>>> };
>>>>>
>>>>> conf/zoo.cfg:
>>>>>
>>>>> tickTime=2000
>>>>> initLimit=10
>>>>> syncLimit=5
>>>>> dataDir=/var/tmp/test/zk/1/data
>>>>> clientPort=2181
>>>>> clientPortAddress=127.0.1.1
>>>>>
>>>>> quorum.auth.enableSasl=true
>>>>> quorum.auth.learnerRequireSasl=true
>>>>> quorum.auth.serverRequireSasl=true
>>>>> quorum.auth.learner.loginContext=QuorumLearner
>>>>> quorum.auth.server.loginContext=QuorumServer
>>>>> quorum.auth.kerberos.servicePrincipal=zookeeper/_HOST
>>>>>
>>>>> server.1=127.0.1.1:2888:3888
>>>>> server.2=127.0.2.1:2888:3888
>>>>> server.3=127.0.3.1:2888:3888
>>>>>
>>>>> 5. Launch the three instances in the foreground in separate terminals.
>>>>> Example for instance 1:
>>>>>
>>>>> $ cd /var/tmp/test/1
>>>>> $ bash ./bin/zkServer.sh start-foreground
>>>>>
>>>>> Having done all this, I can see successful authentication and
>>> bootstrap
>>>> of
>>>>> a quorum.
>>>>>
>>>>> This example shares a keytab. No need to do that if you'd like to be
>>>>> pedantic. Clone the keytab file into the separate conf directories and
>>>>> update the jaas.conf files.
>>>>>
>>>>> I like how Hadoop programmatically builds the JAAS context from its
>>>>> configuration such that the JAAS configuration file and definition of
>>>>> system property pointing to same are not needed. Would be nice if ZK
>>>> could
>>>>> optionally do the same, that would be one fewer config file to get
>>>>> precisely right.
>>>>>
>>>>> Any suggestions on some kind of hammer test to apply now ?
>>>>>
>>>>>
>>>>> On Fri, Oct 21, 2016 at 7:34 AM, Flavio Junqueira <fp...@apache.org>
>>>> wrote:
>>>>>
>>>>>> Michael Han posted an update to the test plan to the jira and I
>>> want to
>>>>>> call the attention of the community to it. It is a big change that
>>> we
>>>>> need
>>>>>> to be extra careful about because it is supposed to go to the 3.4
>>>> branch.
>>>>>> It'd be great to have more folks in the community involved with the
>>>>>> testing. If you have cycles and interest, please help with testing.
>>>>>>
>>>>>> -Flavio
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> - Andy
>>>>>
>>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>> Hein
>>>>> (via Tom White)
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>> (via Tom White)
>>>
>>
>>
Re: Test plan for ZK-1045 - Call for volunteers
Posted by Patrick Hunt <ph...@apache.org>.
I did a bunch of manual testing last week. I tried running multiple cluster
sizes, tried running new server against old server, also went through the
rolling upgrade testing part of the document and that worked fine. afaict
at this point we're ready to commit. If there are any concerns please speak
up now as I intend to commit this soon.
Patrick
On Fri, Nov 18, 2016 at 12:15 PM, Patrick Hunt <ph...@apache.org> wrote:
> As Flavio said originally on this thread this is a big change. Based on
> the current status of the patch and the testing feedback it seems like
> we've done significant work to ensure the quality of the change. Do folks
> feel that there has been sufficient review/testing that we can commit this
> and cut a 3.5.10 release? If not what specifically is left?
>
> Patrick
>
> On Wed, Nov 2, 2016 at 8:51 AM, Andrew Purtell <ap...@apache.org>
> wrote:
>
>> > My view on this-> Since ZK server
>> > already has "jaas.config" in place for client-server auth, IMHO to
>> continue
>>
>> Right, also build the client-server authentication context
>> programatically.
>>
>> > How about pushing the current
>> > approach as a first step and based on the community interests later we
>> > could enhance this feature by programmatically builds the JAAS context.
>>
>> Yes, this is what I was suggesting.
>>
>> Thanks for considering the idea.
>>
>> > 1) A bad host or bad Kerb principal is keep on trying to establish
>> > connection with the Quorum.
>>
>> > 2) Trigger FLE several times.
>>
>> Let me get back to you.
>>
>> I think it would also not be difficult to create a new unit test that
>> extends QuorumHammerTest (like ObserverQuorumHammerTest), sets up a
>> secure
>> configuration using the minikdc, and then uses QuorumBase methods to start
>> and stop quorum peers at random while the generated load is hitting the
>> quorum.
>>
>>
>> On Wed, Nov 2, 2016 at 7:15 AM, Rakesh Radhakrishnan <ra...@apache.org>
>> wrote:
>>
>> > Thanks a lot Andrew Purtell for your time and comments.
>> >
>> > >>>>>I like how Hadoop programmatically builds the JAAS context from its
>> > >>>>>configuration such that the JAAS configuration file and definition
>> of
>> > >>>>>system property pointing to same are not needed. Would be nice if
>> ZK
>> > could
>> > >>>>>optionally do the same, that would be one fewer config file to get
>> > >>>>>precisely right.
>> >
>> > Its really an interesting thought. My view on this-> Since ZK server
>> > already has "jaas.config" in place for client-server auth, IMHO to
>> continue
>> > with this jaas config and allows to configure the 'QuorumServer' &
>> > 'QuorumLearner' quorum auth sections. How about pushing the current
>> > approach as a first step and based on the community interests later we
>> > could enhance this feature by programmatically builds the JAAS context.
>> > Does this makes sense to you?
>> >
>> >
>> > >>>>Any suggestions on some kind of hammer test to apply now ?
>> > 1) A bad host or bad Kerb principal is keep on trying to establish
>> > connection with the Quorum. Mostly this will increase the load on
>> Leader ZK
>> > server and observe how the system behaves.
>> > 2) Trigger FLE several times. This can be done by finding out newly
>> elected
>> > Leader and kill it. Probably can do this many times.
>> >
>> >
>> > Dear Committers,
>> >
>> > Could you please help me pushing ZOOKEEPER-2479 this in. This would
>> help to
>> > evaluate the time taken for FLE(enable or disable auth) and used to
>> compare
>> > time taken in several runs programatically in scripts or so. Thanks!
>> >
>> > Thanks,
>> > Rakesh
>> >
>> > On Wed, Nov 2, 2016 at 6:59 AM, Andrew Purtell <ap...@apache.org>
>> > wrote:
>> >
>> > > I would like to relay a working example of a patched 3.4.9 ZK cluster
>> > > running on one host (without containers). I can confirm mutual SASL
>> auth
>> > > among the quorum is required and enforced because when instances were
>> > > slightly misconfigured with incorrect principal strings they couldn't
>> > > participate in the quorum.
>> > >
>> > > 1. Assign extra loopback addresses. Example below is what you need to
>> do
>> > > for FreeBSD:
>> > >
>> > > $ sudo ifconfig lo0 127.0.1.1 netmask 255.255.255.0 alias
>> > > $ sudo ifconfig lo0 127.0.2.1 netmask 255.255.255.0 alias
>> > > $ sudo ifconfig lo0 127.0.3.1 netmask 255.255.255.0 alias
>> > >
>> > > 2. Create Kerberos principals. Example below is for Heimdal, MIT will
>> be
>> > > similar:
>> > >
>> > > $ sudo kadmin -l
>> > > > add --random-key zookeeper
>> > > > add --random-key zookeeper/127.0.1.1
>> > > > add --random-key zookeeper/127.0.2.1
>> > > > add --random-key zookeeper/127.0.3.1
>> > > > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper
>> > > > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
>> 127.0.1.1
>> > > > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
>> 127.0.2.1
>> > > > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
>> 127.0.3.1
>> > > > exit
>> > >
>> > > 3. Patch ZK with 1045, build a tarball, then unpack it into three
>> install
>> > > directories. Mine are /var/tmp/test/{1,2,3}.
>> > >
>> > > 4. Initialize 'myid' files:
>> > >
>> > > $ mkdir /var/tmp/test/1/data
>> > > $ echo 1 > /var/tmp/test/1/data/myid
>> > > $ mkdir /var/tmp/test/2/data
>> > > $ echo 2 > /var/tmp/test/2/data/myid
>> > > $ mkdir /var/tmp/test/3/data
>> > > $ echo 3 > /var/tmp/test/3/data/myid
>> > >
>> > > 5. Create configuration files for each instance. Below are examples
>> for
>> > > instance 1. You will need to make substitutions for your Kerberos
>> realm
>> > > (mine is LOCAL), and for instances 2 and 3 change the path to the JAAS
>> > > configuration file, path to data directory, and bind address for
>> > > clientPortAddress as appropriate.
>> > >
>> > > conf/java.env:
>> > >
>> > > export
>> > > JVMFLAGS="-Djava.security.auth.login.config=/var/tmp/
>> > test/1/conf/jaas.conf
>> > > -Djavax.security.auth.useSubjectCredsOnly=false"
>> > >
>> > > conf/jaas.conf:
>> > >
>> > > QuorumServer {
>> > > com.sun.security.auth.module.Krb5LoginModule required
>> > > useKeyTab=true
>> > > keyTab="/var/tmp/test/zookeeper.keytab"
>> > > storeKey=true
>> > > useTicketCache=false
>> > > debug=false
>> > > principal="zookeeper/127.0.1.1@LOCAL";
>> > > };
>> > >
>> > > QuorumLearner {
>> > > com.sun.security.auth.module.Krb5LoginModule required
>> > > useKeyTab=true
>> > > keyTab="/var/tmp/test/zookeeper.keytab"
>> > > storeKey=true
>> > > useTicketCache=false
>> > > debug=false
>> > > principal="zookeeper/127.0.1.1@LOCAL";
>> > > };
>> > >
>> > > conf/zoo.cfg:
>> > >
>> > > tickTime=2000
>> > > initLimit=10
>> > > syncLimit=5
>> > > dataDir=/var/tmp/test/zk/1/data
>> > > clientPort=2181
>> > > clientPortAddress=127.0.1.1
>> > >
>> > > quorum.auth.enableSasl=true
>> > > quorum.auth.learnerRequireSasl=true
>> > > quorum.auth.serverRequireSasl=true
>> > > quorum.auth.learner.loginContext=QuorumLearner
>> > > quorum.auth.server.loginContext=QuorumServer
>> > > quorum.auth.kerberos.servicePrincipal=zookeeper/_HOST
>> > >
>> > > server.1=127.0.1.1:2888:3888
>> > > server.2=127.0.2.1:2888:3888
>> > > server.3=127.0.3.1:2888:3888
>> > >
>> > > 5. Launch the three instances in the foreground in separate terminals.
>> > > Example for instance 1:
>> > >
>> > > $ cd /var/tmp/test/1
>> > > $ bash ./bin/zkServer.sh start-foreground
>> > >
>> > > Having done all this, I can see successful authentication and
>> bootstrap
>> > of
>> > > a quorum.
>> > >
>> > > This example shares a keytab. No need to do that if you'd like to be
>> > > pedantic. Clone the keytab file into the separate conf directories and
>> > > update the jaas.conf files.
>> > >
>> > > I like how Hadoop programmatically builds the JAAS context from its
>> > > configuration such that the JAAS configuration file and definition of
>> > > system property pointing to same are not needed. Would be nice if ZK
>> > could
>> > > optionally do the same, that would be one fewer config file to get
>> > > precisely right.
>> > >
>> > > Any suggestions on some kind of hammer test to apply now ?
>> > >
>> > >
>> > > On Fri, Oct 21, 2016 at 7:34 AM, Flavio Junqueira <fp...@apache.org>
>> > wrote:
>> > >
>> > > > Michael Han posted an update to the test plan to the jira and I
>> want to
>> > > > call the attention of the community to it. It is a big change that
>> we
>> > > need
>> > > > to be extra careful about because it is supposed to go to the 3.4
>> > branch.
>> > > > It'd be great to have more folks in the community involved with the
>> > > > testing. If you have cycles and interest, please help with testing.
>> > > >
>> > > > -Flavio
>> > >
>> > >
>> > >
>> > >
>> > > --
>> > > Best regards,
>> > >
>> > > - Andy
>> > >
>> > > Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein
>> > > (via Tom White)
>> > >
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
>
>
Re: Test plan for ZK-1045 - Call for volunteers
Posted by Patrick Hunt <ph...@apache.org>.
As Flavio said originally on this thread this is a big change. Based on the
current status of the patch and the testing feedback it seems like we've
done significant work to ensure the quality of the change. Do folks feel
that there has been sufficient review/testing that we can commit this and
cut a 3.5.10 release? If not what specifically is left?
Patrick
On Wed, Nov 2, 2016 at 8:51 AM, Andrew Purtell <ap...@apache.org> wrote:
> > My view on this-> Since ZK server
> > already has "jaas.config" in place for client-server auth, IMHO to
> continue
>
> Right, also build the client-server authentication context programatically.
>
> > How about pushing the current
> > approach as a first step and based on the community interests later we
> > could enhance this feature by programmatically builds the JAAS context.
>
> Yes, this is what I was suggesting.
>
> Thanks for considering the idea.
>
> > 1) A bad host or bad Kerb principal is keep on trying to establish
> > connection with the Quorum.
>
> > 2) Trigger FLE several times.
>
> Let me get back to you.
>
> I think it would also not be difficult to create a new unit test that
> extends QuorumHammerTest (like ObserverQuorumHammerTest), sets up a secure
> configuration using the minikdc, and then uses QuorumBase methods to start
> and stop quorum peers at random while the generated load is hitting the
> quorum.
>
>
> On Wed, Nov 2, 2016 at 7:15 AM, Rakesh Radhakrishnan <ra...@apache.org>
> wrote:
>
> > Thanks a lot Andrew Purtell for your time and comments.
> >
> > >>>>>I like how Hadoop programmatically builds the JAAS context from its
> > >>>>>configuration such that the JAAS configuration file and definition
> of
> > >>>>>system property pointing to same are not needed. Would be nice if ZK
> > could
> > >>>>>optionally do the same, that would be one fewer config file to get
> > >>>>>precisely right.
> >
> > Its really an interesting thought. My view on this-> Since ZK server
> > already has "jaas.config" in place for client-server auth, IMHO to
> continue
> > with this jaas config and allows to configure the 'QuorumServer' &
> > 'QuorumLearner' quorum auth sections. How about pushing the current
> > approach as a first step and based on the community interests later we
> > could enhance this feature by programmatically builds the JAAS context.
> > Does this makes sense to you?
> >
> >
> > >>>>Any suggestions on some kind of hammer test to apply now ?
> > 1) A bad host or bad Kerb principal is keep on trying to establish
> > connection with the Quorum. Mostly this will increase the load on Leader
> ZK
> > server and observe how the system behaves.
> > 2) Trigger FLE several times. This can be done by finding out newly
> elected
> > Leader and kill it. Probably can do this many times.
> >
> >
> > Dear Committers,
> >
> > Could you please help me pushing ZOOKEEPER-2479 this in. This would help
> to
> > evaluate the time taken for FLE(enable or disable auth) and used to
> compare
> > time taken in several runs programatically in scripts or so. Thanks!
> >
> > Thanks,
> > Rakesh
> >
> > On Wed, Nov 2, 2016 at 6:59 AM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > I would like to relay a working example of a patched 3.4.9 ZK cluster
> > > running on one host (without containers). I can confirm mutual SASL
> auth
> > > among the quorum is required and enforced because when instances were
> > > slightly misconfigured with incorrect principal strings they couldn't
> > > participate in the quorum.
> > >
> > > 1. Assign extra loopback addresses. Example below is what you need to
> do
> > > for FreeBSD:
> > >
> > > $ sudo ifconfig lo0 127.0.1.1 netmask 255.255.255.0 alias
> > > $ sudo ifconfig lo0 127.0.2.1 netmask 255.255.255.0 alias
> > > $ sudo ifconfig lo0 127.0.3.1 netmask 255.255.255.0 alias
> > >
> > > 2. Create Kerberos principals. Example below is for Heimdal, MIT will
> be
> > > similar:
> > >
> > > $ sudo kadmin -l
> > > > add --random-key zookeeper
> > > > add --random-key zookeeper/127.0.1.1
> > > > add --random-key zookeeper/127.0.2.1
> > > > add --random-key zookeeper/127.0.3.1
> > > > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper
> > > > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
> 127.0.1.1
> > > > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
> 127.0.2.1
> > > > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/
> 127.0.3.1
> > > > exit
> > >
> > > 3. Patch ZK with 1045, build a tarball, then unpack it into three
> install
> > > directories. Mine are /var/tmp/test/{1,2,3}.
> > >
> > > 4. Initialize 'myid' files:
> > >
> > > $ mkdir /var/tmp/test/1/data
> > > $ echo 1 > /var/tmp/test/1/data/myid
> > > $ mkdir /var/tmp/test/2/data
> > > $ echo 2 > /var/tmp/test/2/data/myid
> > > $ mkdir /var/tmp/test/3/data
> > > $ echo 3 > /var/tmp/test/3/data/myid
> > >
> > > 5. Create configuration files for each instance. Below are examples for
> > > instance 1. You will need to make substitutions for your Kerberos realm
> > > (mine is LOCAL), and for instances 2 and 3 change the path to the JAAS
> > > configuration file, path to data directory, and bind address for
> > > clientPortAddress as appropriate.
> > >
> > > conf/java.env:
> > >
> > > export
> > > JVMFLAGS="-Djava.security.auth.login.config=/var/tmp/
> > test/1/conf/jaas.conf
> > > -Djavax.security.auth.useSubjectCredsOnly=false"
> > >
> > > conf/jaas.conf:
> > >
> > > QuorumServer {
> > > com.sun.security.auth.module.Krb5LoginModule required
> > > useKeyTab=true
> > > keyTab="/var/tmp/test/zookeeper.keytab"
> > > storeKey=true
> > > useTicketCache=false
> > > debug=false
> > > principal="zookeeper/127.0.1.1@LOCAL";
> > > };
> > >
> > > QuorumLearner {
> > > com.sun.security.auth.module.Krb5LoginModule required
> > > useKeyTab=true
> > > keyTab="/var/tmp/test/zookeeper.keytab"
> > > storeKey=true
> > > useTicketCache=false
> > > debug=false
> > > principal="zookeeper/127.0.1.1@LOCAL";
> > > };
> > >
> > > conf/zoo.cfg:
> > >
> > > tickTime=2000
> > > initLimit=10
> > > syncLimit=5
> > > dataDir=/var/tmp/test/zk/1/data
> > > clientPort=2181
> > > clientPortAddress=127.0.1.1
> > >
> > > quorum.auth.enableSasl=true
> > > quorum.auth.learnerRequireSasl=true
> > > quorum.auth.serverRequireSasl=true
> > > quorum.auth.learner.loginContext=QuorumLearner
> > > quorum.auth.server.loginContext=QuorumServer
> > > quorum.auth.kerberos.servicePrincipal=zookeeper/_HOST
> > >
> > > server.1=127.0.1.1:2888:3888
> > > server.2=127.0.2.1:2888:3888
> > > server.3=127.0.3.1:2888:3888
> > >
> > > 5. Launch the three instances in the foreground in separate terminals.
> > > Example for instance 1:
> > >
> > > $ cd /var/tmp/test/1
> > > $ bash ./bin/zkServer.sh start-foreground
> > >
> > > Having done all this, I can see successful authentication and bootstrap
> > of
> > > a quorum.
> > >
> > > This example shares a keytab. No need to do that if you'd like to be
> > > pedantic. Clone the keytab file into the separate conf directories and
> > > update the jaas.conf files.
> > >
> > > I like how Hadoop programmatically builds the JAAS context from its
> > > configuration such that the JAAS configuration file and definition of
> > > system property pointing to same are not needed. Would be nice if ZK
> > could
> > > optionally do the same, that would be one fewer config file to get
> > > precisely right.
> > >
> > > Any suggestions on some kind of hammer test to apply now ?
> > >
> > >
> > > On Fri, Oct 21, 2016 at 7:34 AM, Flavio Junqueira <fp...@apache.org>
> > wrote:
> > >
> > > > Michael Han posted an update to the test plan to the jira and I want
> to
> > > > call the attention of the community to it. It is a big change that we
> > > need
> > > > to be extra careful about because it is supposed to go to the 3.4
> > branch.
> > > > It'd be great to have more folks in the community involved with the
> > > > testing. If you have cycles and interest, please help with testing.
> > > >
> > > > -Flavio
> > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: Test plan for ZK-1045 - Call for volunteers
Posted by Andrew Purtell <ap...@apache.org>.
> My view on this-> Since ZK server
> already has "jaas.config" in place for client-server auth, IMHO to
continue
Right, also build the client-server authentication context programatically.
> How about pushing the current
> approach as a first step and based on the community interests later we
> could enhance this feature by programmatically builds the JAAS context.
Yes, this is what I was suggesting.
Thanks for considering the idea.
> 1) A bad host or bad Kerb principal is keep on trying to establish
> connection with the Quorum.
> 2) Trigger FLE several times.
Let me get back to you.
I think it would also not be difficult to create a new unit test that
extends QuorumHammerTest (like ObserverQuorumHammerTest), sets up a secure
configuration using the minikdc, and then uses QuorumBase methods to start
and stop quorum peers at random while the generated load is hitting the
quorum.
On Wed, Nov 2, 2016 at 7:15 AM, Rakesh Radhakrishnan <ra...@apache.org>
wrote:
> Thanks a lot Andrew Purtell for your time and comments.
>
> >>>>>I like how Hadoop programmatically builds the JAAS context from its
> >>>>>configuration such that the JAAS configuration file and definition of
> >>>>>system property pointing to same are not needed. Would be nice if ZK
> could
> >>>>>optionally do the same, that would be one fewer config file to get
> >>>>>precisely right.
>
> Its really an interesting thought. My view on this-> Since ZK server
> already has "jaas.config" in place for client-server auth, IMHO to continue
> with this jaas config and allows to configure the 'QuorumServer' &
> 'QuorumLearner' quorum auth sections. How about pushing the current
> approach as a first step and based on the community interests later we
> could enhance this feature by programmatically builds the JAAS context.
> Does this makes sense to you?
>
>
> >>>>Any suggestions on some kind of hammer test to apply now ?
> 1) A bad host or bad Kerb principal is keep on trying to establish
> connection with the Quorum. Mostly this will increase the load on Leader ZK
> server and observe how the system behaves.
> 2) Trigger FLE several times. This can be done by finding out newly elected
> Leader and kill it. Probably can do this many times.
>
>
> Dear Committers,
>
> Could you please help me pushing ZOOKEEPER-2479 this in. This would help to
> evaluate the time taken for FLE(enable or disable auth) and used to compare
> time taken in several runs programatically in scripts or so. Thanks!
>
> Thanks,
> Rakesh
>
> On Wed, Nov 2, 2016 at 6:59 AM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > I would like to relay a working example of a patched 3.4.9 ZK cluster
> > running on one host (without containers). I can confirm mutual SASL auth
> > among the quorum is required and enforced because when instances were
> > slightly misconfigured with incorrect principal strings they couldn't
> > participate in the quorum.
> >
> > 1. Assign extra loopback addresses. Example below is what you need to do
> > for FreeBSD:
> >
> > $ sudo ifconfig lo0 127.0.1.1 netmask 255.255.255.0 alias
> > $ sudo ifconfig lo0 127.0.2.1 netmask 255.255.255.0 alias
> > $ sudo ifconfig lo0 127.0.3.1 netmask 255.255.255.0 alias
> >
> > 2. Create Kerberos principals. Example below is for Heimdal, MIT will be
> > similar:
> >
> > $ sudo kadmin -l
> > > add --random-key zookeeper
> > > add --random-key zookeeper/127.0.1.1
> > > add --random-key zookeeper/127.0.2.1
> > > add --random-key zookeeper/127.0.3.1
> > > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper
> > > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/127.0.1.1
> > > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/127.0.2.1
> > > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/127.0.3.1
> > > exit
> >
> > 3. Patch ZK with 1045, build a tarball, then unpack it into three install
> > directories. Mine are /var/tmp/test/{1,2,3}.
> >
> > 4. Initialize 'myid' files:
> >
> > $ mkdir /var/tmp/test/1/data
> > $ echo 1 > /var/tmp/test/1/data/myid
> > $ mkdir /var/tmp/test/2/data
> > $ echo 2 > /var/tmp/test/2/data/myid
> > $ mkdir /var/tmp/test/3/data
> > $ echo 3 > /var/tmp/test/3/data/myid
> >
> > 5. Create configuration files for each instance. Below are examples for
> > instance 1. You will need to make substitutions for your Kerberos realm
> > (mine is LOCAL), and for instances 2 and 3 change the path to the JAAS
> > configuration file, path to data directory, and bind address for
> > clientPortAddress as appropriate.
> >
> > conf/java.env:
> >
> > export
> > JVMFLAGS="-Djava.security.auth.login.config=/var/tmp/
> test/1/conf/jaas.conf
> > -Djavax.security.auth.useSubjectCredsOnly=false"
> >
> > conf/jaas.conf:
> >
> > QuorumServer {
> > com.sun.security.auth.module.Krb5LoginModule required
> > useKeyTab=true
> > keyTab="/var/tmp/test/zookeeper.keytab"
> > storeKey=true
> > useTicketCache=false
> > debug=false
> > principal="zookeeper/127.0.1.1@LOCAL";
> > };
> >
> > QuorumLearner {
> > com.sun.security.auth.module.Krb5LoginModule required
> > useKeyTab=true
> > keyTab="/var/tmp/test/zookeeper.keytab"
> > storeKey=true
> > useTicketCache=false
> > debug=false
> > principal="zookeeper/127.0.1.1@LOCAL";
> > };
> >
> > conf/zoo.cfg:
> >
> > tickTime=2000
> > initLimit=10
> > syncLimit=5
> > dataDir=/var/tmp/test/zk/1/data
> > clientPort=2181
> > clientPortAddress=127.0.1.1
> >
> > quorum.auth.enableSasl=true
> > quorum.auth.learnerRequireSasl=true
> > quorum.auth.serverRequireSasl=true
> > quorum.auth.learner.loginContext=QuorumLearner
> > quorum.auth.server.loginContext=QuorumServer
> > quorum.auth.kerberos.servicePrincipal=zookeeper/_HOST
> >
> > server.1=127.0.1.1:2888:3888
> > server.2=127.0.2.1:2888:3888
> > server.3=127.0.3.1:2888:3888
> >
> > 5. Launch the three instances in the foreground in separate terminals.
> > Example for instance 1:
> >
> > $ cd /var/tmp/test/1
> > $ bash ./bin/zkServer.sh start-foreground
> >
> > Having done all this, I can see successful authentication and bootstrap
> of
> > a quorum.
> >
> > This example shares a keytab. No need to do that if you'd like to be
> > pedantic. Clone the keytab file into the separate conf directories and
> > update the jaas.conf files.
> >
> > I like how Hadoop programmatically builds the JAAS context from its
> > configuration such that the JAAS configuration file and definition of
> > system property pointing to same are not needed. Would be nice if ZK
> could
> > optionally do the same, that would be one fewer config file to get
> > precisely right.
> >
> > Any suggestions on some kind of hammer test to apply now ?
> >
> >
> > On Fri, Oct 21, 2016 at 7:34 AM, Flavio Junqueira <fp...@apache.org>
> wrote:
> >
> > > Michael Han posted an update to the test plan to the jira and I want to
> > > call the attention of the community to it. It is a big change that we
> > need
> > > to be extra careful about because it is supposed to go to the 3.4
> branch.
> > > It'd be great to have more folks in the community involved with the
> > > testing. If you have cycles and interest, please help with testing.
> > >
> > > -Flavio
> >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Test plan for ZK-1045 - Call for volunteers
Posted by Rakesh Radhakrishnan <ra...@apache.org>.
Thanks a lot Andrew Purtell for your time and comments.
>>>>>I like how Hadoop programmatically builds the JAAS context from its
>>>>>configuration such that the JAAS configuration file and definition of
>>>>>system property pointing to same are not needed. Would be nice if ZK
could
>>>>>optionally do the same, that would be one fewer config file to get
>>>>>precisely right.
Its really an interesting thought. My view on this-> Since ZK server
already has "jaas.config" in place for client-server auth, IMHO to continue
with this jaas config and allows to configure the 'QuorumServer' &
'QuorumLearner' quorum auth sections. How about pushing the current
approach as a first step and based on the community interests later we
could enhance this feature by programmatically builds the JAAS context.
Does this makes sense to you?
>>>>Any suggestions on some kind of hammer test to apply now ?
1) A bad host or bad Kerb principal is keep on trying to establish
connection with the Quorum. Mostly this will increase the load on Leader ZK
server and observe how the system behaves.
2) Trigger FLE several times. This can be done by finding out newly elected
Leader and kill it. Probably can do this many times.
Dear Committers,
Could you please help me pushing ZOOKEEPER-2479 this in. This would help to
evaluate the time taken for FLE(enable or disable auth) and used to compare
time taken in several runs programatically in scripts or so. Thanks!
Thanks,
Rakesh
On Wed, Nov 2, 2016 at 6:59 AM, Andrew Purtell <ap...@apache.org> wrote:
> I would like to relay a working example of a patched 3.4.9 ZK cluster
> running on one host (without containers). I can confirm mutual SASL auth
> among the quorum is required and enforced because when instances were
> slightly misconfigured with incorrect principal strings they couldn't
> participate in the quorum.
>
> 1. Assign extra loopback addresses. Example below is what you need to do
> for FreeBSD:
>
> $ sudo ifconfig lo0 127.0.1.1 netmask 255.255.255.0 alias
> $ sudo ifconfig lo0 127.0.2.1 netmask 255.255.255.0 alias
> $ sudo ifconfig lo0 127.0.3.1 netmask 255.255.255.0 alias
>
> 2. Create Kerberos principals. Example below is for Heimdal, MIT will be
> similar:
>
> $ sudo kadmin -l
> > add --random-key zookeeper
> > add --random-key zookeeper/127.0.1.1
> > add --random-key zookeeper/127.0.2.1
> > add --random-key zookeeper/127.0.3.1
> > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper
> > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/127.0.1.1
> > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/127.0.2.1
> > ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/127.0.3.1
> > exit
>
> 3. Patch ZK with 1045, build a tarball, then unpack it into three install
> directories. Mine are /var/tmp/test/{1,2,3}.
>
> 4. Initialize 'myid' files:
>
> $ mkdir /var/tmp/test/1/data
> $ echo 1 > /var/tmp/test/1/data/myid
> $ mkdir /var/tmp/test/2/data
> $ echo 2 > /var/tmp/test/2/data/myid
> $ mkdir /var/tmp/test/3/data
> $ echo 3 > /var/tmp/test/3/data/myid
>
> 5. Create configuration files for each instance. Below are examples for
> instance 1. You will need to make substitutions for your Kerberos realm
> (mine is LOCAL), and for instances 2 and 3 change the path to the JAAS
> configuration file, path to data directory, and bind address for
> clientPortAddress as appropriate.
>
> conf/java.env:
>
> export
> JVMFLAGS="-Djava.security.auth.login.config=/var/tmp/test/1/conf/jaas.conf
> -Djavax.security.auth.useSubjectCredsOnly=false"
>
> conf/jaas.conf:
>
> QuorumServer {
> com.sun.security.auth.module.Krb5LoginModule required
> useKeyTab=true
> keyTab="/var/tmp/test/zookeeper.keytab"
> storeKey=true
> useTicketCache=false
> debug=false
> principal="zookeeper/127.0.1.1@LOCAL";
> };
>
> QuorumLearner {
> com.sun.security.auth.module.Krb5LoginModule required
> useKeyTab=true
> keyTab="/var/tmp/test/zookeeper.keytab"
> storeKey=true
> useTicketCache=false
> debug=false
> principal="zookeeper/127.0.1.1@LOCAL";
> };
>
> conf/zoo.cfg:
>
> tickTime=2000
> initLimit=10
> syncLimit=5
> dataDir=/var/tmp/test/zk/1/data
> clientPort=2181
> clientPortAddress=127.0.1.1
>
> quorum.auth.enableSasl=true
> quorum.auth.learnerRequireSasl=true
> quorum.auth.serverRequireSasl=true
> quorum.auth.learner.loginContext=QuorumLearner
> quorum.auth.server.loginContext=QuorumServer
> quorum.auth.kerberos.servicePrincipal=zookeeper/_HOST
>
> server.1=127.0.1.1:2888:3888
> server.2=127.0.2.1:2888:3888
> server.3=127.0.3.1:2888:3888
>
> 5. Launch the three instances in the foreground in separate terminals.
> Example for instance 1:
>
> $ cd /var/tmp/test/1
> $ bash ./bin/zkServer.sh start-foreground
>
> Having done all this, I can see successful authentication and bootstrap of
> a quorum.
>
> This example shares a keytab. No need to do that if you'd like to be
> pedantic. Clone the keytab file into the separate conf directories and
> update the jaas.conf files.
>
> I like how Hadoop programmatically builds the JAAS context from its
> configuration such that the JAAS configuration file and definition of
> system property pointing to same are not needed. Would be nice if ZK could
> optionally do the same, that would be one fewer config file to get
> precisely right.
>
> Any suggestions on some kind of hammer test to apply now ?
>
>
> On Fri, Oct 21, 2016 at 7:34 AM, Flavio Junqueira <fp...@apache.org> wrote:
>
> > Michael Han posted an update to the test plan to the jira and I want to
> > call the attention of the community to it. It is a big change that we
> need
> > to be extra careful about because it is supposed to go to the 3.4 branch.
> > It'd be great to have more folks in the community involved with the
> > testing. If you have cycles and interest, please help with testing.
> >
> > -Flavio
>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: Test plan for ZK-1045 - Call for volunteers
Posted by Andrew Purtell <ap...@apache.org>.
I would like to relay a working example of a patched 3.4.9 ZK cluster
running on one host (without containers). I can confirm mutual SASL auth
among the quorum is required and enforced because when instances were
slightly misconfigured with incorrect principal strings they couldn't
participate in the quorum.
1. Assign extra loopback addresses. Example below is what you need to do
for FreeBSD:
$ sudo ifconfig lo0 127.0.1.1 netmask 255.255.255.0 alias
$ sudo ifconfig lo0 127.0.2.1 netmask 255.255.255.0 alias
$ sudo ifconfig lo0 127.0.3.1 netmask 255.255.255.0 alias
2. Create Kerberos principals. Example below is for Heimdal, MIT will be
similar:
$ sudo kadmin -l
> add --random-key zookeeper
> add --random-key zookeeper/127.0.1.1
> add --random-key zookeeper/127.0.2.1
> add --random-key zookeeper/127.0.3.1
> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper
> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/127.0.1.1
> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/127.0.2.1
> ext_keytab --keytab /var/tmp/test/zookeeper.keytab zookeeper/127.0.3.1
> exit
3. Patch ZK with 1045, build a tarball, then unpack it into three install
directories. Mine are /var/tmp/test/{1,2,3}.
4. Initialize 'myid' files:
$ mkdir /var/tmp/test/1/data
$ echo 1 > /var/tmp/test/1/data/myid
$ mkdir /var/tmp/test/2/data
$ echo 2 > /var/tmp/test/2/data/myid
$ mkdir /var/tmp/test/3/data
$ echo 3 > /var/tmp/test/3/data/myid
5. Create configuration files for each instance. Below are examples for
instance 1. You will need to make substitutions for your Kerberos realm
(mine is LOCAL), and for instances 2 and 3 change the path to the JAAS
configuration file, path to data directory, and bind address for
clientPortAddress as appropriate.
conf/java.env:
export
JVMFLAGS="-Djava.security.auth.login.config=/var/tmp/test/1/conf/jaas.conf
-Djavax.security.auth.useSubjectCredsOnly=false"
conf/jaas.conf:
QuorumServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/var/tmp/test/zookeeper.keytab"
storeKey=true
useTicketCache=false
debug=false
principal="zookeeper/127.0.1.1@LOCAL";
};
QuorumLearner {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/var/tmp/test/zookeeper.keytab"
storeKey=true
useTicketCache=false
debug=false
principal="zookeeper/127.0.1.1@LOCAL";
};
conf/zoo.cfg:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/tmp/test/zk/1/data
clientPort=2181
clientPortAddress=127.0.1.1
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=true
quorum.auth.serverRequireSasl=true
quorum.auth.learner.loginContext=QuorumLearner
quorum.auth.server.loginContext=QuorumServer
quorum.auth.kerberos.servicePrincipal=zookeeper/_HOST
server.1=127.0.1.1:2888:3888
server.2=127.0.2.1:2888:3888
server.3=127.0.3.1:2888:3888
5. Launch the three instances in the foreground in separate terminals.
Example for instance 1:
$ cd /var/tmp/test/1
$ bash ./bin/zkServer.sh start-foreground
Having done all this, I can see successful authentication and bootstrap of
a quorum.
This example shares a keytab. No need to do that if you'd like to be
pedantic. Clone the keytab file into the separate conf directories and
update the jaas.conf files.
I like how Hadoop programmatically builds the JAAS context from its
configuration such that the JAAS configuration file and definition of
system property pointing to same are not needed. Would be nice if ZK could
optionally do the same, that would be one fewer config file to get
precisely right.
Any suggestions on some kind of hammer test to apply now ?
On Fri, Oct 21, 2016 at 7:34 AM, Flavio Junqueira <fp...@apache.org> wrote:
> Michael Han posted an update to the test plan to the jira and I want to
> call the attention of the community to it. It is a big change that we need
> to be extra careful about because it is supposed to go to the 3.4 branch.
> It'd be great to have more folks in the community involved with the
> testing. If you have cycles and interest, please help with testing.
>
> -Flavio
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)