You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Todd Greenwood <to...@audiencescience.com> on 2009/08/01 04:15:15 UTC
Unending Leader Elections in WAN deploy
This repro's in both branch-3.2, and branch-3.2+patches(473, 479, 481).
Basically, it seems like the nodes are electing pd4-zook02 to be the
leader. However, pd4-zook02 seems to realize it's not supposed to be and
then disconnects everyone. Then they re-elect it again, and it loops
over and over.
-------------
Server config
-------------
server.1=dc1-zook01.dc01.revsci.net:2888:3888
server.2=dc1-zook02.dc01.revsci.net:2888:3888
server.3=dc1-zook03.dc01.revsci.net:2888:3888
server.4=dc1-zook04.dc01.revsci.net:2888:3888
server.5=dc1-zook05.dc01.revsci.net:2888:3888
server.6=pd1-zook01.pd01.revsci.net:2888:3888
server.7=pd1-zook02.pd01.revsci.net:2888:3888
server.8=pd4-zook01.iad1.audsci.net:2888:3888
server.9=pd4-zook02.iad1.audsci.net:2888:3888
group.1:1:2:3:4:5
weight.1=1
weight.2=1
weight.3=1
weight.4=1
weight.5=1
group.2:6:7:8:9
weight.6=0
weight.7=0
weight.8=0
weight.9=0
Note that we have 2 groups, composed of machines in 3 different
locations (dc1, pd1, and pd4). The idea is that only machines in dc1
have voting rights, and the ability to become a leader. The machines in
the pods all have a weight of zero, and are not expected to become
leaders, or to vote on transactions.
Let me know what I can do to help resolve this issue.
-Todd
RE: Unending Leader Elections in WAN deploy
Posted by Todd Greenwood <to...@audiencescience.com>.
IT says yes, there are firewalls, but that yes, there is full
connectivity between each of the zk servers.
> -----Original Message-----
> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> Sent: Tuesday, August 04, 2009 6:01 PM
> To: zookeeper-dev@hadoop.apache.org
> Subject: Re: Unending Leader Elections in WAN deploy
>
> Hi todd,
> I see a lot of
>
> java.net.ConnectException: Connection refused
> at sun.nio.ch.Net.connect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at
java.nio.channels.SocketChannel.open(SocketChannel.java:146)
> at
>
org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnx
Ma
> na
> ger.java:324)
> at
>
org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxMana
ge
> r.
> java:304)
> at
>
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSe
nd
> er
> .process(FastLeaderElection.java:317)
> at
>
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSe
nd
> er
> .run(FastLeaderElection.java:290)
> at java.lang.Thread.run(Thread.java:619)
>
>
> Is it possible that there is some firewall? Can all the servers 1-9
> connect
> to all the others using ports that you specified in zoo.cfg i.e
2888/3888?
>
>
> Thanks
> mahadev
>
>
> On 8/4/09 4:56 PM, "Todd Greenwood" <to...@audiencescience.com> wrote:
>
> > Looks like we're not getting *any* leader elected now.... Logs
attached.
> >
> >> -----Original Message-----
> >> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> >> Sent: Tuesday, August 04, 2009 4:07 PM
> >> To: zookeeper-dev@hadoop.apache.org
> >> Subject: RE: Unending Leader Elections in WAN deploy
> >>
> >> Patrick, thanks! I'll forward on to IT and I'll report back to you
> >> shortly...
> >>
> >>> -----Original Message-----
> >>> From: Patrick Hunt [mailto:phunt@apache.org]
> >>> Sent: Tuesday, August 04, 2009 3:55 PM
> >>> To: zookeeper-dev@hadoop.apache.org
> >>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>
> >>> Todd, Mahadev and I looked at this and it turns out to be a
> >> regression.
> >>> Ironically a patch I created for 3.2 branch to add quorum tests
> >> actually
> >>> broke the quorum config -- a default value for a config parameter
> > was
> >>> lost. I'm going to submit a patch asap to get the default back,
but
> >> for
> >>> the time being you can set:
> >>>
> >>> electionAlg=3
> >>>
> >>> in each of your config files.
> >>>
> >>> You should see reference to FastLeaderElection in your log files
if
> >> this
> >>> parameter is set correctly.
> >>>
> >>> Sorry for the trouble,
> >>>
> >>> Patrick
> >>>
> >>> Todd Greenwood wrote:
> >>>> Mahadev,
> >>>>
> >>>> I just heard from IT that this build behaves in exactly the same
> > way
> >> as
> >>>> previous versions, e.g. we get continuous leader elections that
> >>>> disconnect the followers and then get re-elected, and
> >> disconnect...etc.
> >>>>
> >>>> This is from a fresh sync to the 3.2 branch:
> >>>>
> >>>> svn co
> >>>>
> > http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2
> >>>> ./branch-3.2
> >>>>
> >>>> CHANGES.TXT show the various fixes included:
> >>>>
> >>>>
> >>
> >
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> >>>> /src/original$ head -n 50 branch-3.2/CHANGES.txt
> >>>> Release 3.2.1
> >>>>
> >>>> Backward compatibile changes:
> >>>>
> >>>> BUGFIXES:
> >>>> ZOOKEEPER-468. avoid compile warning in send_auth_info().
(chris
> >> via
> >>>> flavio)
> >>>>
> >>>> ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten
(chris
> >> via
> >>>> mahadev)
> >>>>
> >>>> ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via
> > mahadev)
> >>>>
> >>>> ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris
> > via
> >>>> mahadev)
> >>>>
> >>>> ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure)
> >>>> (giri via mahadev)
> >>>>
> >>>> ZOOKEEPER-467. Change log level in BookieHandle (flavio via
> >> mahadev)
> >>>>
> >>>> ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent
> >> immediate
> >>>> failure. (chris via mahadev)
> >>>>
> >>>> ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev
> >> via
> >>>> phunt)
> >>>>
> >>>> ZOOKEEPER-457. Make ZookeeperMain public, support for HBase
(and
> >>>> other)
> >>>> embedded clients (ryan rawson via phunt)
> >>>>
> >>>> ZOOKEEPER-481. Add lastMessageSent to QuorumCnxManager. (flavio
> >> via
> >>>> mahadev)
> >>>>
> >>>> ZOOKEEPER-479. QuorumHierarchical does not count groups
> > correctly
> >>>> (flavio via mahadev)
> >>>>
> >>>> ZOOKEEPER-466. crash on zookeeper_close() when using auth with
> >> empty
> >>>> cert
> >>>> (Chris Darroch via phunt)
> >>>>
> >>>> ZOOKEEPER-480. FLE should perform leader check when node is not
> >>>> leading and
> >>>> add vote of follower (flavio via mahadev)
> >>>>
> >>>> ZOOKEEPER-491. Prevent zero-weight servers from being elected
> >> (flavio
> >>>> via
> >>>> mahadev)
> >>>>
> >>>> What can I do to assist you with this issue?
> >>>>
> >>>> -Todd
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> >>>>> Sent: Tuesday, August 04, 2009 12:43 PM
> >>>>> To: zookeeper-dev@hadoop.apache.org
> >>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>
> >>>>> Hi todd,
> >>>>> comments in line
> >>>>>
> >>>>>
> >>>>> On 8/4/09 12:38 PM, "Todd Greenwood" <to...@audiencescience.com>
> >>>> wrote:
> >>>>>> Mahadev,
> >>>>>>
> >>>>>> Some quick questions:
> >>>>>>
> >>>>>> 1. Version
> >>>>>>
> >>>>>> I see that the CHANGES.txt calls this 3.2.1, but the build.xml
> > is
> >>>> still
> >>>>>> calling this 3.2.0. Should this be rev'd, and am I correct in
> >>>> calling
> >>>>>> this release 3.2.1?
> >>>>> Yes the release is 3.2.1. The build.xml will be fixed as soon as
> > we
> >>>> tag
> >>>>> the
> >>>>> release.
> >>>>>
> >>>>>> 2. Build targets
> >>>>>>
> >>>>>> The package target fails b/c the create-cppunit-configure
target
> >>>> fails
> >>>>>> due to various problems w/ respect to autoconf. Are these
> >>>> dependencies
> >>>>>> documented somewhere ? I'd like to have a fully building
system.
> >>>>>>
> >>>>>> create-cppunit-configure:
> >>>>>> [exec] Can't exec "libtoolize": No such file or directory
> > at
> >>>>>> /usr/bin/autoreconf line 188.
> >>>>>> [exec] Use of uninitialized value $libtoolize in pattern
> >> match
> >>>>>> (m//) at /usr/bin/autoreconf line 188.
> >>>>>> [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT'
> > not
> >>>> found
> >>>>>> in library
> >>>>>> [exec] configure.ac:33: error: possibly undefined macro:
> >>>>>> AM_PATH_CPPUNIT
> >>>>>> [exec] If this token and others are legitimate,
> > please
> >>>> use
> >>>>>> m4_pattern_allow.
> >>>>>> [exec] See the Autoconf documentation.
> >>>>>> [exec] configure.ac:53: error: possibly undefined macro:
> >>>>>> AC_PROG_LIBTOOL
> >>>>>> [exec] autoreconf: /usr/bin/autoconf failed with exit
> > status:
> >> 1
> >>>>>>
> >>>>> You need auto tools to run this. Please read the README for
> >> building c
> >>>>> client library at src/c/ for the installation requirements.
> >>>>>> 3. Sync failure:
> >>>>>>
> >>>>>> This is still failing.
> >>>>>>
> >>>>>> svn: URL
> >>>>>>
> > 'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch'
> >>>>>> doesn't exist
> >>>>>>
> >>>>> Yes this hasn't been fixed yet!
> >>>>>
> >>>>> Thanks
> >>>>> mahadev
> >>>>>> -Todd
> >>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Todd Greenwood
> >>>>>>> Sent: Tuesday, August 04, 2009 11:26 AM
> >>>>>>> To: 'zookeeper-user@hadoop.apache.org'
> >>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
> >>>>>>>
> >>>>>>> Great news. Thank you Mahadev. I'll report our findings later
> >>>> today.
> >>>>>>> -Todd
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> >>>>>>>> Sent: Tuesday, August 04, 2009 11:20 AM
> >>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>
> >>>>>>>> Hi Todd,
> >>>>>>>> I just committed 480 and 491. You can checkout the 3.2
branch
> >>>> now.
> >>>>>>>> Thanks
> >>>>>>>> mahadev
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 8/3/09 4:29 PM, "Todd Greenwood"
> > <to...@audiencescience.com>
> >>>>>> wrote:
> >>>>>>>>> That'd be perfect. Thanks!
> >>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> >>>>>>>>>> Sent: Monday, August 03, 2009 4:24 PM
> >>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>>>
> >>>>>>>>>> Hi Todd,
> >>>>>>>>>> Most of the patches that you mention should be in the
> > branch
> >>>>>> 3.2 by
> >>>>>>>>> tomm
> >>>>>>>>>> or so. 481, 479 are already in. 480 and 491 should be in by
> >>>> tomm.
> >>>>>>>>> Would
> >>>>>>>>>> that
> >>>>>>>>>> suffice for you?
> >>>>>>>>>>
> >>>>>>>>>> Thanks
> >>>>>>>>>> mahadev
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 8/3/09 4:21 PM, "Todd Greenwood"
> >> <to...@audiencescience.com>
> >>>>>>> wrote:
> >>>>>>>>>>> Another problem...I've reverted to the latest versions of
> > the
> >>>>>>>>> patches
> >>>>>>>>>>> that are not specific to branch-3.2, and I'm getting two
> >>>>>> compilation
> >>>>>>>>>>> errors:
> >>>>>>>>>>>
> >>>>>>>>>>> build-generated:
> >>>>>>>>>>> [javac] Compiling 44 source files to
> >>>>>>>>>>>
> >>>>
> >>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>>>>>>> atched/branch-3.2/build/classes
> >>>>>>>>>>>
> >>>>>>>>>>> compile-main:
> >>>>>>>>>>> [javac] Compiling 2 source files to
> >>>>>>>>>>>
> >>>>
> >>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>>>>>>> atched/branch-3.2/build/classes
> >>>>>>>>>>> [javac]
> >>>>>>>>>>>
> >>>>
> >>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>>>>> atched/branch-
> >>>>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> >>>>>>>>>>> mStats.java:30: name clash: getQuorumPeers() and
> >>>>>> getQuorumPeers()
> >>>>>>>>> have
> >>>>>>>>>>> the same erasure
> >>>>>>>>>>> [javac] public String[] getQuorumPeers();
> >>>>>>>>>>> [javac] ^
> >>>>>>>>>>> [javac]
> >>>>>>>>>>>
> >>>>
> >>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>>>>> atched/branch-
> >>>>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> >>>>>>>>>>> mStats.java:31: name clash: getServerState() and
> >>>>>> getServerState()
> >>>>>>>>> have
> >>>>>>>>>>> the same erasure
> >>>>>>>>>>> [javac] public String getServerState();
> >>>>>>>>>>> [javac] ^
> >>>>>>>>>>> [javac] 2 errors
> >>>>>>>>>>>
> >>>>>>>>>>> My build process is pretty simple:
> >>>>>>>>>>>
> >>>>>>>>>>> 1. copy the branch-3.2 source to a temp directory
> >>>>>>>>>>> (src/patched/branch-3.2)
> >>>>>>>>>>> 2. apply the ZOOKEEPER patches in my patches directory
> >>>>>>>>>>> 3. build zookeeper in the temp directory
> >>>>>>>>>>>
> >>>>>>>>>>> -Todd
> >>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> >>>>>>>>>>>> Sent: Monday, August 03, 2009 4:09 PM
> >>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
> >>>>>>>>>>>>
> >>>>>>>>>>>> Flavio,
> >>>>>>>>>>>> I notice that you've updated the patches referenced for
> > the
> >>>> WAN
> >>>>>>>>>>>> deployment. There appears to be an order dependency w/
> >> respect
> >>>>>> to
> >>>>>>>>>>> these
> >>>>>>>>>>>> four patches...
> >>>>>>>>>>>>
> >>>>>>>>>>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
> >>>>>>>>>>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
> >>>>>>>>>>>>
> >>>>>>>>>>>> 473 -> 479 (479 fails)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>
> >>
> >
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> >>>>>>>>>>>> /src/patched/branch-3.2$ patch -p0 <
> >>>>>>>>>>>> ../patches/ZOOKEEPER-479-branch3.2.patch
> >>>>>>>>>>>> patching file
> >>>>>>>>>>>>
> >>>>
> >>
> >
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
> >>>>>>>>>>>> ical.java
> >>>>>>>>>>>> patching file
> >>>>>>>>>>>>
> >>>>
> >>
> >
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
> >>>>>>>>>>>> patching file
> >>>>>>>>>>>>
> >>>>
> >>
> >
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
> >>>>>>>>>>>> .java
> >>>>>>>>>>>> patching file
> >>>>>>>>>>>>
> >>>>>>
> >> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
> >>>>>>>>>>>> Hunk #1 FAILED at 93.
> >>>>>>>>>>>> Hunk #2 FAILED at 145.
> >>>>>>>>>>>> 2 out of 2 hunks FAILED -- saving rejects to file
> >>>>>>>>>>>>
> >>>>
> >>
> >
src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
> >>>>
> >>
> >
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> >>>>>>>>>>>> /src/patched/branch-3.2$ h ../patches/
> >>>>>>>>>>>>
> >>>>>>>>>>>> Could you advise as to which patches I need to apply, and
> > in
> >>>>>> what
> >>>>>>>>>>> order?
> >>>>>>>>>>>> -Todd
> >>>>>>>>>>>>
> >>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>>>>>>>>> Sent: Friday, July 31, 2009 9:51 PM
> >>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>>>>>
> >>>>>>>>>>>> Perfect! Thanks for the update, Todd.
> >>>>>>>>>>>>
> >>>>>>>>>>>> -Flavio
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks. You were right, I had a stale version of 479.
> >>>>>> Compilation
> >>>>>>>>>>>> succeeds and all tests pass on branch-3.2 with the
> > latest
> >>>>>> patches
> >>>>>>>>>>>> 473,
> >>>>>>>>>>>> 479, 481, and 491.
> >>>>>>>>>>>>
> >>>>>>>>>>>> -Todd
> >>>>>>>>>>>>
> >>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>>>>>>>>> Sent: Friday, July 31, 2009 7:48 PM
> >>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>>>>>
> >>>>>>>>>>>> It should be in 479. Perhaps you have a stale version
> > of
> >>>> the
> >>>>>>>>>>> patch.
> >>>>>>>>>>>> -Flavio
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Flavio,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'm getting a compilation error for patch 491:
> >>>>>>>>>>>>
> >>>>>>>>>>>> compile-main:
> >>>>>>>>>>>> [javac] Compiling 1 source file to
> >>>>>>>>>>>>
> >>>>>>
> >> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> >>>>>>>>>>>> src/p
> >>>>>>>>>>>> atched/branch-3.2/build/classes
> >>>>>>>>>>>> [javac]
> >>>>>>>>>>>>
> >>>>>>
> >> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> >>>>>>>>>>>> src/p
> >>>>>>>>>>>>
> >>>>>>
> >> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
> >>>>>>>>>>>> FastL
> >>>>>>>>>>>> eaderElection.java:601: cannot find symbol
> >>>>>>>>>>>> [javac] symbol : method getWeight(long)
> >>>>>>>>>>>> [javac] location: interface
> >>>>>>>>>>>>
> >> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
> >>>>>>>>>>>> [javac]
> >>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> >>>>>>>>>>>> [javac]
> >>>>>> ^
> >>>>>>>>>>>> [javac] 1 error
> >>>>>>>>>>>>
> >>>>>>>>>>>> I see a reference to getWeight in both
> >>>>>> FastLeaderElection.java
> >>>>>>>>>>> in
> >>>>>>>>>>>> patch
> >>>>>>>>>>>> 491:
> >>>>>>>>>>>>
> >>>>>>>>>>>> patches/ZOOKEEPER-491.patch:+
> >>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> >>>>>>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
> >>>>>>>>>>>> FastLeaderElection.java
> >>>>>>>>>>>> :
> >>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
> >>>>>>>>>>>> 0)
> >>>>>>>>>>>>
> >>>>>>>>>>>> However, I don't see a reference to this method in
> >> patches
> >>>>>> 473,
> >>>>>>>>>>>> 479,
> >>>>>>>>>>>> or
> >>>>>>>>>>>> 481. I also don't see a reference to this method in
> > the
> >>>>>>>>> trunk...
> >>>>>>>>>>>> -Todd
> >>>>>>>>>>>>
> >>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>> From: Todd Greenwood
> > [mailto:toddg@audiencescience.com]
> >>>>>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
> >>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
> >>>>>>>>>>>>
> >>>>>>>>>>>> Ok, I'll apply that patch and report back.
> >>>>>>>>>>>> -Todd
> >>>>>>>>>>>>
> >>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
> >>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>>>>>
> >>>>>>>>>>>> You're missing 491 from your set of patches.
> >>>>>>>>>>>>
> >>>>>>>>>>>> -Flavio
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> This repro's in both branch-3.2, and
> >>>>>> branch-3.2+patches(473,
> >>>>>>>>>>>> 479,
> >>>>>>>>>>>> 481).
> >>>>>>>>>>>>
> >>>>>>>>>>>> Basically, it seems like the nodes are electing
> >>>>>> pd4-zook02
> >>>>>>>>> to
> >>>>>>>>>>>> be
> >>>>>>>>>>>> the
> >>>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
> >>>>>>>>>>> supposed
> >>>>>>>>>>>> to
> >>>>>>>>>>>> be
> >>>>>>>>>>>> and
> >>>>>>>>>>>> then disconnects everyone. Then they re-elect it
> > again,
> >>>>>> and
> >>>>>>>>>>> it
> >>>>>>>>>>>> loops
> >>>>>>>>>>>> over and over.
> >>>>>>>>>>>>
> >>>>>>>>>>>> -------------
> >>>>>>>>>>>> Server config
> >>>>>>>>>>>> -------------
> >>>>>>>>>>>>
> >>>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
> >>>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
> >>>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
> >>>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
> >>>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
> >>>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
> >>>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
> >>>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
> >>>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
> >>>>>>>>>>>>
> >>>>>>>>>>>> group.1:1:2:3:4:5
> >>>>>>>>>>>> weight.1=1
> >>>>>>>>>>>> weight.2=1
> >>>>>>>>>>>> weight.3=1
> >>>>>>>>>>>> weight.4=1
> >>>>>>>>>>>> weight.5=1
> >>>>>>>>>>>>
> >>>>>>>>>>>> group.2:6:7:8:9
> >>>>>>>>>>>> weight.6=0
> >>>>>>>>>>>> weight.7=0
> >>>>>>>>>>>> weight.8=0
> >>>>>>>>>>>> weight.9=0
> >>>>>>>>>>>>
> >>>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
> >>>>>>>>>>> different
> >>>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
> >>>>>>>>> machines
> >>>>>>>>>>>> in
> >>>>>>>>>>>> dc1
> >>>>>>>>>>>> have voting rights, and the ability to become a
> > leader.
> >>>>>> The
> >>>>>>>>>>>> machines
> >>>>>>>>>>>> in
> >>>>>>>>>>>> the pods all have a weight of zero, and are not
> >> expected
> >>>>>> to
> >>>>>>>>>>>> become
> >>>>>>>>>>>> leaders, or to vote on transactions.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Let me know what I can do to help resolve this issue.
> >>>>>>>>>>>>
> >>>>>>>>>>>> -Todd
> >>>>
Re: Unending Leader Elections in WAN deploy
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi Todd,
Can you attach the files to the jira? I will takea look at this and will
get back to you by end of day today.
Thanks
mahadev
On 8/4/09 4:56 PM, "Todd Greenwood" <to...@audiencescience.com> wrote:
> Looks like we're not getting *any* leader elected now.... Logs attached.
>
>> -----Original Message-----
>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>> Sent: Tuesday, August 04, 2009 4:07 PM
>> To: zookeeper-dev@hadoop.apache.org
>> Subject: RE: Unending Leader Elections in WAN deploy
>>
>> Patrick, thanks! I'll forward on to IT and I'll report back to you
>> shortly...
>>
>>> -----Original Message-----
>>> From: Patrick Hunt [mailto:phunt@apache.org]
>>> Sent: Tuesday, August 04, 2009 3:55 PM
>>> To: zookeeper-dev@hadoop.apache.org
>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>
>>> Todd, Mahadev and I looked at this and it turns out to be a
>> regression.
>>> Ironically a patch I created for 3.2 branch to add quorum tests
>> actually
>>> broke the quorum config -- a default value for a config parameter
> was
>>> lost. I'm going to submit a patch asap to get the default back, but
>> for
>>> the time being you can set:
>>>
>>> electionAlg=3
>>>
>>> in each of your config files.
>>>
>>> You should see reference to FastLeaderElection in your log files if
>> this
>>> parameter is set correctly.
>>>
>>> Sorry for the trouble,
>>>
>>> Patrick
>>>
>>> Todd Greenwood wrote:
>>>> Mahadev,
>>>>
>>>> I just heard from IT that this build behaves in exactly the same
> way
>> as
>>>> previous versions, e.g. we get continuous leader elections that
>>>> disconnect the followers and then get re-elected, and
>> disconnect...etc.
>>>>
>>>> This is from a fresh sync to the 3.2 branch:
>>>>
>>>> svn co
>>>>
> http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2
>>>> ./branch-3.2
>>>>
>>>> CHANGES.TXT show the various fixes included:
>>>>
>>>>
>>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>> /src/original$ head -n 50 branch-3.2/CHANGES.txt
>>>> Release 3.2.1
>>>>
>>>> Backward compatibile changes:
>>>>
>>>> BUGFIXES:
>>>> ZOOKEEPER-468. avoid compile warning in send_auth_info(). (chris
>> via
>>>> flavio)
>>>>
>>>> ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten (chris
>> via
>>>> mahadev)
>>>>
>>>> ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via
> mahadev)
>>>>
>>>> ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris
> via
>>>> mahadev)
>>>>
>>>> ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure)
>>>> (giri via mahadev)
>>>>
>>>> ZOOKEEPER-467. Change log level in BookieHandle (flavio via
>> mahadev)
>>>>
>>>> ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent
>> immediate
>>>> failure. (chris via mahadev)
>>>>
>>>> ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev
>> via
>>>> phunt)
>>>>
>>>> ZOOKEEPER-457. Make ZookeeperMain public, support for HBase (and
>>>> other)
>>>> embedded clients (ryan rawson via phunt)
>>>>
>>>> ZOOKEEPER-481. Add lastMessageSent to QuorumCnxManager. (flavio
>> via
>>>> mahadev)
>>>>
>>>> ZOOKEEPER-479. QuorumHierarchical does not count groups
> correctly
>>>> (flavio via mahadev)
>>>>
>>>> ZOOKEEPER-466. crash on zookeeper_close() when using auth with
>> empty
>>>> cert
>>>> (Chris Darroch via phunt)
>>>>
>>>> ZOOKEEPER-480. FLE should perform leader check when node is not
>>>> leading and
>>>> add vote of follower (flavio via mahadev)
>>>>
>>>> ZOOKEEPER-491. Prevent zero-weight servers from being elected
>> (flavio
>>>> via
>>>> mahadev)
>>>>
>>>> What can I do to assist you with this issue?
>>>>
>>>> -Todd
>>>>
>>>>> -----Original Message-----
>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>> Sent: Tuesday, August 04, 2009 12:43 PM
>>>>> To: zookeeper-dev@hadoop.apache.org
>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>
>>>>> Hi todd,
>>>>> comments in line
>>>>>
>>>>>
>>>>> On 8/4/09 12:38 PM, "Todd Greenwood" <to...@audiencescience.com>
>>>> wrote:
>>>>>> Mahadev,
>>>>>>
>>>>>> Some quick questions:
>>>>>>
>>>>>> 1. Version
>>>>>>
>>>>>> I see that the CHANGES.txt calls this 3.2.1, but the build.xml
> is
>>>> still
>>>>>> calling this 3.2.0. Should this be rev'd, and am I correct in
>>>> calling
>>>>>> this release 3.2.1?
>>>>> Yes the release is 3.2.1. The build.xml will be fixed as soon as
> we
>>>> tag
>>>>> the
>>>>> release.
>>>>>
>>>>>> 2. Build targets
>>>>>>
>>>>>> The package target fails b/c the create-cppunit-configure target
>>>> fails
>>>>>> due to various problems w/ respect to autoconf. Are these
>>>> dependencies
>>>>>> documented somewhere ? I'd like to have a fully building system.
>>>>>>
>>>>>> create-cppunit-configure:
>>>>>> [exec] Can't exec "libtoolize": No such file or directory
> at
>>>>>> /usr/bin/autoreconf line 188.
>>>>>> [exec] Use of uninitialized value $libtoolize in pattern
>> match
>>>>>> (m//) at /usr/bin/autoreconf line 188.
>>>>>> [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT'
> not
>>>> found
>>>>>> in library
>>>>>> [exec] configure.ac:33: error: possibly undefined macro:
>>>>>> AM_PATH_CPPUNIT
>>>>>> [exec] If this token and others are legitimate,
> please
>>>> use
>>>>>> m4_pattern_allow.
>>>>>> [exec] See the Autoconf documentation.
>>>>>> [exec] configure.ac:53: error: possibly undefined macro:
>>>>>> AC_PROG_LIBTOOL
>>>>>> [exec] autoreconf: /usr/bin/autoconf failed with exit
> status:
>> 1
>>>>>>
>>>>> You need auto tools to run this. Please read the README for
>> building c
>>>>> client library at src/c/ for the installation requirements.
>>>>>> 3. Sync failure:
>>>>>>
>>>>>> This is still failing.
>>>>>>
>>>>>> svn: URL
>>>>>>
> 'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch'
>>>>>> doesn't exist
>>>>>>
>>>>> Yes this hasn't been fixed yet!
>>>>>
>>>>> Thanks
>>>>> mahadev
>>>>>> -Todd
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Todd Greenwood
>>>>>>> Sent: Tuesday, August 04, 2009 11:26 AM
>>>>>>> To: 'zookeeper-user@hadoop.apache.org'
>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>
>>>>>>> Great news. Thank you Mahadev. I'll report our findings later
>>>> today.
>>>>>>> -Todd
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>>>>> Sent: Tuesday, August 04, 2009 11:20 AM
>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>
>>>>>>>> Hi Todd,
>>>>>>>> I just committed 480 and 491. You can checkout the 3.2 branch
>>>> now.
>>>>>>>> Thanks
>>>>>>>> mahadev
>>>>>>>>
>>>>>>>>
>>>>>>>> On 8/3/09 4:29 PM, "Todd Greenwood"
> <to...@audiencescience.com>
>>>>>> wrote:
>>>>>>>>> That'd be perfect. Thanks!
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>>>>>>> Sent: Monday, August 03, 2009 4:24 PM
>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>
>>>>>>>>>> Hi Todd,
>>>>>>>>>> Most of the patches that you mention should be in the
> branch
>>>>>> 3.2 by
>>>>>>>>> tomm
>>>>>>>>>> or so. 481, 479 are already in. 480 and 491 should be in by
>>>> tomm.
>>>>>>>>> Would
>>>>>>>>>> that
>>>>>>>>>> suffice for you?
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> mahadev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 8/3/09 4:21 PM, "Todd Greenwood"
>> <to...@audiencescience.com>
>>>>>>> wrote:
>>>>>>>>>>> Another problem...I've reverted to the latest versions of
> the
>>>>>>>>> patches
>>>>>>>>>>> that are not specific to branch-3.2, and I'm getting two
>>>>>> compilation
>>>>>>>>>>> errors:
>>>>>>>>>>>
>>>>>>>>>>> build-generated:
>>>>>>>>>>> [javac] Compiling 44 source files to
>>>>>>>>>>>
>>>>
>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>>
>>>>>>>>>>> compile-main:
>>>>>>>>>>> [javac] Compiling 2 source files to
>>>>>>>>>>>
>>>>
>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>> [javac]
>>>>>>>>>>>
>>>>
>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>> atched/branch-
>>>>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>>>>>>> mStats.java:30: name clash: getQuorumPeers() and
>>>>>> getQuorumPeers()
>>>>>>>>> have
>>>>>>>>>>> the same erasure
>>>>>>>>>>> [javac] public String[] getQuorumPeers();
>>>>>>>>>>> [javac] ^
>>>>>>>>>>> [javac]
>>>>>>>>>>>
>>>>
>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>> atched/branch-
>>>>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>>>>>>> mStats.java:31: name clash: getServerState() and
>>>>>> getServerState()
>>>>>>>>> have
>>>>>>>>>>> the same erasure
>>>>>>>>>>> [javac] public String getServerState();
>>>>>>>>>>> [javac] ^
>>>>>>>>>>> [javac] 2 errors
>>>>>>>>>>>
>>>>>>>>>>> My build process is pretty simple:
>>>>>>>>>>>
>>>>>>>>>>> 1. copy the branch-3.2 source to a temp directory
>>>>>>>>>>> (src/patched/branch-3.2)
>>>>>>>>>>> 2. apply the ZOOKEEPER patches in my patches directory
>>>>>>>>>>> 3. build zookeeper in the temp directory
>>>>>>>>>>>
>>>>>>>>>>> -Todd
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>>>>>>>>>> Sent: Monday, August 03, 2009 4:09 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> Flavio,
>>>>>>>>>>>> I notice that you've updated the patches referenced for
> the
>>>> WAN
>>>>>>>>>>>> deployment. There appears to be an order dependency w/
>> respect
>>>>>> to
>>>>>>>>>>> these
>>>>>>>>>>>> four patches...
>>>>>>>>>>>>
>>>>>>>>>>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
>>>>>>>>>>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
>>>>>>>>>>>>
>>>>>>>>>>>> 473 -> 479 (479 fails)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>
>>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>>>>>>> /src/patched/branch-3.2$ patch -p0 <
>>>>>>>>>>>> ../patches/ZOOKEEPER-479-branch3.2.patch
>>>>>>>>>>>> patching file
>>>>>>>>>>>>
>>>>
>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
>>>>>>>>>>>> ical.java
>>>>>>>>>>>> patching file
>>>>>>>>>>>>
>>>>
>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
>>>>>>>>>>>> patching file
>>>>>>>>>>>>
>>>>
>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
>>>>>>>>>>>> .java
>>>>>>>>>>>> patching file
>>>>>>>>>>>>
>>>>>>
>> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
>>>>>>>>>>>> Hunk #1 FAILED at 93.
>>>>>>>>>>>> Hunk #2 FAILED at 145.
>>>>>>>>>>>> 2 out of 2 hunks FAILED -- saving rejects to file
>>>>>>>>>>>>
>>>>
>>
> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
>>>>
>>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>>>>>>> /src/patched/branch-3.2$ h ../patches/
>>>>>>>>>>>>
>>>>>>>>>>>> Could you advise as to which patches I need to apply, and
> in
>>>>>> what
>>>>>>>>>>> order?
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>> Sent: Friday, July 31, 2009 9:51 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> Perfect! Thanks for the update, Todd.
>>>>>>>>>>>>
>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>
>>>>>>>>>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks. You were right, I had a stale version of 479.
>>>>>> Compilation
>>>>>>>>>>>> succeeds and all tests pass on branch-3.2 with the
> latest
>>>>>> patches
>>>>>>>>>>>> 473,
>>>>>>>>>>>> 479, 481, and 491.
>>>>>>>>>>>>
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:48 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> It should be in 479. Perhaps you have a stale version
> of
>>>> the
>>>>>>>>>>> patch.
>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>
>>>>>>>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Flavio,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm getting a compilation error for patch 491:
>>>>>>>>>>>>
>>>>>>>>>>>> compile-main:
>>>>>>>>>>>> [javac] Compiling 1 source file to
>>>>>>>>>>>>
>>>>>>
>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>>> src/p
>>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>>> [javac]
>>>>>>>>>>>>
>>>>>>
>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>>> src/p
>>>>>>>>>>>>
>>>>>>
>> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>>> FastL
>>>>>>>>>>>> eaderElection.java:601: cannot find symbol
>>>>>>>>>>>> [javac] symbol : method getWeight(long)
>>>>>>>>>>>> [javac] location: interface
>>>>>>>>>>>>
>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
>>>>>>>>>>>> [javac]
>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>>> [javac]
>>>>>> ^
>>>>>>>>>>>> [javac] 1 error
>>>>>>>>>>>>
>>>>>>>>>>>> I see a reference to getWeight in both
>>>>>> FastLeaderElection.java
>>>>>>>>>>> in
>>>>>>>>>>>> patch
>>>>>>>>>>>> 491:
>>>>>>>>>>>>
>>>>>>>>>>>> patches/ZOOKEEPER-491.patch:+
>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>>> FastLeaderElection.java
>>>>>>>>>>>> :
>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
>>>>>>>>>>>> 0)
>>>>>>>>>>>>
>>>>>>>>>>>> However, I don't see a reference to this method in
>> patches
>>>>>> 473,
>>>>>>>>>>>> 479,
>>>>>>>>>>>> or
>>>>>>>>>>>> 481. I also don't see a reference to this method in
> the
>>>>>>>>> trunk...
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Todd Greenwood
> [mailto:toddg@audiencescience.com]
>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> Ok, I'll apply that patch and report back.
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> You're missing 491 from your set of patches.
>>>>>>>>>>>>
>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>
>>>>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> This repro's in both branch-3.2, and
>>>>>> branch-3.2+patches(473,
>>>>>>>>>>>> 479,
>>>>>>>>>>>> 481).
>>>>>>>>>>>>
>>>>>>>>>>>> Basically, it seems like the nodes are electing
>>>>>> pd4-zook02
>>>>>>>>> to
>>>>>>>>>>>> be
>>>>>>>>>>>> the
>>>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
>>>>>>>>>>> supposed
>>>>>>>>>>>> to
>>>>>>>>>>>> be
>>>>>>>>>>>> and
>>>>>>>>>>>> then disconnects everyone. Then they re-elect it
> again,
>>>>>> and
>>>>>>>>>>> it
>>>>>>>>>>>> loops
>>>>>>>>>>>> over and over.
>>>>>>>>>>>>
>>>>>>>>>>>> -------------
>>>>>>>>>>>> Server config
>>>>>>>>>>>> -------------
>>>>>>>>>>>>
>>>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
>>>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
>>>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
>>>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
>>>>>>>>>>>>
>>>>>>>>>>>> group.1:1:2:3:4:5
>>>>>>>>>>>> weight.1=1
>>>>>>>>>>>> weight.2=1
>>>>>>>>>>>> weight.3=1
>>>>>>>>>>>> weight.4=1
>>>>>>>>>>>> weight.5=1
>>>>>>>>>>>>
>>>>>>>>>>>> group.2:6:7:8:9
>>>>>>>>>>>> weight.6=0
>>>>>>>>>>>> weight.7=0
>>>>>>>>>>>> weight.8=0
>>>>>>>>>>>> weight.9=0
>>>>>>>>>>>>
>>>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
>>>>>>>>>>> different
>>>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
>>>>>>>>> machines
>>>>>>>>>>>> in
>>>>>>>>>>>> dc1
>>>>>>>>>>>> have voting rights, and the ability to become a
> leader.
>>>>>> The
>>>>>>>>>>>> machines
>>>>>>>>>>>> in
>>>>>>>>>>>> the pods all have a weight of zero, and are not
>> expected
>>>>>> to
>>>>>>>>>>>> become
>>>>>>>>>>>> leaders, or to vote on transactions.
>>>>>>>>>>>>
>>>>>>>>>>>> Let me know what I can do to help resolve this issue.
>>>>>>>>>>>>
>>>>>>>>>>>> -Todd
>>>>
Re: Unending Leader Elections in WAN deploy
Posted by Patrick Hunt <ph...@apache.org>.
(I see the same error in fle0weighttest using latest 3.2 btw)
Patrick Hunt wrote:
> Mahadev/Flavio -- looks like 0 weight is still busted, fle0weighttest is
> actually failing on my machine, however it's reported as success:
> ------------- Standard Error -----------------
> Exception in thread "Thread-108" junit.framework.AssertionFailedError:
> Elected zero-weight server
> at junit.framework.Assert.fail(Assert.java:47)
> at
> org.apache.zookeeper.test.FLEZeroWeightTest$LEThread.run(FLEZeroWeightTest.java:138)
>
> ------------- ---------------- ---------------
>
> this is probably due because the test is calling assert in a thread
> other than the main test thread - which junit will not track/knowabout.
>
> One problem I see with these tests (0weight test I looked at) -- it
> doesn't have a client attempt to connect to the various servers as part
> of declaring success. Really we should only consider "success"ful test
> (ie assert that) if a client can connect to each server in the cluster
> and change/seechanges. As part of fixing this we really need to do a
> sanity check by testing the various command lines and checking that a
> client can connect.
>
> I'm not even sure FLEnewepochtest/fletest/etc... are passing either. new
> epoch seems to just thrash...
>
> Also I tried 3 & 5 server quorums "by hand from the command line" with 0
> weight and they see similar issues to what Todd is seeing.
>
> I'm using the latest code in mainline btw.
>
> Patrick
>
> Mahadev Konar wrote:
>> Hi todd,
>> I see a lot of
>> java.net.ConnectException: Connection refused
>> at sun.nio.ch.Net.connect(Native Method)
>> at
>> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
>> at java.nio.channels.SocketChannel.open(SocketChannel.java:146)
>> at
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxMana
>>
>> ger.java:324)
>> at
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.
>>
>> java:304)
>> at
>> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender
>>
>> .process(FastLeaderElection.java:317)
>> at
>> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender
>>
>> .run(FastLeaderElection.java:290)
>> at java.lang.Thread.run(Thread.java:619)
>>
>>
>> Is it possible that there is some firewall? Can all the servers 1-9
>> connect
>> to all the others using ports that you specified in zoo.cfg i.e
>> 2888/3888?
>>
>>
>> Thanks
>> mahadev
>>
>>
>> On 8/4/09 4:56 PM, "Todd Greenwood" <to...@audiencescience.com> wrote:
>>
>>> Looks like we're not getting *any* leader elected now.... Logs attached.
>>>
>>>> -----Original Message-----
>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>> Sent: Tuesday, August 04, 2009 4:07 PM
>>>> To: zookeeper-dev@hadoop.apache.org
>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>
>>>> Patrick, thanks! I'll forward on to IT and I'll report back to you
>>>> shortly...
>>>>
>>>>> -----Original Message-----
>>>>> From: Patrick Hunt [mailto:phunt@apache.org]
>>>>> Sent: Tuesday, August 04, 2009 3:55 PM
>>>>> To: zookeeper-dev@hadoop.apache.org
>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>
>>>>> Todd, Mahadev and I looked at this and it turns out to be a
>>>> regression.
>>>>> Ironically a patch I created for 3.2 branch to add quorum tests
>>>> actually
>>>>> broke the quorum config -- a default value for a config parameter
>>> was
>>>>> lost. I'm going to submit a patch asap to get the default back, but
>>>> for
>>>>> the time being you can set:
>>>>>
>>>>> electionAlg=3
>>>>>
>>>>> in each of your config files.
>>>>>
>>>>> You should see reference to FastLeaderElection in your log files if
>>>> this
>>>>> parameter is set correctly.
>>>>>
>>>>> Sorry for the trouble,
>>>>>
>>>>> Patrick
>>>>>
>>>>> Todd Greenwood wrote:
>>>>>> Mahadev,
>>>>>>
>>>>>> I just heard from IT that this build behaves in exactly the same
>>> way
>>>> as
>>>>>> previous versions, e.g. we get continuous leader elections that
>>>>>> disconnect the followers and then get re-elected, and
>>>> disconnect...etc.
>>>>>> This is from a fresh sync to the 3.2 branch:
>>>>>>
>>>>>> svn co
>>>>>>
>>> http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2
>>>>>> ./branch-3.2
>>>>>>
>>>>>> CHANGES.TXT show the various fixes included:
>>>>>>
>>>>>>
>>> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>> /src/original$ head -n 50 branch-3.2/CHANGES.txt
>>>>>> Release 3.2.1
>>>>>>
>>>>>> Backward compatibile changes:
>>>>>>
>>>>>> BUGFIXES:
>>>>>> ZOOKEEPER-468. avoid compile warning in send_auth_info(). (chris
>>>> via
>>>>>> flavio)
>>>>>>
>>>>>> ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten (chris
>>>> via
>>>>>> mahadev)
>>>>>>
>>>>>> ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via
>>> mahadev)
>>>>>> ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris
>>> via
>>>>>> mahadev)
>>>>>>
>>>>>> ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure)
>>>>>> (giri via mahadev)
>>>>>>
>>>>>> ZOOKEEPER-467. Change log level in BookieHandle (flavio via
>>>> mahadev)
>>>>>> ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent
>>>> immediate
>>>>>> failure. (chris via mahadev)
>>>>>>
>>>>>> ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev
>>>> via
>>>>>> phunt)
>>>>>>
>>>>>> ZOOKEEPER-457. Make ZookeeperMain public, support for HBase (and
>>>>>> other)
>>>>>> embedded clients (ryan rawson via phunt)
>>>>>>
>>>>>> ZOOKEEPER-481. Add lastMessageSent to QuorumCnxManager. (flavio
>>>> via
>>>>>> mahadev)
>>>>>>
>>>>>> ZOOKEEPER-479. QuorumHierarchical does not count groups
>>> correctly
>>>>>> (flavio via mahadev)
>>>>>>
>>>>>> ZOOKEEPER-466. crash on zookeeper_close() when using auth with
>>>> empty
>>>>>> cert
>>>>>> (Chris Darroch via phunt)
>>>>>>
>>>>>> ZOOKEEPER-480. FLE should perform leader check when node is not
>>>>>> leading and
>>>>>> add vote of follower (flavio via mahadev)
>>>>>>
>>>>>> ZOOKEEPER-491. Prevent zero-weight servers from being elected
>>>> (flavio
>>>>>> via
>>>>>> mahadev)
>>>>>>
>>>>>> What can I do to assist you with this issue?
>>>>>>
>>>>>> -Todd
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>>>> Sent: Tuesday, August 04, 2009 12:43 PM
>>>>>>> To: zookeeper-dev@hadoop.apache.org
>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>
>>>>>>> Hi todd,
>>>>>>> comments in line
>>>>>>>
>>>>>>>
>>>>>>> On 8/4/09 12:38 PM, "Todd Greenwood" <to...@audiencescience.com>
>>>>>> wrote:
>>>>>>>> Mahadev,
>>>>>>>>
>>>>>>>> Some quick questions:
>>>>>>>>
>>>>>>>> 1. Version
>>>>>>>>
>>>>>>>> I see that the CHANGES.txt calls this 3.2.1, but the build.xml
>>> is
>>>>>> still
>>>>>>>> calling this 3.2.0. Should this be rev'd, and am I correct in
>>>>>> calling
>>>>>>>> this release 3.2.1?
>>>>>>> Yes the release is 3.2.1. The build.xml will be fixed as soon as
>>> we
>>>>>> tag
>>>>>>> the
>>>>>>> release.
>>>>>>>
>>>>>>>> 2. Build targets
>>>>>>>>
>>>>>>>> The package target fails b/c the create-cppunit-configure target
>>>>>> fails
>>>>>>>> due to various problems w/ respect to autoconf. Are these
>>>>>> dependencies
>>>>>>>> documented somewhere ? I'd like to have a fully building system.
>>>>>>>>
>>>>>>>> create-cppunit-configure:
>>>>>>>> [exec] Can't exec "libtoolize": No such file or directory
>>> at
>>>>>>>> /usr/bin/autoreconf line 188.
>>>>>>>> [exec] Use of uninitialized value $libtoolize in pattern
>>>> match
>>>>>>>> (m//) at /usr/bin/autoreconf line 188.
>>>>>>>> [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT'
>>> not
>>>>>> found
>>>>>>>> in library
>>>>>>>> [exec] configure.ac:33: error: possibly undefined macro:
>>>>>>>> AM_PATH_CPPUNIT
>>>>>>>> [exec] If this token and others are legitimate,
>>> please
>>>>>> use
>>>>>>>> m4_pattern_allow.
>>>>>>>> [exec] See the Autoconf documentation.
>>>>>>>> [exec] configure.ac:53: error: possibly undefined macro:
>>>>>>>> AC_PROG_LIBTOOL
>>>>>>>> [exec] autoreconf: /usr/bin/autoconf failed with exit
>>> status:
>>>> 1
>>>>>>> You need auto tools to run this. Please read the README for
>>>> building c
>>>>>>> client library at src/c/ for the installation requirements.
>>>>>>>> 3. Sync failure:
>>>>>>>>
>>>>>>>> This is still failing.
>>>>>>>>
>>>>>>>> svn: URL
>>>>>>>>
>>> 'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch'
>>>>>>>> doesn't exist
>>>>>>>>
>>>>>>> Yes this hasn't been fixed yet!
>>>>>>>
>>>>>>> Thanks
>>>>>>> mahadev
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Todd Greenwood
>>>>>>>>> Sent: Tuesday, August 04, 2009 11:26 AM
>>>>>>>>> To: 'zookeeper-user@hadoop.apache.org'
>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>
>>>>>>>>> Great news. Thank you Mahadev. I'll report our findings later
>>>>>> today.
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>>>>>>> Sent: Tuesday, August 04, 2009 11:20 AM
>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>
>>>>>>>>>> Hi Todd,
>>>>>>>>>> I just committed 480 and 491. You can checkout the 3.2 branch
>>>>>> now.
>>>>>>>>>> Thanks
>>>>>>>>>> mahadev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 8/3/09 4:29 PM, "Todd Greenwood"
>>> <to...@audiencescience.com>
>>>>>>>> wrote:
>>>>>>>>>>> That'd be perfect. Thanks!
>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>>>>>>>>> Sent: Monday, August 03, 2009 4:24 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Todd,
>>>>>>>>>>>> Most of the patches that you mention should be in the
>>> branch
>>>>>>>> 3.2 by
>>>>>>>>>>> tomm
>>>>>>>>>>>> or so. 481, 479 are already in. 480 and 491 should be in by
>>>>>> tomm.
>>>>>>>>>>> Would
>>>>>>>>>>>> that
>>>>>>>>>>>> suffice for you?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> mahadev
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 8/3/09 4:21 PM, "Todd Greenwood"
>>>> <to...@audiencescience.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>> Another problem...I've reverted to the latest versions of
>>> the
>>>>>>>>>>> patches
>>>>>>>>>>>>> that are not specific to branch-3.2, and I'm getting two
>>>>>>>> compilation
>>>>>>>>>>>>> errors:
>>>>>>>>>>>>>
>>>>>>>>>>>>> build-generated:
>>>>>>>>>>>>> [javac] Compiling 44 source files to
>>>>>>>>>>>>>
>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>>>>
>>>>>>>>>>>>> compile-main:
>>>>>>>>>>>>> [javac] Compiling 2 source files to
>>>>>>>>>>>>>
>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>>>> [javac]
>>>>>>>>>>>>>
>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>>>> atched/branch-
>>>>>>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>>>>>>>>> mStats.java:30: name clash: getQuorumPeers() and
>>>>>>>> getQuorumPeers()
>>>>>>>>>>> have
>>>>>>>>>>>>> the same erasure
>>>>>>>>>>>>> [javac] public String[] getQuorumPeers();
>>>>>>>>>>>>> [javac] ^
>>>>>>>>>>>>> [javac]
>>>>>>>>>>>>>
>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>>>> atched/branch-
>>>>>>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>>>>>>>>> mStats.java:31: name clash: getServerState() and
>>>>>>>> getServerState()
>>>>>>>>>>> have
>>>>>>>>>>>>> the same erasure
>>>>>>>>>>>>> [javac] public String getServerState();
>>>>>>>>>>>>> [javac] ^
>>>>>>>>>>>>> [javac] 2 errors
>>>>>>>>>>>>>
>>>>>>>>>>>>> My build process is pretty simple:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. copy the branch-3.2 source to a temp directory
>>>>>>>>>>>>> (src/patched/branch-3.2)
>>>>>>>>>>>>> 2. apply the ZOOKEEPER patches in my patches directory
>>>>>>>>>>>>> 3. build zookeeper in the temp directory
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>>>>>>>>>>>> Sent: Monday, August 03, 2009 4:09 PM
>>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Flavio,
>>>>>>>>>>>>>> I notice that you've updated the patches referenced for
>>> the
>>>>>> WAN
>>>>>>>>>>>>>> deployment. There appears to be an order dependency w/
>>>> respect
>>>>>>>> to
>>>>>>>>>>>>> these
>>>>>>>>>>>>>> four patches...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
>>>>>>>>>>>>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 473 -> 479 (479 fails)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>>>>>>>>> /src/patched/branch-3.2$ patch -p0 <
>>>>>>>>>>>>>> ../patches/ZOOKEEPER-479-branch3.2.patch
>>>>>>>>>>>>>> patching file
>>>>>>>>>>>>>>
>>> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
>>>>>>>>>>>>>> ical.java
>>>>>>>>>>>>>> patching file
>>>>>>>>>>>>>>
>>> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
>>>>>>>>>>>>>> patching file
>>>>>>>>>>>>>>
>>> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
>>>>>>>>>>>>>> .java
>>>>>>>>>>>>>> patching file
>>>>>>>>>>>>>>
>>>> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
>>>>>>>>>>>>>> Hunk #1 FAILED at 93.
>>>>>>>>>>>>>> Hunk #2 FAILED at 145.
>>>>>>>>>>>>>> 2 out of 2 hunks FAILED -- saving rejects to file
>>>>>>>>>>>>>>
>>> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
>>> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>>>>>>>>> /src/patched/branch-3.2$ h ../patches/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Could you advise as to which patches I need to apply, and
>>> in
>>>>>>>> what
>>>>>>>>>>>>> order?
>>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>>>> Sent: Friday, July 31, 2009 9:51 PM
>>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Perfect! Thanks for the update, Todd.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks. You were right, I had a stale version of 479.
>>>>>>>> Compilation
>>>>>>>>>>>>>> succeeds and all tests pass on branch-3.2 with the
>>> latest
>>>>>>>> patches
>>>>>>>>>>>>>> 473,
>>>>>>>>>>>>>> 479, 481, and 491.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:48 PM
>>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It should be in 479. Perhaps you have a stale version
>>> of
>>>>>> the
>>>>>>>>>>>>> patch.
>>>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Flavio,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm getting a compilation error for patch 491:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> compile-main:
>>>>>>>>>>>>>> [javac] Compiling 1 source file to
>>>>>>>>>>>>>>
>>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>>>>> src/p
>>>>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>>>>> [javac]
>>>>>>>>>>>>>>
>>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>>>>> src/p
>>>>>>>>>>>>>>
>>>> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>>>>> FastL
>>>>>>>>>>>>>> eaderElection.java:601: cannot find symbol
>>>>>>>>>>>>>> [javac] symbol : method getWeight(long)
>>>>>>>>>>>>>> [javac] location: interface
>>>>>>>>>>>>>>
>>>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
>>>>>>>>>>>>>> [javac]
>>>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>>>>> [javac]
>>>>>>>> ^
>>>>>>>>>>>>>> [javac] 1 error
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I see a reference to getWeight in both
>>>>>>>> FastLeaderElection.java
>>>>>>>>>>>>> in
>>>>>>>>>>>>>> patch
>>>>>>>>>>>>>> 491:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> patches/ZOOKEEPER-491.patch:+
>>>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>>>>> FastLeaderElection.java
>>>>>>>>>>>>>> :
>>>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
>>>>>>>>>>>>>> 0)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> However, I don't see a reference to this method in
>>>> patches
>>>>>>>> 473,
>>>>>>>>>>>>>> 479,
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>> 481. I also don't see a reference to this method in
>>> the
>>>>>>>>>>> trunk...
>>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>> From: Todd Greenwood
>>> [mailto:toddg@audiencescience.com]
>>>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
>>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ok, I'll apply that patch and report back.
>>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
>>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You're missing 491 from your set of patches.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This repro's in both branch-3.2, and
>>>>>>>> branch-3.2+patches(473,
>>>>>>>>>>>>>> 479,
>>>>>>>>>>>>>> 481).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Basically, it seems like the nodes are electing
>>>>>>>> pd4-zook02
>>>>>>>>>>> to
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
>>>>>>>>>>>>> supposed
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> then disconnects everyone. Then they re-elect it
>>> again,
>>>>>>>> and
>>>>>>>>>>>>> it
>>>>>>>>>>>>>> loops
>>>>>>>>>>>>>> over and over.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -------------
>>>>>>>>>>>>>> Server config
>>>>>>>>>>>>>> -------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
>>>>>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
>>>>>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
>>>>>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
>>>>>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
>>>>>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
>>>>>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
>>>>>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
>>>>>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> group.1:1:2:3:4:5
>>>>>>>>>>>>>> weight.1=1
>>>>>>>>>>>>>> weight.2=1
>>>>>>>>>>>>>> weight.3=1
>>>>>>>>>>>>>> weight.4=1
>>>>>>>>>>>>>> weight.5=1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> group.2:6:7:8:9
>>>>>>>>>>>>>> weight.6=0
>>>>>>>>>>>>>> weight.7=0
>>>>>>>>>>>>>> weight.8=0
>>>>>>>>>>>>>> weight.9=0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
>>>>>>>>>>>>> different
>>>>>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
>>>>>>>>>>> machines
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> dc1
>>>>>>>>>>>>>> have voting rights, and the ability to become a
>>> leader.
>>>>>>>> The
>>>>>>>>>>>>>> machines
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> the pods all have a weight of zero, and are not
>>>> expected
>>>>>>>> to
>>>>>>>>>>>>>> become
>>>>>>>>>>>>>> leaders, or to vote on transactions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let me know what I can do to help resolve this issue.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Todd
>>
Re: Unending Leader Elections in WAN deploy
Posted by Patrick Hunt <ph...@apache.org>.
Mahadev/Flavio -- looks like 0 weight is still busted, fle0weighttest is
actually failing on my machine, however it's reported as success:
------------- Standard Error -----------------
Exception in thread "Thread-108" junit.framework.AssertionFailedError:
Elected zero-weight server
at junit.framework.Assert.fail(Assert.java:47)
at
org.apache.zookeeper.test.FLEZeroWeightTest$LEThread.run(FLEZeroWeightTest.java:138)
------------- ---------------- ---------------
this is probably due because the test is calling assert in a thread
other than the main test thread - which junit will not track/knowabout.
One problem I see with these tests (0weight test I looked at) -- it
doesn't have a client attempt to connect to the various servers as part
of declaring success. Really we should only consider "success"ful test
(ie assert that) if a client can connect to each server in the cluster
and change/seechanges. As part of fixing this we really need to do a
sanity check by testing the various command lines and checking that a
client can connect.
I'm not even sure FLEnewepochtest/fletest/etc... are passing either. new
epoch seems to just thrash...
Also I tried 3 & 5 server quorums "by hand from the command line" with 0
weight and they see similar issues to what Todd is seeing.
I'm using the latest code in mainline btw.
Patrick
Mahadev Konar wrote:
> Hi todd,
> I see a lot of
>
> java.net.ConnectException: Connection refused
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
> at java.nio.channels.SocketChannel.open(SocketChannel.java:146)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxMana
> ger.java:324)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.
> java:304)
> at
> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender
> .process(FastLeaderElection.java:317)
> at
> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender
> .run(FastLeaderElection.java:290)
> at java.lang.Thread.run(Thread.java:619)
>
>
> Is it possible that there is some firewall? Can all the servers 1-9 connect
> to all the others using ports that you specified in zoo.cfg i.e 2888/3888?
>
>
> Thanks
> mahadev
>
>
> On 8/4/09 4:56 PM, "Todd Greenwood" <to...@audiencescience.com> wrote:
>
>> Looks like we're not getting *any* leader elected now.... Logs attached.
>>
>>> -----Original Message-----
>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>> Sent: Tuesday, August 04, 2009 4:07 PM
>>> To: zookeeper-dev@hadoop.apache.org
>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>
>>> Patrick, thanks! I'll forward on to IT and I'll report back to you
>>> shortly...
>>>
>>>> -----Original Message-----
>>>> From: Patrick Hunt [mailto:phunt@apache.org]
>>>> Sent: Tuesday, August 04, 2009 3:55 PM
>>>> To: zookeeper-dev@hadoop.apache.org
>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>
>>>> Todd, Mahadev and I looked at this and it turns out to be a
>>> regression.
>>>> Ironically a patch I created for 3.2 branch to add quorum tests
>>> actually
>>>> broke the quorum config -- a default value for a config parameter
>> was
>>>> lost. I'm going to submit a patch asap to get the default back, but
>>> for
>>>> the time being you can set:
>>>>
>>>> electionAlg=3
>>>>
>>>> in each of your config files.
>>>>
>>>> You should see reference to FastLeaderElection in your log files if
>>> this
>>>> parameter is set correctly.
>>>>
>>>> Sorry for the trouble,
>>>>
>>>> Patrick
>>>>
>>>> Todd Greenwood wrote:
>>>>> Mahadev,
>>>>>
>>>>> I just heard from IT that this build behaves in exactly the same
>> way
>>> as
>>>>> previous versions, e.g. we get continuous leader elections that
>>>>> disconnect the followers and then get re-elected, and
>>> disconnect...etc.
>>>>> This is from a fresh sync to the 3.2 branch:
>>>>>
>>>>> svn co
>>>>>
>> http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2
>>>>> ./branch-3.2
>>>>>
>>>>> CHANGES.TXT show the various fixes included:
>>>>>
>>>>>
>> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>> /src/original$ head -n 50 branch-3.2/CHANGES.txt
>>>>> Release 3.2.1
>>>>>
>>>>> Backward compatibile changes:
>>>>>
>>>>> BUGFIXES:
>>>>> ZOOKEEPER-468. avoid compile warning in send_auth_info(). (chris
>>> via
>>>>> flavio)
>>>>>
>>>>> ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten (chris
>>> via
>>>>> mahadev)
>>>>>
>>>>> ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via
>> mahadev)
>>>>> ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris
>> via
>>>>> mahadev)
>>>>>
>>>>> ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure)
>>>>> (giri via mahadev)
>>>>>
>>>>> ZOOKEEPER-467. Change log level in BookieHandle (flavio via
>>> mahadev)
>>>>> ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent
>>> immediate
>>>>> failure. (chris via mahadev)
>>>>>
>>>>> ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev
>>> via
>>>>> phunt)
>>>>>
>>>>> ZOOKEEPER-457. Make ZookeeperMain public, support for HBase (and
>>>>> other)
>>>>> embedded clients (ryan rawson via phunt)
>>>>>
>>>>> ZOOKEEPER-481. Add lastMessageSent to QuorumCnxManager. (flavio
>>> via
>>>>> mahadev)
>>>>>
>>>>> ZOOKEEPER-479. QuorumHierarchical does not count groups
>> correctly
>>>>> (flavio via mahadev)
>>>>>
>>>>> ZOOKEEPER-466. crash on zookeeper_close() when using auth with
>>> empty
>>>>> cert
>>>>> (Chris Darroch via phunt)
>>>>>
>>>>> ZOOKEEPER-480. FLE should perform leader check when node is not
>>>>> leading and
>>>>> add vote of follower (flavio via mahadev)
>>>>>
>>>>> ZOOKEEPER-491. Prevent zero-weight servers from being elected
>>> (flavio
>>>>> via
>>>>> mahadev)
>>>>>
>>>>> What can I do to assist you with this issue?
>>>>>
>>>>> -Todd
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>>> Sent: Tuesday, August 04, 2009 12:43 PM
>>>>>> To: zookeeper-dev@hadoop.apache.org
>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>
>>>>>> Hi todd,
>>>>>> comments in line
>>>>>>
>>>>>>
>>>>>> On 8/4/09 12:38 PM, "Todd Greenwood" <to...@audiencescience.com>
>>>>> wrote:
>>>>>>> Mahadev,
>>>>>>>
>>>>>>> Some quick questions:
>>>>>>>
>>>>>>> 1. Version
>>>>>>>
>>>>>>> I see that the CHANGES.txt calls this 3.2.1, but the build.xml
>> is
>>>>> still
>>>>>>> calling this 3.2.0. Should this be rev'd, and am I correct in
>>>>> calling
>>>>>>> this release 3.2.1?
>>>>>> Yes the release is 3.2.1. The build.xml will be fixed as soon as
>> we
>>>>> tag
>>>>>> the
>>>>>> release.
>>>>>>
>>>>>>> 2. Build targets
>>>>>>>
>>>>>>> The package target fails b/c the create-cppunit-configure target
>>>>> fails
>>>>>>> due to various problems w/ respect to autoconf. Are these
>>>>> dependencies
>>>>>>> documented somewhere ? I'd like to have a fully building system.
>>>>>>>
>>>>>>> create-cppunit-configure:
>>>>>>> [exec] Can't exec "libtoolize": No such file or directory
>> at
>>>>>>> /usr/bin/autoreconf line 188.
>>>>>>> [exec] Use of uninitialized value $libtoolize in pattern
>>> match
>>>>>>> (m//) at /usr/bin/autoreconf line 188.
>>>>>>> [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT'
>> not
>>>>> found
>>>>>>> in library
>>>>>>> [exec] configure.ac:33: error: possibly undefined macro:
>>>>>>> AM_PATH_CPPUNIT
>>>>>>> [exec] If this token and others are legitimate,
>> please
>>>>> use
>>>>>>> m4_pattern_allow.
>>>>>>> [exec] See the Autoconf documentation.
>>>>>>> [exec] configure.ac:53: error: possibly undefined macro:
>>>>>>> AC_PROG_LIBTOOL
>>>>>>> [exec] autoreconf: /usr/bin/autoconf failed with exit
>> status:
>>> 1
>>>>>> You need auto tools to run this. Please read the README for
>>> building c
>>>>>> client library at src/c/ for the installation requirements.
>>>>>>> 3. Sync failure:
>>>>>>>
>>>>>>> This is still failing.
>>>>>>>
>>>>>>> svn: URL
>>>>>>>
>> 'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch'
>>>>>>> doesn't exist
>>>>>>>
>>>>>> Yes this hasn't been fixed yet!
>>>>>>
>>>>>> Thanks
>>>>>> mahadev
>>>>>>> -Todd
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Todd Greenwood
>>>>>>>> Sent: Tuesday, August 04, 2009 11:26 AM
>>>>>>>> To: 'zookeeper-user@hadoop.apache.org'
>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>
>>>>>>>> Great news. Thank you Mahadev. I'll report our findings later
>>>>> today.
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>>>>>> Sent: Tuesday, August 04, 2009 11:20 AM
>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>
>>>>>>>>> Hi Todd,
>>>>>>>>> I just committed 480 and 491. You can checkout the 3.2 branch
>>>>> now.
>>>>>>>>> Thanks
>>>>>>>>> mahadev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 8/3/09 4:29 PM, "Todd Greenwood"
>> <to...@audiencescience.com>
>>>>>>> wrote:
>>>>>>>>>> That'd be perfect. Thanks!
>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>>>>>>>> Sent: Monday, August 03, 2009 4:24 PM
>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>
>>>>>>>>>>> Hi Todd,
>>>>>>>>>>> Most of the patches that you mention should be in the
>> branch
>>>>>>> 3.2 by
>>>>>>>>>> tomm
>>>>>>>>>>> or so. 481, 479 are already in. 480 and 491 should be in by
>>>>> tomm.
>>>>>>>>>> Would
>>>>>>>>>>> that
>>>>>>>>>>> suffice for you?
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> mahadev
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 8/3/09 4:21 PM, "Todd Greenwood"
>>> <to...@audiencescience.com>
>>>>>>>> wrote:
>>>>>>>>>>>> Another problem...I've reverted to the latest versions of
>> the
>>>>>>>>>> patches
>>>>>>>>>>>> that are not specific to branch-3.2, and I'm getting two
>>>>>>> compilation
>>>>>>>>>>>> errors:
>>>>>>>>>>>>
>>>>>>>>>>>> build-generated:
>>>>>>>>>>>> [javac] Compiling 44 source files to
>>>>>>>>>>>>
>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>>>
>>>>>>>>>>>> compile-main:
>>>>>>>>>>>> [javac] Compiling 2 source files to
>>>>>>>>>>>>
>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>>> [javac]
>>>>>>>>>>>>
>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>>> atched/branch-
>>>>>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>>>>>>>> mStats.java:30: name clash: getQuorumPeers() and
>>>>>>> getQuorumPeers()
>>>>>>>>>> have
>>>>>>>>>>>> the same erasure
>>>>>>>>>>>> [javac] public String[] getQuorumPeers();
>>>>>>>>>>>> [javac] ^
>>>>>>>>>>>> [javac]
>>>>>>>>>>>>
>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>>> atched/branch-
>>>>>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>>>>>>>> mStats.java:31: name clash: getServerState() and
>>>>>>> getServerState()
>>>>>>>>>> have
>>>>>>>>>>>> the same erasure
>>>>>>>>>>>> [javac] public String getServerState();
>>>>>>>>>>>> [javac] ^
>>>>>>>>>>>> [javac] 2 errors
>>>>>>>>>>>>
>>>>>>>>>>>> My build process is pretty simple:
>>>>>>>>>>>>
>>>>>>>>>>>> 1. copy the branch-3.2 source to a temp directory
>>>>>>>>>>>> (src/patched/branch-3.2)
>>>>>>>>>>>> 2. apply the ZOOKEEPER patches in my patches directory
>>>>>>>>>>>> 3. build zookeeper in the temp directory
>>>>>>>>>>>>
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>>>>>>>>>>> Sent: Monday, August 03, 2009 4:09 PM
>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>>
>>>>>>>>>>>>> Flavio,
>>>>>>>>>>>>> I notice that you've updated the patches referenced for
>> the
>>>>> WAN
>>>>>>>>>>>>> deployment. There appears to be an order dependency w/
>>> respect
>>>>>>> to
>>>>>>>>>>>> these
>>>>>>>>>>>>> four patches...
>>>>>>>>>>>>>
>>>>>>>>>>>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
>>>>>>>>>>>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
>>>>>>>>>>>>>
>>>>>>>>>>>>> 473 -> 479 (479 fails)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>>>>>>>> /src/patched/branch-3.2$ patch -p0 <
>>>>>>>>>>>>> ../patches/ZOOKEEPER-479-branch3.2.patch
>>>>>>>>>>>>> patching file
>>>>>>>>>>>>>
>> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
>>>>>>>>>>>>> ical.java
>>>>>>>>>>>>> patching file
>>>>>>>>>>>>>
>> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
>>>>>>>>>>>>> patching file
>>>>>>>>>>>>>
>> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
>>>>>>>>>>>>> .java
>>>>>>>>>>>>> patching file
>>>>>>>>>>>>>
>>> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
>>>>>>>>>>>>> Hunk #1 FAILED at 93.
>>>>>>>>>>>>> Hunk #2 FAILED at 145.
>>>>>>>>>>>>> 2 out of 2 hunks FAILED -- saving rejects to file
>>>>>>>>>>>>>
>> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
>> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>>>>>>>> /src/patched/branch-3.2$ h ../patches/
>>>>>>>>>>>>>
>>>>>>>>>>>>> Could you advise as to which patches I need to apply, and
>> in
>>>>>>> what
>>>>>>>>>>>> order?
>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>>> Sent: Friday, July 31, 2009 9:51 PM
>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>>
>>>>>>>>>>>>> Perfect! Thanks for the update, Todd.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks. You were right, I had a stale version of 479.
>>>>>>> Compilation
>>>>>>>>>>>>> succeeds and all tests pass on branch-3.2 with the
>> latest
>>>>>>> patches
>>>>>>>>>>>>> 473,
>>>>>>>>>>>>> 479, 481, and 491.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:48 PM
>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>>
>>>>>>>>>>>>> It should be in 479. Perhaps you have a stale version
>> of
>>>>> the
>>>>>>>>>>>> patch.
>>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Flavio,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm getting a compilation error for patch 491:
>>>>>>>>>>>>>
>>>>>>>>>>>>> compile-main:
>>>>>>>>>>>>> [javac] Compiling 1 source file to
>>>>>>>>>>>>>
>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>>>> src/p
>>>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>>>> [javac]
>>>>>>>>>>>>>
>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>>>> src/p
>>>>>>>>>>>>>
>>> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>>>> FastL
>>>>>>>>>>>>> eaderElection.java:601: cannot find symbol
>>>>>>>>>>>>> [javac] symbol : method getWeight(long)
>>>>>>>>>>>>> [javac] location: interface
>>>>>>>>>>>>>
>>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
>>>>>>>>>>>>> [javac]
>>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>>>> [javac]
>>>>>>> ^
>>>>>>>>>>>>> [javac] 1 error
>>>>>>>>>>>>>
>>>>>>>>>>>>> I see a reference to getWeight in both
>>>>>>> FastLeaderElection.java
>>>>>>>>>>>> in
>>>>>>>>>>>>> patch
>>>>>>>>>>>>> 491:
>>>>>>>>>>>>>
>>>>>>>>>>>>> patches/ZOOKEEPER-491.patch:+
>>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>>>> FastLeaderElection.java
>>>>>>>>>>>>> :
>>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
>>>>>>>>>>>>> 0)
>>>>>>>>>>>>>
>>>>>>>>>>>>> However, I don't see a reference to this method in
>>> patches
>>>>>>> 473,
>>>>>>>>>>>>> 479,
>>>>>>>>>>>>> or
>>>>>>>>>>>>> 481. I also don't see a reference to this method in
>> the
>>>>>>>>>> trunk...
>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>> From: Todd Greenwood
>> [mailto:toddg@audiencescience.com]
>>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ok, I'll apply that patch and report back.
>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>>
>>>>>>>>>>>>> You're missing 491 from your set of patches.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> This repro's in both branch-3.2, and
>>>>>>> branch-3.2+patches(473,
>>>>>>>>>>>>> 479,
>>>>>>>>>>>>> 481).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Basically, it seems like the nodes are electing
>>>>>>> pd4-zook02
>>>>>>>>>> to
>>>>>>>>>>>>> be
>>>>>>>>>>>>> the
>>>>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
>>>>>>>>>>>> supposed
>>>>>>>>>>>>> to
>>>>>>>>>>>>> be
>>>>>>>>>>>>> and
>>>>>>>>>>>>> then disconnects everyone. Then they re-elect it
>> again,
>>>>>>> and
>>>>>>>>>>>> it
>>>>>>>>>>>>> loops
>>>>>>>>>>>>> over and over.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -------------
>>>>>>>>>>>>> Server config
>>>>>>>>>>>>> -------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
>>>>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
>>>>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
>>>>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
>>>>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
>>>>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
>>>>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
>>>>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
>>>>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
>>>>>>>>>>>>>
>>>>>>>>>>>>> group.1:1:2:3:4:5
>>>>>>>>>>>>> weight.1=1
>>>>>>>>>>>>> weight.2=1
>>>>>>>>>>>>> weight.3=1
>>>>>>>>>>>>> weight.4=1
>>>>>>>>>>>>> weight.5=1
>>>>>>>>>>>>>
>>>>>>>>>>>>> group.2:6:7:8:9
>>>>>>>>>>>>> weight.6=0
>>>>>>>>>>>>> weight.7=0
>>>>>>>>>>>>> weight.8=0
>>>>>>>>>>>>> weight.9=0
>>>>>>>>>>>>>
>>>>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
>>>>>>>>>>>> different
>>>>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
>>>>>>>>>> machines
>>>>>>>>>>>>> in
>>>>>>>>>>>>> dc1
>>>>>>>>>>>>> have voting rights, and the ability to become a
>> leader.
>>>>>>> The
>>>>>>>>>>>>> machines
>>>>>>>>>>>>> in
>>>>>>>>>>>>> the pods all have a weight of zero, and are not
>>> expected
>>>>>>> to
>>>>>>>>>>>>> become
>>>>>>>>>>>>> leaders, or to vote on transactions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Let me know what I can do to help resolve this issue.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Todd
>
Re: Unending Leader Elections in WAN deploy
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi todd,
I see a lot of
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
at java.nio.channels.SocketChannel.open(SocketChannel.java:146)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxMana
ger.java:324)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.
java:304)
at
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender
.process(FastLeaderElection.java:317)
at
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender
.run(FastLeaderElection.java:290)
at java.lang.Thread.run(Thread.java:619)
Is it possible that there is some firewall? Can all the servers 1-9 connect
to all the others using ports that you specified in zoo.cfg i.e 2888/3888?
Thanks
mahadev
On 8/4/09 4:56 PM, "Todd Greenwood" <to...@audiencescience.com> wrote:
> Looks like we're not getting *any* leader elected now.... Logs attached.
>
>> -----Original Message-----
>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>> Sent: Tuesday, August 04, 2009 4:07 PM
>> To: zookeeper-dev@hadoop.apache.org
>> Subject: RE: Unending Leader Elections in WAN deploy
>>
>> Patrick, thanks! I'll forward on to IT and I'll report back to you
>> shortly...
>>
>>> -----Original Message-----
>>> From: Patrick Hunt [mailto:phunt@apache.org]
>>> Sent: Tuesday, August 04, 2009 3:55 PM
>>> To: zookeeper-dev@hadoop.apache.org
>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>
>>> Todd, Mahadev and I looked at this and it turns out to be a
>> regression.
>>> Ironically a patch I created for 3.2 branch to add quorum tests
>> actually
>>> broke the quorum config -- a default value for a config parameter
> was
>>> lost. I'm going to submit a patch asap to get the default back, but
>> for
>>> the time being you can set:
>>>
>>> electionAlg=3
>>>
>>> in each of your config files.
>>>
>>> You should see reference to FastLeaderElection in your log files if
>> this
>>> parameter is set correctly.
>>>
>>> Sorry for the trouble,
>>>
>>> Patrick
>>>
>>> Todd Greenwood wrote:
>>>> Mahadev,
>>>>
>>>> I just heard from IT that this build behaves in exactly the same
> way
>> as
>>>> previous versions, e.g. we get continuous leader elections that
>>>> disconnect the followers and then get re-elected, and
>> disconnect...etc.
>>>>
>>>> This is from a fresh sync to the 3.2 branch:
>>>>
>>>> svn co
>>>>
> http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2
>>>> ./branch-3.2
>>>>
>>>> CHANGES.TXT show the various fixes included:
>>>>
>>>>
>>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>> /src/original$ head -n 50 branch-3.2/CHANGES.txt
>>>> Release 3.2.1
>>>>
>>>> Backward compatibile changes:
>>>>
>>>> BUGFIXES:
>>>> ZOOKEEPER-468. avoid compile warning in send_auth_info(). (chris
>> via
>>>> flavio)
>>>>
>>>> ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten (chris
>> via
>>>> mahadev)
>>>>
>>>> ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via
> mahadev)
>>>>
>>>> ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris
> via
>>>> mahadev)
>>>>
>>>> ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure)
>>>> (giri via mahadev)
>>>>
>>>> ZOOKEEPER-467. Change log level in BookieHandle (flavio via
>> mahadev)
>>>>
>>>> ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent
>> immediate
>>>> failure. (chris via mahadev)
>>>>
>>>> ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev
>> via
>>>> phunt)
>>>>
>>>> ZOOKEEPER-457. Make ZookeeperMain public, support for HBase (and
>>>> other)
>>>> embedded clients (ryan rawson via phunt)
>>>>
>>>> ZOOKEEPER-481. Add lastMessageSent to QuorumCnxManager. (flavio
>> via
>>>> mahadev)
>>>>
>>>> ZOOKEEPER-479. QuorumHierarchical does not count groups
> correctly
>>>> (flavio via mahadev)
>>>>
>>>> ZOOKEEPER-466. crash on zookeeper_close() when using auth with
>> empty
>>>> cert
>>>> (Chris Darroch via phunt)
>>>>
>>>> ZOOKEEPER-480. FLE should perform leader check when node is not
>>>> leading and
>>>> add vote of follower (flavio via mahadev)
>>>>
>>>> ZOOKEEPER-491. Prevent zero-weight servers from being elected
>> (flavio
>>>> via
>>>> mahadev)
>>>>
>>>> What can I do to assist you with this issue?
>>>>
>>>> -Todd
>>>>
>>>>> -----Original Message-----
>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>> Sent: Tuesday, August 04, 2009 12:43 PM
>>>>> To: zookeeper-dev@hadoop.apache.org
>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>
>>>>> Hi todd,
>>>>> comments in line
>>>>>
>>>>>
>>>>> On 8/4/09 12:38 PM, "Todd Greenwood" <to...@audiencescience.com>
>>>> wrote:
>>>>>> Mahadev,
>>>>>>
>>>>>> Some quick questions:
>>>>>>
>>>>>> 1. Version
>>>>>>
>>>>>> I see that the CHANGES.txt calls this 3.2.1, but the build.xml
> is
>>>> still
>>>>>> calling this 3.2.0. Should this be rev'd, and am I correct in
>>>> calling
>>>>>> this release 3.2.1?
>>>>> Yes the release is 3.2.1. The build.xml will be fixed as soon as
> we
>>>> tag
>>>>> the
>>>>> release.
>>>>>
>>>>>> 2. Build targets
>>>>>>
>>>>>> The package target fails b/c the create-cppunit-configure target
>>>> fails
>>>>>> due to various problems w/ respect to autoconf. Are these
>>>> dependencies
>>>>>> documented somewhere ? I'd like to have a fully building system.
>>>>>>
>>>>>> create-cppunit-configure:
>>>>>> [exec] Can't exec "libtoolize": No such file or directory
> at
>>>>>> /usr/bin/autoreconf line 188.
>>>>>> [exec] Use of uninitialized value $libtoolize in pattern
>> match
>>>>>> (m//) at /usr/bin/autoreconf line 188.
>>>>>> [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT'
> not
>>>> found
>>>>>> in library
>>>>>> [exec] configure.ac:33: error: possibly undefined macro:
>>>>>> AM_PATH_CPPUNIT
>>>>>> [exec] If this token and others are legitimate,
> please
>>>> use
>>>>>> m4_pattern_allow.
>>>>>> [exec] See the Autoconf documentation.
>>>>>> [exec] configure.ac:53: error: possibly undefined macro:
>>>>>> AC_PROG_LIBTOOL
>>>>>> [exec] autoreconf: /usr/bin/autoconf failed with exit
> status:
>> 1
>>>>>>
>>>>> You need auto tools to run this. Please read the README for
>> building c
>>>>> client library at src/c/ for the installation requirements.
>>>>>> 3. Sync failure:
>>>>>>
>>>>>> This is still failing.
>>>>>>
>>>>>> svn: URL
>>>>>>
> 'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch'
>>>>>> doesn't exist
>>>>>>
>>>>> Yes this hasn't been fixed yet!
>>>>>
>>>>> Thanks
>>>>> mahadev
>>>>>> -Todd
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Todd Greenwood
>>>>>>> Sent: Tuesday, August 04, 2009 11:26 AM
>>>>>>> To: 'zookeeper-user@hadoop.apache.org'
>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>
>>>>>>> Great news. Thank you Mahadev. I'll report our findings later
>>>> today.
>>>>>>> -Todd
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>>>>> Sent: Tuesday, August 04, 2009 11:20 AM
>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>
>>>>>>>> Hi Todd,
>>>>>>>> I just committed 480 and 491. You can checkout the 3.2 branch
>>>> now.
>>>>>>>> Thanks
>>>>>>>> mahadev
>>>>>>>>
>>>>>>>>
>>>>>>>> On 8/3/09 4:29 PM, "Todd Greenwood"
> <to...@audiencescience.com>
>>>>>> wrote:
>>>>>>>>> That'd be perfect. Thanks!
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>>>>>>> Sent: Monday, August 03, 2009 4:24 PM
>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>
>>>>>>>>>> Hi Todd,
>>>>>>>>>> Most of the patches that you mention should be in the
> branch
>>>>>> 3.2 by
>>>>>>>>> tomm
>>>>>>>>>> or so. 481, 479 are already in. 480 and 491 should be in by
>>>> tomm.
>>>>>>>>> Would
>>>>>>>>>> that
>>>>>>>>>> suffice for you?
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> mahadev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 8/3/09 4:21 PM, "Todd Greenwood"
>> <to...@audiencescience.com>
>>>>>>> wrote:
>>>>>>>>>>> Another problem...I've reverted to the latest versions of
> the
>>>>>>>>> patches
>>>>>>>>>>> that are not specific to branch-3.2, and I'm getting two
>>>>>> compilation
>>>>>>>>>>> errors:
>>>>>>>>>>>
>>>>>>>>>>> build-generated:
>>>>>>>>>>> [javac] Compiling 44 source files to
>>>>>>>>>>>
>>>>
>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>>
>>>>>>>>>>> compile-main:
>>>>>>>>>>> [javac] Compiling 2 source files to
>>>>>>>>>>>
>>>>
>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>> [javac]
>>>>>>>>>>>
>>>>
>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>> atched/branch-
>>>>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>>>>>>> mStats.java:30: name clash: getQuorumPeers() and
>>>>>> getQuorumPeers()
>>>>>>>>> have
>>>>>>>>>>> the same erasure
>>>>>>>>>>> [javac] public String[] getQuorumPeers();
>>>>>>>>>>> [javac] ^
>>>>>>>>>>> [javac]
>>>>>>>>>>>
>>>>
>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>> atched/branch-
>>>>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>>>>>>> mStats.java:31: name clash: getServerState() and
>>>>>> getServerState()
>>>>>>>>> have
>>>>>>>>>>> the same erasure
>>>>>>>>>>> [javac] public String getServerState();
>>>>>>>>>>> [javac] ^
>>>>>>>>>>> [javac] 2 errors
>>>>>>>>>>>
>>>>>>>>>>> My build process is pretty simple:
>>>>>>>>>>>
>>>>>>>>>>> 1. copy the branch-3.2 source to a temp directory
>>>>>>>>>>> (src/patched/branch-3.2)
>>>>>>>>>>> 2. apply the ZOOKEEPER patches in my patches directory
>>>>>>>>>>> 3. build zookeeper in the temp directory
>>>>>>>>>>>
>>>>>>>>>>> -Todd
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>>>>>>>>>> Sent: Monday, August 03, 2009 4:09 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> Flavio,
>>>>>>>>>>>> I notice that you've updated the patches referenced for
> the
>>>> WAN
>>>>>>>>>>>> deployment. There appears to be an order dependency w/
>> respect
>>>>>> to
>>>>>>>>>>> these
>>>>>>>>>>>> four patches...
>>>>>>>>>>>>
>>>>>>>>>>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
>>>>>>>>>>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
>>>>>>>>>>>>
>>>>>>>>>>>> 473 -> 479 (479 fails)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>
>>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>>>>>>> /src/patched/branch-3.2$ patch -p0 <
>>>>>>>>>>>> ../patches/ZOOKEEPER-479-branch3.2.patch
>>>>>>>>>>>> patching file
>>>>>>>>>>>>
>>>>
>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
>>>>>>>>>>>> ical.java
>>>>>>>>>>>> patching file
>>>>>>>>>>>>
>>>>
>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
>>>>>>>>>>>> patching file
>>>>>>>>>>>>
>>>>
>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
>>>>>>>>>>>> .java
>>>>>>>>>>>> patching file
>>>>>>>>>>>>
>>>>>>
>> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
>>>>>>>>>>>> Hunk #1 FAILED at 93.
>>>>>>>>>>>> Hunk #2 FAILED at 145.
>>>>>>>>>>>> 2 out of 2 hunks FAILED -- saving rejects to file
>>>>>>>>>>>>
>>>>
>>
> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
>>>>
>>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>>>>>>> /src/patched/branch-3.2$ h ../patches/
>>>>>>>>>>>>
>>>>>>>>>>>> Could you advise as to which patches I need to apply, and
> in
>>>>>> what
>>>>>>>>>>> order?
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>> Sent: Friday, July 31, 2009 9:51 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> Perfect! Thanks for the update, Todd.
>>>>>>>>>>>>
>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>
>>>>>>>>>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks. You were right, I had a stale version of 479.
>>>>>> Compilation
>>>>>>>>>>>> succeeds and all tests pass on branch-3.2 with the
> latest
>>>>>> patches
>>>>>>>>>>>> 473,
>>>>>>>>>>>> 479, 481, and 491.
>>>>>>>>>>>>
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:48 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> It should be in 479. Perhaps you have a stale version
> of
>>>> the
>>>>>>>>>>> patch.
>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>
>>>>>>>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Flavio,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm getting a compilation error for patch 491:
>>>>>>>>>>>>
>>>>>>>>>>>> compile-main:
>>>>>>>>>>>> [javac] Compiling 1 source file to
>>>>>>>>>>>>
>>>>>>
>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>>> src/p
>>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>>> [javac]
>>>>>>>>>>>>
>>>>>>
>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>>> src/p
>>>>>>>>>>>>
>>>>>>
>> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>>> FastL
>>>>>>>>>>>> eaderElection.java:601: cannot find symbol
>>>>>>>>>>>> [javac] symbol : method getWeight(long)
>>>>>>>>>>>> [javac] location: interface
>>>>>>>>>>>>
>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
>>>>>>>>>>>> [javac]
>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>>> [javac]
>>>>>> ^
>>>>>>>>>>>> [javac] 1 error
>>>>>>>>>>>>
>>>>>>>>>>>> I see a reference to getWeight in both
>>>>>> FastLeaderElection.java
>>>>>>>>>>> in
>>>>>>>>>>>> patch
>>>>>>>>>>>> 491:
>>>>>>>>>>>>
>>>>>>>>>>>> patches/ZOOKEEPER-491.patch:+
>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>>> FastLeaderElection.java
>>>>>>>>>>>> :
>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
>>>>>>>>>>>> 0)
>>>>>>>>>>>>
>>>>>>>>>>>> However, I don't see a reference to this method in
>> patches
>>>>>> 473,
>>>>>>>>>>>> 479,
>>>>>>>>>>>> or
>>>>>>>>>>>> 481. I also don't see a reference to this method in
> the
>>>>>>>>> trunk...
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Todd Greenwood
> [mailto:toddg@audiencescience.com]
>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> Ok, I'll apply that patch and report back.
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> You're missing 491 from your set of patches.
>>>>>>>>>>>>
>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>
>>>>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> This repro's in both branch-3.2, and
>>>>>> branch-3.2+patches(473,
>>>>>>>>>>>> 479,
>>>>>>>>>>>> 481).
>>>>>>>>>>>>
>>>>>>>>>>>> Basically, it seems like the nodes are electing
>>>>>> pd4-zook02
>>>>>>>>> to
>>>>>>>>>>>> be
>>>>>>>>>>>> the
>>>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
>>>>>>>>>>> supposed
>>>>>>>>>>>> to
>>>>>>>>>>>> be
>>>>>>>>>>>> and
>>>>>>>>>>>> then disconnects everyone. Then they re-elect it
> again,
>>>>>> and
>>>>>>>>>>> it
>>>>>>>>>>>> loops
>>>>>>>>>>>> over and over.
>>>>>>>>>>>>
>>>>>>>>>>>> -------------
>>>>>>>>>>>> Server config
>>>>>>>>>>>> -------------
>>>>>>>>>>>>
>>>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
>>>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
>>>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
>>>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
>>>>>>>>>>>>
>>>>>>>>>>>> group.1:1:2:3:4:5
>>>>>>>>>>>> weight.1=1
>>>>>>>>>>>> weight.2=1
>>>>>>>>>>>> weight.3=1
>>>>>>>>>>>> weight.4=1
>>>>>>>>>>>> weight.5=1
>>>>>>>>>>>>
>>>>>>>>>>>> group.2:6:7:8:9
>>>>>>>>>>>> weight.6=0
>>>>>>>>>>>> weight.7=0
>>>>>>>>>>>> weight.8=0
>>>>>>>>>>>> weight.9=0
>>>>>>>>>>>>
>>>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
>>>>>>>>>>> different
>>>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
>>>>>>>>> machines
>>>>>>>>>>>> in
>>>>>>>>>>>> dc1
>>>>>>>>>>>> have voting rights, and the ability to become a
> leader.
>>>>>> The
>>>>>>>>>>>> machines
>>>>>>>>>>>> in
>>>>>>>>>>>> the pods all have a weight of zero, and are not
>> expected
>>>>>> to
>>>>>>>>>>>> become
>>>>>>>>>>>> leaders, or to vote on transactions.
>>>>>>>>>>>>
>>>>>>>>>>>> Let me know what I can do to help resolve this issue.
>>>>>>>>>>>>
>>>>>>>>>>>> -Todd
>>>>
RE: Unending Leader Elections in WAN deploy
Posted by Todd Greenwood <to...@audiencescience.com>.
Looks like we're not getting *any* leader elected now.... Logs attached.
> -----Original Message-----
> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> Sent: Tuesday, August 04, 2009 4:07 PM
> To: zookeeper-dev@hadoop.apache.org
> Subject: RE: Unending Leader Elections in WAN deploy
>
> Patrick, thanks! I'll forward on to IT and I'll report back to you
> shortly...
>
> > -----Original Message-----
> > From: Patrick Hunt [mailto:phunt@apache.org]
> > Sent: Tuesday, August 04, 2009 3:55 PM
> > To: zookeeper-dev@hadoop.apache.org
> > Subject: Re: Unending Leader Elections in WAN deploy
> >
> > Todd, Mahadev and I looked at this and it turns out to be a
> regression.
> > Ironically a patch I created for 3.2 branch to add quorum tests
> actually
> > broke the quorum config -- a default value for a config parameter
was
> > lost. I'm going to submit a patch asap to get the default back, but
> for
> > the time being you can set:
> >
> > electionAlg=3
> >
> > in each of your config files.
> >
> > You should see reference to FastLeaderElection in your log files if
> this
> > parameter is set correctly.
> >
> > Sorry for the trouble,
> >
> > Patrick
> >
> > Todd Greenwood wrote:
> > > Mahadev,
> > >
> > > I just heard from IT that this build behaves in exactly the same
way
> as
> > > previous versions, e.g. we get continuous leader elections that
> > > disconnect the followers and then get re-elected, and
> disconnect...etc.
> > >
> > > This is from a fresh sync to the 3.2 branch:
> > >
> > > svn co
> > >
http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2
> > > ./branch-3.2
> > >
> > > CHANGES.TXT show the various fixes included:
> > >
> > >
>
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> > > /src/original$ head -n 50 branch-3.2/CHANGES.txt
> > > Release 3.2.1
> > >
> > > Backward compatibile changes:
> > >
> > > BUGFIXES:
> > > ZOOKEEPER-468. avoid compile warning in send_auth_info(). (chris
> via
> > > flavio)
> > >
> > > ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten (chris
> via
> > > mahadev)
> > >
> > > ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via
mahadev)
> > >
> > > ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris
via
> > > mahadev)
> > >
> > > ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure)
> > > (giri via mahadev)
> > >
> > > ZOOKEEPER-467. Change log level in BookieHandle (flavio via
> mahadev)
> > >
> > > ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent
> immediate
> > > failure. (chris via mahadev)
> > >
> > > ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev
> via
> > > phunt)
> > >
> > > ZOOKEEPER-457. Make ZookeeperMain public, support for HBase (and
> > > other)
> > > embedded clients (ryan rawson via phunt)
> > >
> > > ZOOKEEPER-481. Add lastMessageSent to QuorumCnxManager. (flavio
> via
> > > mahadev)
> > >
> > > ZOOKEEPER-479. QuorumHierarchical does not count groups
correctly
> > > (flavio via mahadev)
> > >
> > > ZOOKEEPER-466. crash on zookeeper_close() when using auth with
> empty
> > > cert
> > > (Chris Darroch via phunt)
> > >
> > > ZOOKEEPER-480. FLE should perform leader check when node is not
> > > leading and
> > > add vote of follower (flavio via mahadev)
> > >
> > > ZOOKEEPER-491. Prevent zero-weight servers from being elected
> (flavio
> > > via
> > > mahadev)
> > >
> > > What can I do to assist you with this issue?
> > >
> > > -Todd
> > >
> > >> -----Original Message-----
> > >> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> > >> Sent: Tuesday, August 04, 2009 12:43 PM
> > >> To: zookeeper-dev@hadoop.apache.org
> > >> Subject: Re: Unending Leader Elections in WAN deploy
> > >>
> > >> Hi todd,
> > >> comments in line
> > >>
> > >>
> > >> On 8/4/09 12:38 PM, "Todd Greenwood" <to...@audiencescience.com>
> > > wrote:
> > >>> Mahadev,
> > >>>
> > >>> Some quick questions:
> > >>>
> > >>> 1. Version
> > >>>
> > >>> I see that the CHANGES.txt calls this 3.2.1, but the build.xml
is
> > > still
> > >>> calling this 3.2.0. Should this be rev'd, and am I correct in
> > > calling
> > >>> this release 3.2.1?
> > >> Yes the release is 3.2.1. The build.xml will be fixed as soon as
we
> > > tag
> > >> the
> > >> release.
> > >>
> > >>> 2. Build targets
> > >>>
> > >>> The package target fails b/c the create-cppunit-configure target
> > > fails
> > >>> due to various problems w/ respect to autoconf. Are these
> > > dependencies
> > >>> documented somewhere ? I'd like to have a fully building system.
> > >>>
> > >>> create-cppunit-configure:
> > >>> [exec] Can't exec "libtoolize": No such file or directory
at
> > >>> /usr/bin/autoreconf line 188.
> > >>> [exec] Use of uninitialized value $libtoolize in pattern
> match
> > >>> (m//) at /usr/bin/autoreconf line 188.
> > >>> [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT'
not
> > > found
> > >>> in library
> > >>> [exec] configure.ac:33: error: possibly undefined macro:
> > >>> AM_PATH_CPPUNIT
> > >>> [exec] If this token and others are legitimate,
please
> > > use
> > >>> m4_pattern_allow.
> > >>> [exec] See the Autoconf documentation.
> > >>> [exec] configure.ac:53: error: possibly undefined macro:
> > >>> AC_PROG_LIBTOOL
> > >>> [exec] autoreconf: /usr/bin/autoconf failed with exit
status:
> 1
> > >>>
> > >> You need auto tools to run this. Please read the README for
> building c
> > >> client library at src/c/ for the installation requirements.
> > >>> 3. Sync failure:
> > >>>
> > >>> This is still failing.
> > >>>
> > >>> svn: URL
> > >>>
'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch'
> > >>> doesn't exist
> > >>>
> > >> Yes this hasn't been fixed yet!
> > >>
> > >> Thanks
> > >> mahadev
> > >>> -Todd
> > >>>
> > >>>> -----Original Message-----
> > >>>> From: Todd Greenwood
> > >>>> Sent: Tuesday, August 04, 2009 11:26 AM
> > >>>> To: 'zookeeper-user@hadoop.apache.org'
> > >>>> Subject: RE: Unending Leader Elections in WAN deploy
> > >>>>
> > >>>> Great news. Thank you Mahadev. I'll report our findings later
> > > today.
> > >>>> -Todd
> > >>>>
> > >>>>> -----Original Message-----
> > >>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> > >>>>> Sent: Tuesday, August 04, 2009 11:20 AM
> > >>>>> To: zookeeper-user@hadoop.apache.org
> > >>>>> Subject: Re: Unending Leader Elections in WAN deploy
> > >>>>>
> > >>>>> Hi Todd,
> > >>>>> I just committed 480 and 491. You can checkout the 3.2 branch
> > > now.
> > >>>>> Thanks
> > >>>>> mahadev
> > >>>>>
> > >>>>>
> > >>>>> On 8/3/09 4:29 PM, "Todd Greenwood"
<to...@audiencescience.com>
> > >>> wrote:
> > >>>>>> That'd be perfect. Thanks!
> > >>>>>>
> > >>>>>>> -----Original Message-----
> > >>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> > >>>>>>> Sent: Monday, August 03, 2009 4:24 PM
> > >>>>>>> To: zookeeper-user@hadoop.apache.org
> > >>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> > >>>>>>>
> > >>>>>>> Hi Todd,
> > >>>>>>> Most of the patches that you mention should be in the
branch
> > >>> 3.2 by
> > >>>>>> tomm
> > >>>>>>> or so. 481, 479 are already in. 480 and 491 should be in by
> > > tomm.
> > >>>>>> Would
> > >>>>>>> that
> > >>>>>>> suffice for you?
> > >>>>>>>
> > >>>>>>> Thanks
> > >>>>>>> mahadev
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On 8/3/09 4:21 PM, "Todd Greenwood"
> <to...@audiencescience.com>
> > >>>> wrote:
> > >>>>>>>> Another problem...I've reverted to the latest versions of
the
> > >>>>>> patches
> > >>>>>>>> that are not specific to branch-3.2, and I'm getting two
> > >>> compilation
> > >>>>>>>> errors:
> > >>>>>>>>
> > >>>>>>>> build-generated:
> > >>>>>>>> [javac] Compiling 44 source files to
> > >>>>>>>>
> > >
>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> > >>>>>>>> atched/branch-3.2/build/classes
> > >>>>>>>>
> > >>>>>>>> compile-main:
> > >>>>>>>> [javac] Compiling 2 source files to
> > >>>>>>>>
> > >
>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> > >>>>>>>> atched/branch-3.2/build/classes
> > >>>>>>>> [javac]
> > >>>>>>>>
> > >
>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> > >>>>>> atched/branch-
> > >>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> > >>>>>>>> mStats.java:30: name clash: getQuorumPeers() and
> > >>> getQuorumPeers()
> > >>>>>> have
> > >>>>>>>> the same erasure
> > >>>>>>>> [javac] public String[] getQuorumPeers();
> > >>>>>>>> [javac] ^
> > >>>>>>>> [javac]
> > >>>>>>>>
> > >
>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> > >>>>>> atched/branch-
> > >>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> > >>>>>>>> mStats.java:31: name clash: getServerState() and
> > >>> getServerState()
> > >>>>>> have
> > >>>>>>>> the same erasure
> > >>>>>>>> [javac] public String getServerState();
> > >>>>>>>> [javac] ^
> > >>>>>>>> [javac] 2 errors
> > >>>>>>>>
> > >>>>>>>> My build process is pretty simple:
> > >>>>>>>>
> > >>>>>>>> 1. copy the branch-3.2 source to a temp directory
> > >>>>>>>> (src/patched/branch-3.2)
> > >>>>>>>> 2. apply the ZOOKEEPER patches in my patches directory
> > >>>>>>>> 3. build zookeeper in the temp directory
> > >>>>>>>>
> > >>>>>>>> -Todd
> > >>>>>>>>> -----Original Message-----
> > >>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> > >>>>>>>>> Sent: Monday, August 03, 2009 4:09 PM
> > >>>>>>>>> To: zookeeper-user@hadoop.apache.org
> > >>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
> > >>>>>>>>>
> > >>>>>>>>> Flavio,
> > >>>>>>>>> I notice that you've updated the patches referenced for
the
> > > WAN
> > >>>>>>>>> deployment. There appears to be an order dependency w/
> respect
> > >>> to
> > >>>>>>>> these
> > >>>>>>>>> four patches...
> > >>>>>>>>>
> > >>>>>>>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
> > >>>>>>>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
> > >>>>>>>>>
> > >>>>>>>>> 473 -> 479 (479 fails)
> > >>>>>>>>>
> > >>>>>>>>>
> > >
>
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> > >>>>>>>>> /src/patched/branch-3.2$ patch -p0 <
> > >>>>>>>>> ../patches/ZOOKEEPER-479-branch3.2.patch
> > >>>>>>>>> patching file
> > >>>>>>>>>
> > >
>
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
> > >>>>>>>>> ical.java
> > >>>>>>>>> patching file
> > >>>>>>>>>
> > >
>
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
> > >>>>>>>>> patching file
> > >>>>>>>>>
> > >
>
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
> > >>>>>>>>> .java
> > >>>>>>>>> patching file
> > >>>>>>>>>
> > >>>
> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
> > >>>>>>>>> Hunk #1 FAILED at 93.
> > >>>>>>>>> Hunk #2 FAILED at 145.
> > >>>>>>>>> 2 out of 2 hunks FAILED -- saving rejects to file
> > >>>>>>>>>
> > >
>
src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
> > >
>
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> > >>>>>>>>> /src/patched/branch-3.2$ h ../patches/
> > >>>>>>>>>
> > >>>>>>>>> Could you advise as to which patches I need to apply, and
in
> > >>> what
> > >>>>>>>> order?
> > >>>>>>>>> -Todd
> > >>>>>>>>>
> > >>>>>>>>>> -----Original Message-----
> > >>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> > >>>>>>>>>> Sent: Friday, July 31, 2009 9:51 PM
> > >>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> > >>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> > >>>>>>>>>>
> > >>>>>>>>>> Perfect! Thanks for the update, Todd.
> > >>>>>>>>>>
> > >>>>>>>>>> -Flavio
> > >>>>>>>>>>
> > >>>>>>>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Thanks. You were right, I had a stale version of 479.
> > >>> Compilation
> > >>>>>>>>>>> succeeds and all tests pass on branch-3.2 with the
latest
> > >>> patches
> > >>>>>>>>> 473,
> > >>>>>>>>>>> 479, 481, and 491.
> > >>>>>>>>>>>
> > >>>>>>>>>>> -Todd
> > >>>>>>>>>>>
> > >>>>>>>>>>>> -----Original Message-----
> > >>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> > >>>>>>>>>>>> Sent: Friday, July 31, 2009 7:48 PM
> > >>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> > >>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> It should be in 479. Perhaps you have a stale version
of
> > > the
> > >>>>>>>> patch.
> > >>>>>>>>>>>> -Flavio
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Flavio,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I'm getting a compilation error for patch 491:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> compile-main:
> > >>>>>>>>>>>>> [javac] Compiling 1 source file to
> > >>>>>>>>>>>>>
> > >>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> > >>>>>>>>>>>>> src/p
> > >>>>>>>>>>>>> atched/branch-3.2/build/classes
> > >>>>>>>>>>>>> [javac]
> > >>>>>>>>>>>>>
> > >>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> > >>>>>>>>>>>>> src/p
> > >>>>>>>>>>>>>
> > >>>
> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
> > >>>>>>>>>>>>> FastL
> > >>>>>>>>>>>>> eaderElection.java:601: cannot find symbol
> > >>>>>>>>>>>>> [javac] symbol : method getWeight(long)
> > >>>>>>>>>>>>> [javac] location: interface
> > >>>>>>>>>>>>>
> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
> > >>>>>>>>>>>>> [javac]
> > >>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> > >>>>>>>>>>>>> [javac]
> > >>> ^
> > >>>>>>>>>>>>> [javac] 1 error
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I see a reference to getWeight in both
> > >>> FastLeaderElection.java
> > >>>>>>>> in
> > >>>>>>>>>>>>> patch
> > >>>>>>>>>>>>> 491:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> patches/ZOOKEEPER-491.patch:+
> > >>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> > >>>>>>>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
> > >>>>>>>>>>>>> FastLeaderElection.java
> > >>>>>>>>>>>>> :
> > >>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
> > >>>>>>>>>>>>> 0)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> However, I don't see a reference to this method in
> patches
> > >>> 473,
> > >>>>>>>>> 479,
> > >>>>>>>>>>>>> or
> > >>>>>>>>>>>>> 481. I also don't see a reference to this method in
the
> > >>>>>> trunk...
> > >>>>>>>>>>>>> -Todd
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> -----Original Message-----
> > >>>>>>>>>>>>>> From: Todd Greenwood
[mailto:toddg@audiencescience.com]
> > >>>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
> > >>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> > >>>>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Ok, I'll apply that patch and report back.
> > >>>>>>>>>>>>>> -Todd
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> -----Original Message-----
> > >>>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> > >>>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
> > >>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> > >>>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> You're missing 491 from your set of patches.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> -Flavio
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> This repro's in both branch-3.2, and
> > >>> branch-3.2+patches(473,
> > >>>>>>>>> 479,
> > >>>>>>>>>>>>>> 481).
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Basically, it seems like the nodes are electing
> > >>> pd4-zook02
> > >>>>>> to
> > >>>>>>>>> be
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
> > >>>>>>>> supposed
> > >>>>>>>>> to
> > >>>>>>>>>>>>> be
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>> then disconnects everyone. Then they re-elect it
again,
> > >>> and
> > >>>>>>>> it
> > >>>>>>>>>>>>> loops
> > >>>>>>>>>>>>>> over and over.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> -------------
> > >>>>>>>>>>>>>> Server config
> > >>>>>>>>>>>>>> -------------
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
> > >>>>>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
> > >>>>>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
> > >>>>>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
> > >>>>>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
> > >>>>>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
> > >>>>>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
> > >>>>>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
> > >>>>>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> group.1:1:2:3:4:5
> > >>>>>>>>>>>>>> weight.1=1
> > >>>>>>>>>>>>>> weight.2=1
> > >>>>>>>>>>>>>> weight.3=1
> > >>>>>>>>>>>>>> weight.4=1
> > >>>>>>>>>>>>>> weight.5=1
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> group.2:6:7:8:9
> > >>>>>>>>>>>>>> weight.6=0
> > >>>>>>>>>>>>>> weight.7=0
> > >>>>>>>>>>>>>> weight.8=0
> > >>>>>>>>>>>>>> weight.9=0
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
> > >>>>>>>> different
> > >>>>>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
> > >>>>>> machines
> > >>>>>>>>> in
> > >>>>>>>>>>>>> dc1
> > >>>>>>>>>>>>>> have voting rights, and the ability to become a
leader.
> > >>> The
> > >>>>>>>>>>>>> machines
> > >>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>> the pods all have a weight of zero, and are not
> expected
> > >>> to
> > >>>>>>>>>>> become
> > >>>>>>>>>>>>>> leaders, or to vote on transactions.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Let me know what I can do to help resolve this issue.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> -Todd
> > >
RE: Unending Leader Elections in WAN deploy
Posted by Todd Greenwood <to...@audiencescience.com>.
Patrick, thanks! I'll forward on to IT and I'll report back to you
shortly...
> -----Original Message-----
> From: Patrick Hunt [mailto:phunt@apache.org]
> Sent: Tuesday, August 04, 2009 3:55 PM
> To: zookeeper-dev@hadoop.apache.org
> Subject: Re: Unending Leader Elections in WAN deploy
>
> Todd, Mahadev and I looked at this and it turns out to be a
regression.
> Ironically a patch I created for 3.2 branch to add quorum tests
actually
> broke the quorum config -- a default value for a config parameter was
> lost. I'm going to submit a patch asap to get the default back, but
for
> the time being you can set:
>
> electionAlg=3
>
> in each of your config files.
>
> You should see reference to FastLeaderElection in your log files if
this
> parameter is set correctly.
>
> Sorry for the trouble,
>
> Patrick
>
> Todd Greenwood wrote:
> > Mahadev,
> >
> > I just heard from IT that this build behaves in exactly the same way
as
> > previous versions, e.g. we get continuous leader elections that
> > disconnect the followers and then get re-elected, and
disconnect...etc.
> >
> > This is from a fresh sync to the 3.2 branch:
> >
> > svn co
> > http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2
> > ./branch-3.2
> >
> > CHANGES.TXT show the various fixes included:
> >
> >
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> > /src/original$ head -n 50 branch-3.2/CHANGES.txt
> > Release 3.2.1
> >
> > Backward compatibile changes:
> >
> > BUGFIXES:
> > ZOOKEEPER-468. avoid compile warning in send_auth_info(). (chris
via
> > flavio)
> >
> > ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten (chris
via
> > mahadev)
> >
> > ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via mahadev)
> >
> > ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris via
> > mahadev)
> >
> > ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure)
> > (giri via mahadev)
> >
> > ZOOKEEPER-467. Change log level in BookieHandle (flavio via
mahadev)
> >
> > ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent
immediate
> > failure. (chris via mahadev)
> >
> > ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev
via
> > phunt)
> >
> > ZOOKEEPER-457. Make ZookeeperMain public, support for HBase (and
> > other)
> > embedded clients (ryan rawson via phunt)
> >
> > ZOOKEEPER-481. Add lastMessageSent to QuorumCnxManager. (flavio
via
> > mahadev)
> >
> > ZOOKEEPER-479. QuorumHierarchical does not count groups correctly
> > (flavio via mahadev)
> >
> > ZOOKEEPER-466. crash on zookeeper_close() when using auth with
empty
> > cert
> > (Chris Darroch via phunt)
> >
> > ZOOKEEPER-480. FLE should perform leader check when node is not
> > leading and
> > add vote of follower (flavio via mahadev)
> >
> > ZOOKEEPER-491. Prevent zero-weight servers from being elected
(flavio
> > via
> > mahadev)
> >
> > What can I do to assist you with this issue?
> >
> > -Todd
> >
> >> -----Original Message-----
> >> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> >> Sent: Tuesday, August 04, 2009 12:43 PM
> >> To: zookeeper-dev@hadoop.apache.org
> >> Subject: Re: Unending Leader Elections in WAN deploy
> >>
> >> Hi todd,
> >> comments in line
> >>
> >>
> >> On 8/4/09 12:38 PM, "Todd Greenwood" <to...@audiencescience.com>
> > wrote:
> >>> Mahadev,
> >>>
> >>> Some quick questions:
> >>>
> >>> 1. Version
> >>>
> >>> I see that the CHANGES.txt calls this 3.2.1, but the build.xml is
> > still
> >>> calling this 3.2.0. Should this be rev'd, and am I correct in
> > calling
> >>> this release 3.2.1?
> >> Yes the release is 3.2.1. The build.xml will be fixed as soon as we
> > tag
> >> the
> >> release.
> >>
> >>> 2. Build targets
> >>>
> >>> The package target fails b/c the create-cppunit-configure target
> > fails
> >>> due to various problems w/ respect to autoconf. Are these
> > dependencies
> >>> documented somewhere ? I'd like to have a fully building system.
> >>>
> >>> create-cppunit-configure:
> >>> [exec] Can't exec "libtoolize": No such file or directory at
> >>> /usr/bin/autoreconf line 188.
> >>> [exec] Use of uninitialized value $libtoolize in pattern
match
> >>> (m//) at /usr/bin/autoreconf line 188.
> >>> [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT' not
> > found
> >>> in library
> >>> [exec] configure.ac:33: error: possibly undefined macro:
> >>> AM_PATH_CPPUNIT
> >>> [exec] If this token and others are legitimate, please
> > use
> >>> m4_pattern_allow.
> >>> [exec] See the Autoconf documentation.
> >>> [exec] configure.ac:53: error: possibly undefined macro:
> >>> AC_PROG_LIBTOOL
> >>> [exec] autoreconf: /usr/bin/autoconf failed with exit status:
1
> >>>
> >> You need auto tools to run this. Please read the README for
building c
> >> client library at src/c/ for the installation requirements.
> >>> 3. Sync failure:
> >>>
> >>> This is still failing.
> >>>
> >>> svn: URL
> >>> 'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch'
> >>> doesn't exist
> >>>
> >> Yes this hasn't been fixed yet!
> >>
> >> Thanks
> >> mahadev
> >>> -Todd
> >>>
> >>>> -----Original Message-----
> >>>> From: Todd Greenwood
> >>>> Sent: Tuesday, August 04, 2009 11:26 AM
> >>>> To: 'zookeeper-user@hadoop.apache.org'
> >>>> Subject: RE: Unending Leader Elections in WAN deploy
> >>>>
> >>>> Great news. Thank you Mahadev. I'll report our findings later
> > today.
> >>>> -Todd
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> >>>>> Sent: Tuesday, August 04, 2009 11:20 AM
> >>>>> To: zookeeper-user@hadoop.apache.org
> >>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>
> >>>>> Hi Todd,
> >>>>> I just committed 480 and 491. You can checkout the 3.2 branch
> > now.
> >>>>> Thanks
> >>>>> mahadev
> >>>>>
> >>>>>
> >>>>> On 8/3/09 4:29 PM, "Todd Greenwood" <to...@audiencescience.com>
> >>> wrote:
> >>>>>> That'd be perfect. Thanks!
> >>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> >>>>>>> Sent: Monday, August 03, 2009 4:24 PM
> >>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>
> >>>>>>> Hi Todd,
> >>>>>>> Most of the patches that you mention should be in the branch
> >>> 3.2 by
> >>>>>> tomm
> >>>>>>> or so. 481, 479 are already in. 480 and 491 should be in by
> > tomm.
> >>>>>> Would
> >>>>>>> that
> >>>>>>> suffice for you?
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>> mahadev
> >>>>>>>
> >>>>>>>
> >>>>>>> On 8/3/09 4:21 PM, "Todd Greenwood"
<to...@audiencescience.com>
> >>>> wrote:
> >>>>>>>> Another problem...I've reverted to the latest versions of the
> >>>>>> patches
> >>>>>>>> that are not specific to branch-3.2, and I'm getting two
> >>> compilation
> >>>>>>>> errors:
> >>>>>>>>
> >>>>>>>> build-generated:
> >>>>>>>> [javac] Compiling 44 source files to
> >>>>>>>>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>>>> atched/branch-3.2/build/classes
> >>>>>>>>
> >>>>>>>> compile-main:
> >>>>>>>> [javac] Compiling 2 source files to
> >>>>>>>>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>>>> atched/branch-3.2/build/classes
> >>>>>>>> [javac]
> >>>>>>>>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>> atched/branch-
> >>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> >>>>>>>> mStats.java:30: name clash: getQuorumPeers() and
> >>> getQuorumPeers()
> >>>>>> have
> >>>>>>>> the same erasure
> >>>>>>>> [javac] public String[] getQuorumPeers();
> >>>>>>>> [javac] ^
> >>>>>>>> [javac]
> >>>>>>>>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>> atched/branch-
> >>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> >>>>>>>> mStats.java:31: name clash: getServerState() and
> >>> getServerState()
> >>>>>> have
> >>>>>>>> the same erasure
> >>>>>>>> [javac] public String getServerState();
> >>>>>>>> [javac] ^
> >>>>>>>> [javac] 2 errors
> >>>>>>>>
> >>>>>>>> My build process is pretty simple:
> >>>>>>>>
> >>>>>>>> 1. copy the branch-3.2 source to a temp directory
> >>>>>>>> (src/patched/branch-3.2)
> >>>>>>>> 2. apply the ZOOKEEPER patches in my patches directory
> >>>>>>>> 3. build zookeeper in the temp directory
> >>>>>>>>
> >>>>>>>> -Todd
> >>>>>>>>> -----Original Message-----
> >>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> >>>>>>>>> Sent: Monday, August 03, 2009 4:09 PM
> >>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
> >>>>>>>>>
> >>>>>>>>> Flavio,
> >>>>>>>>> I notice that you've updated the patches referenced for the
> > WAN
> >>>>>>>>> deployment. There appears to be an order dependency w/
respect
> >>> to
> >>>>>>>> these
> >>>>>>>>> four patches...
> >>>>>>>>>
> >>>>>>>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
> >>>>>>>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
> >>>>>>>>>
> >>>>>>>>> 473 -> 479 (479 fails)
> >>>>>>>>>
> >>>>>>>>>
> >
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> >>>>>>>>> /src/patched/branch-3.2$ patch -p0 <
> >>>>>>>>> ../patches/ZOOKEEPER-479-branch3.2.patch
> >>>>>>>>> patching file
> >>>>>>>>>
> >
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
> >>>>>>>>> ical.java
> >>>>>>>>> patching file
> >>>>>>>>>
> >
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
> >>>>>>>>> patching file
> >>>>>>>>>
> >
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
> >>>>>>>>> .java
> >>>>>>>>> patching file
> >>>>>>>>>
> >>>
src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
> >>>>>>>>> Hunk #1 FAILED at 93.
> >>>>>>>>> Hunk #2 FAILED at 145.
> >>>>>>>>> 2 out of 2 hunks FAILED -- saving rejects to file
> >>>>>>>>>
> >
src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
> >
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> >>>>>>>>> /src/patched/branch-3.2$ h ../patches/
> >>>>>>>>>
> >>>>>>>>> Could you advise as to which patches I need to apply, and in
> >>> what
> >>>>>>>> order?
> >>>>>>>>> -Todd
> >>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>>>>>>> Sent: Friday, July 31, 2009 9:51 PM
> >>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>>>
> >>>>>>>>>> Perfect! Thanks for the update, Todd.
> >>>>>>>>>>
> >>>>>>>>>> -Flavio
> >>>>>>>>>>
> >>>>>>>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Thanks. You were right, I had a stale version of 479.
> >>> Compilation
> >>>>>>>>>>> succeeds and all tests pass on branch-3.2 with the latest
> >>> patches
> >>>>>>>>> 473,
> >>>>>>>>>>> 479, 481, and 491.
> >>>>>>>>>>>
> >>>>>>>>>>> -Todd
> >>>>>>>>>>>
> >>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>>>>>>>>> Sent: Friday, July 31, 2009 7:48 PM
> >>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>>>>>
> >>>>>>>>>>>> It should be in 479. Perhaps you have a stale version of
> > the
> >>>>>>>> patch.
> >>>>>>>>>>>> -Flavio
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Flavio,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I'm getting a compilation error for patch 491:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> compile-main:
> >>>>>>>>>>>>> [javac] Compiling 1 source file to
> >>>>>>>>>>>>>
> >>>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> >>>>>>>>>>>>> src/p
> >>>>>>>>>>>>> atched/branch-3.2/build/classes
> >>>>>>>>>>>>> [javac]
> >>>>>>>>>>>>>
> >>>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> >>>>>>>>>>>>> src/p
> >>>>>>>>>>>>>
> >>>
atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
> >>>>>>>>>>>>> FastL
> >>>>>>>>>>>>> eaderElection.java:601: cannot find symbol
> >>>>>>>>>>>>> [javac] symbol : method getWeight(long)
> >>>>>>>>>>>>> [javac] location: interface
> >>>>>>>>>>>>>
org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
> >>>>>>>>>>>>> [javac]
> >>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> >>>>>>>>>>>>> [javac]
> >>> ^
> >>>>>>>>>>>>> [javac] 1 error
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I see a reference to getWeight in both
> >>> FastLeaderElection.java
> >>>>>>>> in
> >>>>>>>>>>>>> patch
> >>>>>>>>>>>>> 491:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> patches/ZOOKEEPER-491.patch:+
> >>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> >>>>>>>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
> >>>>>>>>>>>>> FastLeaderElection.java
> >>>>>>>>>>>>> :
> >>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
> >>>>>>>>>>>>> 0)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> However, I don't see a reference to this method in
patches
> >>> 473,
> >>>>>>>>> 479,
> >>>>>>>>>>>>> or
> >>>>>>>>>>>>> 481. I also don't see a reference to this method in the
> >>>>>> trunk...
> >>>>>>>>>>>>> -Todd
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> >>>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
> >>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ok, I'll apply that patch and report back.
> >>>>>>>>>>>>>> -Todd
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
> >>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> You're missing 491 from your set of patches.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -Flavio
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> This repro's in both branch-3.2, and
> >>> branch-3.2+patches(473,
> >>>>>>>>> 479,
> >>>>>>>>>>>>>> 481).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Basically, it seems like the nodes are electing
> >>> pd4-zook02
> >>>>>> to
> >>>>>>>>> be
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
> >>>>>>>> supposed
> >>>>>>>>> to
> >>>>>>>>>>>>> be
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>> then disconnects everyone. Then they re-elect it again,
> >>> and
> >>>>>>>> it
> >>>>>>>>>>>>> loops
> >>>>>>>>>>>>>> over and over.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -------------
> >>>>>>>>>>>>>> Server config
> >>>>>>>>>>>>>> -------------
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
> >>>>>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
> >>>>>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
> >>>>>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
> >>>>>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
> >>>>>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
> >>>>>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
> >>>>>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
> >>>>>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> group.1:1:2:3:4:5
> >>>>>>>>>>>>>> weight.1=1
> >>>>>>>>>>>>>> weight.2=1
> >>>>>>>>>>>>>> weight.3=1
> >>>>>>>>>>>>>> weight.4=1
> >>>>>>>>>>>>>> weight.5=1
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> group.2:6:7:8:9
> >>>>>>>>>>>>>> weight.6=0
> >>>>>>>>>>>>>> weight.7=0
> >>>>>>>>>>>>>> weight.8=0
> >>>>>>>>>>>>>> weight.9=0
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
> >>>>>>>> different
> >>>>>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
> >>>>>> machines
> >>>>>>>>> in
> >>>>>>>>>>>>> dc1
> >>>>>>>>>>>>>> have voting rights, and the ability to become a leader.
> >>> The
> >>>>>>>>>>>>> machines
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>> the pods all have a weight of zero, and are not
expected
> >>> to
> >>>>>>>>>>> become
> >>>>>>>>>>>>>> leaders, or to vote on transactions.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Let me know what I can do to help resolve this issue.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -Todd
> >
Re: Unending Leader Elections in WAN deploy
Posted by Patrick Hunt <ph...@apache.org>.
Todd, Mahadev and I looked at this and it turns out to be a regression.
Ironically a patch I created for 3.2 branch to add quorum tests actually
broke the quorum config -- a default value for a config parameter was
lost. I'm going to submit a patch asap to get the default back, but for
the time being you can set:
electionAlg=3
in each of your config files.
You should see reference to FastLeaderElection in your log files if this
parameter is set correctly.
Sorry for the trouble,
Patrick
Todd Greenwood wrote:
> Mahadev,
>
> I just heard from IT that this build behaves in exactly the same way as
> previous versions, e.g. we get continuous leader elections that
> disconnect the followers and then get re-elected, and disconnect...etc.
>
> This is from a fresh sync to the 3.2 branch:
>
> svn co
> http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2
> ./branch-3.2
>
> CHANGES.TXT show the various fixes included:
>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> /src/original$ head -n 50 branch-3.2/CHANGES.txt
> Release 3.2.1
>
> Backward compatibile changes:
>
> BUGFIXES:
> ZOOKEEPER-468. avoid compile warning in send_auth_info(). (chris via
> flavio)
>
> ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten (chris via
> mahadev)
>
> ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via mahadev)
>
> ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris via
> mahadev)
>
> ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure)
> (giri via mahadev)
>
> ZOOKEEPER-467. Change log level in BookieHandle (flavio via mahadev)
>
> ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent immediate
> failure. (chris via mahadev)
>
> ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev via
> phunt)
>
> ZOOKEEPER-457. Make ZookeeperMain public, support for HBase (and
> other)
> embedded clients (ryan rawson via phunt)
>
> ZOOKEEPER-481. Add lastMessageSent to QuorumCnxManager. (flavio via
> mahadev)
>
> ZOOKEEPER-479. QuorumHierarchical does not count groups correctly
> (flavio via mahadev)
>
> ZOOKEEPER-466. crash on zookeeper_close() when using auth with empty
> cert
> (Chris Darroch via phunt)
>
> ZOOKEEPER-480. FLE should perform leader check when node is not
> leading and
> add vote of follower (flavio via mahadev)
>
> ZOOKEEPER-491. Prevent zero-weight servers from being elected (flavio
> via
> mahadev)
>
> What can I do to assist you with this issue?
>
> -Todd
>
>> -----Original Message-----
>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>> Sent: Tuesday, August 04, 2009 12:43 PM
>> To: zookeeper-dev@hadoop.apache.org
>> Subject: Re: Unending Leader Elections in WAN deploy
>>
>> Hi todd,
>> comments in line
>>
>>
>> On 8/4/09 12:38 PM, "Todd Greenwood" <to...@audiencescience.com>
> wrote:
>>> Mahadev,
>>>
>>> Some quick questions:
>>>
>>> 1. Version
>>>
>>> I see that the CHANGES.txt calls this 3.2.1, but the build.xml is
> still
>>> calling this 3.2.0. Should this be rev'd, and am I correct in
> calling
>>> this release 3.2.1?
>> Yes the release is 3.2.1. The build.xml will be fixed as soon as we
> tag
>> the
>> release.
>>
>>> 2. Build targets
>>>
>>> The package target fails b/c the create-cppunit-configure target
> fails
>>> due to various problems w/ respect to autoconf. Are these
> dependencies
>>> documented somewhere ? I'd like to have a fully building system.
>>>
>>> create-cppunit-configure:
>>> [exec] Can't exec "libtoolize": No such file or directory at
>>> /usr/bin/autoreconf line 188.
>>> [exec] Use of uninitialized value $libtoolize in pattern match
>>> (m//) at /usr/bin/autoreconf line 188.
>>> [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT' not
> found
>>> in library
>>> [exec] configure.ac:33: error: possibly undefined macro:
>>> AM_PATH_CPPUNIT
>>> [exec] If this token and others are legitimate, please
> use
>>> m4_pattern_allow.
>>> [exec] See the Autoconf documentation.
>>> [exec] configure.ac:53: error: possibly undefined macro:
>>> AC_PROG_LIBTOOL
>>> [exec] autoreconf: /usr/bin/autoconf failed with exit status: 1
>>>
>> You need auto tools to run this. Please read the README for building c
>> client library at src/c/ for the installation requirements.
>>> 3. Sync failure:
>>>
>>> This is still failing.
>>>
>>> svn: URL
>>> 'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch'
>>> doesn't exist
>>>
>> Yes this hasn't been fixed yet!
>>
>> Thanks
>> mahadev
>>> -Todd
>>>
>>>> -----Original Message-----
>>>> From: Todd Greenwood
>>>> Sent: Tuesday, August 04, 2009 11:26 AM
>>>> To: 'zookeeper-user@hadoop.apache.org'
>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>
>>>> Great news. Thank you Mahadev. I'll report our findings later
> today.
>>>> -Todd
>>>>
>>>>> -----Original Message-----
>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>> Sent: Tuesday, August 04, 2009 11:20 AM
>>>>> To: zookeeper-user@hadoop.apache.org
>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>
>>>>> Hi Todd,
>>>>> I just committed 480 and 491. You can checkout the 3.2 branch
> now.
>>>>> Thanks
>>>>> mahadev
>>>>>
>>>>>
>>>>> On 8/3/09 4:29 PM, "Todd Greenwood" <to...@audiencescience.com>
>>> wrote:
>>>>>> That'd be perfect. Thanks!
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>>>> Sent: Monday, August 03, 2009 4:24 PM
>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>
>>>>>>> Hi Todd,
>>>>>>> Most of the patches that you mention should be in the branch
>>> 3.2 by
>>>>>> tomm
>>>>>>> or so. 481, 479 are already in. 480 and 491 should be in by
> tomm.
>>>>>> Would
>>>>>>> that
>>>>>>> suffice for you?
>>>>>>>
>>>>>>> Thanks
>>>>>>> mahadev
>>>>>>>
>>>>>>>
>>>>>>> On 8/3/09 4:21 PM, "Todd Greenwood" <to...@audiencescience.com>
>>>> wrote:
>>>>>>>> Another problem...I've reverted to the latest versions of the
>>>>>> patches
>>>>>>>> that are not specific to branch-3.2, and I'm getting two
>>> compilation
>>>>>>>> errors:
>>>>>>>>
>>>>>>>> build-generated:
>>>>>>>> [javac] Compiling 44 source files to
>>>>>>>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>
>>>>>>>> compile-main:
>>>>>>>> [javac] Compiling 2 source files to
>>>>>>>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>> [javac]
>>>>>>>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>> atched/branch-
>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>>>> mStats.java:30: name clash: getQuorumPeers() and
>>> getQuorumPeers()
>>>>>> have
>>>>>>>> the same erasure
>>>>>>>> [javac] public String[] getQuorumPeers();
>>>>>>>> [javac] ^
>>>>>>>> [javac]
>>>>>>>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>> atched/branch-
>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>>>> mStats.java:31: name clash: getServerState() and
>>> getServerState()
>>>>>> have
>>>>>>>> the same erasure
>>>>>>>> [javac] public String getServerState();
>>>>>>>> [javac] ^
>>>>>>>> [javac] 2 errors
>>>>>>>>
>>>>>>>> My build process is pretty simple:
>>>>>>>>
>>>>>>>> 1. copy the branch-3.2 source to a temp directory
>>>>>>>> (src/patched/branch-3.2)
>>>>>>>> 2. apply the ZOOKEEPER patches in my patches directory
>>>>>>>> 3. build zookeeper in the temp directory
>>>>>>>>
>>>>>>>> -Todd
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>>>>>>> Sent: Monday, August 03, 2009 4:09 PM
>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>
>>>>>>>>> Flavio,
>>>>>>>>> I notice that you've updated the patches referenced for the
> WAN
>>>>>>>>> deployment. There appears to be an order dependency w/ respect
>>> to
>>>>>>>> these
>>>>>>>>> four patches...
>>>>>>>>>
>>>>>>>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
>>>>>>>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
>>>>>>>>>
>>>>>>>>> 473 -> 479 (479 fails)
>>>>>>>>>
>>>>>>>>>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>>>> /src/patched/branch-3.2$ patch -p0 <
>>>>>>>>> ../patches/ZOOKEEPER-479-branch3.2.patch
>>>>>>>>> patching file
>>>>>>>>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
>>>>>>>>> ical.java
>>>>>>>>> patching file
>>>>>>>>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
>>>>>>>>> patching file
>>>>>>>>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
>>>>>>>>> .java
>>>>>>>>> patching file
>>>>>>>>>
>>> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
>>>>>>>>> Hunk #1 FAILED at 93.
>>>>>>>>> Hunk #2 FAILED at 145.
>>>>>>>>> 2 out of 2 hunks FAILED -- saving rejects to file
>>>>>>>>>
> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>>>> /src/patched/branch-3.2$ h ../patches/
>>>>>>>>>
>>>>>>>>> Could you advise as to which patches I need to apply, and in
>>> what
>>>>>>>> order?
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>> Sent: Friday, July 31, 2009 9:51 PM
>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>
>>>>>>>>>> Perfect! Thanks for the update, Todd.
>>>>>>>>>>
>>>>>>>>>> -Flavio
>>>>>>>>>>
>>>>>>>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks. You were right, I had a stale version of 479.
>>> Compilation
>>>>>>>>>>> succeeds and all tests pass on branch-3.2 with the latest
>>> patches
>>>>>>>>> 473,
>>>>>>>>>>> 479, 481, and 491.
>>>>>>>>>>>
>>>>>>>>>>> -Todd
>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:48 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> It should be in 479. Perhaps you have a stale version of
> the
>>>>>>>> patch.
>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>
>>>>>>>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Flavio,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm getting a compilation error for patch 491:
>>>>>>>>>>>>>
>>>>>>>>>>>>> compile-main:
>>>>>>>>>>>>> [javac] Compiling 1 source file to
>>>>>>>>>>>>>
>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>>>> src/p
>>>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>>>> [javac]
>>>>>>>>>>>>>
>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>>>> src/p
>>>>>>>>>>>>>
>>> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>>>> FastL
>>>>>>>>>>>>> eaderElection.java:601: cannot find symbol
>>>>>>>>>>>>> [javac] symbol : method getWeight(long)
>>>>>>>>>>>>> [javac] location: interface
>>>>>>>>>>>>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
>>>>>>>>>>>>> [javac]
>>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>>>> [javac]
>>> ^
>>>>>>>>>>>>> [javac] 1 error
>>>>>>>>>>>>>
>>>>>>>>>>>>> I see a reference to getWeight in both
>>> FastLeaderElection.java
>>>>>>>> in
>>>>>>>>>>>>> patch
>>>>>>>>>>>>> 491:
>>>>>>>>>>>>>
>>>>>>>>>>>>> patches/ZOOKEEPER-491.patch:+
>>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>>>> FastLeaderElection.java
>>>>>>>>>>>>> :
>>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
>>>>>>>>>>>>> 0)
>>>>>>>>>>>>>
>>>>>>>>>>>>> However, I don't see a reference to this method in patches
>>> 473,
>>>>>>>>> 479,
>>>>>>>>>>>>> or
>>>>>>>>>>>>> 481. I also don't see a reference to this method in the
>>>>>> trunk...
>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>
>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
>>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ok, I'll apply that patch and report back.
>>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
>>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You're missing 491 from your set of patches.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This repro's in both branch-3.2, and
>>> branch-3.2+patches(473,
>>>>>>>>> 479,
>>>>>>>>>>>>>> 481).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Basically, it seems like the nodes are electing
>>> pd4-zook02
>>>>>> to
>>>>>>>>> be
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
>>>>>>>> supposed
>>>>>>>>> to
>>>>>>>>>>>>> be
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> then disconnects everyone. Then they re-elect it again,
>>> and
>>>>>>>> it
>>>>>>>>>>>>> loops
>>>>>>>>>>>>>> over and over.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -------------
>>>>>>>>>>>>>> Server config
>>>>>>>>>>>>>> -------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
>>>>>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
>>>>>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
>>>>>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
>>>>>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
>>>>>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
>>>>>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
>>>>>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
>>>>>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> group.1:1:2:3:4:5
>>>>>>>>>>>>>> weight.1=1
>>>>>>>>>>>>>> weight.2=1
>>>>>>>>>>>>>> weight.3=1
>>>>>>>>>>>>>> weight.4=1
>>>>>>>>>>>>>> weight.5=1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> group.2:6:7:8:9
>>>>>>>>>>>>>> weight.6=0
>>>>>>>>>>>>>> weight.7=0
>>>>>>>>>>>>>> weight.8=0
>>>>>>>>>>>>>> weight.9=0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
>>>>>>>> different
>>>>>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
>>>>>> machines
>>>>>>>>> in
>>>>>>>>>>>>> dc1
>>>>>>>>>>>>>> have voting rights, and the ability to become a leader.
>>> The
>>>>>>>>>>>>> machines
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> the pods all have a weight of zero, and are not expected
>>> to
>>>>>>>>>>> become
>>>>>>>>>>>>>> leaders, or to vote on transactions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let me know what I can do to help resolve this issue.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Todd
>
RE: Unending Leader Elections in WAN deploy
Posted by Todd Greenwood <to...@audiencescience.com>.
Will do.
> -----Original Message-----
> From: Patrick Hunt [mailto:phunt@apache.org]
> Sent: Tuesday, August 04, 2009 1:34 PM
> To: zookeeper-dev@hadoop.apache.org
> Subject: Re: Unending Leader Elections in WAN deploy
>
> It would be better to create a JIRA with configs as well as logs.
>
> Patrick
>
> Mahadev Konar wrote:
> > Hi Todd,
> >
> > What is the synclimit you are using? Can you post your config? For
> WAN's
> > you will have to use much bigger values for synclimit and others.
> >
> > Thanks
> > mahadev
> >
> >
> > On 8/4/09 1:24 PM, "Todd Greenwood" <to...@audiencescience.com>
wrote:
> >
> >> Mahadev,
> >>
> >> I just heard from IT that this build behaves in exactly the same
way as
> >> previous versions, e.g. we get continuous leader elections that
> >> disconnect the followers and then get re-elected, and
disconnect...etc.
> >>
> >> This is from a fresh sync to the 3.2 branch:
> >>
> >> svn co
> >>
http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2
> >> ./branch-3.2
> >>
> >> CHANGES.TXT show the various fixes included:
> >>
> >>
>
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> >> /src/original$ head -n 50 branch-3.2/CHANGES.txt
> >> Release 3.2.1
> >>
> >> Backward compatibile changes:
> >>
> >> BUGFIXES:
> >> ZOOKEEPER-468. avoid compile warning in send_auth_info(). (chris
via
> >> flavio)
> >>
> >> ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten (chris
via
> >> mahadev)
> >>
> >> ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via
mahadev)
> >>
> >> ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris via
> >> mahadev)
> >>
> >> ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure)
> >> (giri via mahadev)
> >>
> >> ZOOKEEPER-467. Change log level in BookieHandle (flavio via
mahadev)
> >>
> >> ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent
immediate
> >> failure. (chris via mahadev)
> >>
> >> ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev
via
> >> phunt)
> >>
> >> ZOOKEEPER-457. Make ZookeeperMain public, support for HBase (and
> >> other)
> >> embedded clients (ryan rawson via phunt)
> >>
> >> ZOOKEEPER-481. Add lastMessageSent to QuorumCnxManager. (flavio
via
> >> mahadev)
> >>
> >> ZOOKEEPER-479. QuorumHierarchical does not count groups
correctly
> >> (flavio via mahadev)
> >>
> >> ZOOKEEPER-466. crash on zookeeper_close() when using auth with
empty
> >> cert
> >> (Chris Darroch via phunt)
> >>
> >> ZOOKEEPER-480. FLE should perform leader check when node is not
> >> leading and
> >> add vote of follower (flavio via mahadev)
> >>
> >> ZOOKEEPER-491. Prevent zero-weight servers from being elected
(flavio
> >> via
> >> mahadev)
> >>
> >> What can I do to assist you with this issue?
> >>
> >> -Todd
> >>
> >>> -----Original Message-----
> >>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> >>> Sent: Tuesday, August 04, 2009 12:43 PM
> >>> To: zookeeper-dev@hadoop.apache.org
> >>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>
> >>> Hi todd,
> >>> comments in line
> >>>
> >>>
> >>> On 8/4/09 12:38 PM, "Todd Greenwood" <to...@audiencescience.com>
> >> wrote:
> >>>> Mahadev,
> >>>>
> >>>> Some quick questions:
> >>>>
> >>>> 1. Version
> >>>>
> >>>> I see that the CHANGES.txt calls this 3.2.1, but the build.xml is
> >> still
> >>>> calling this 3.2.0. Should this be rev'd, and am I correct in
> >> calling
> >>>> this release 3.2.1?
> >>> Yes the release is 3.2.1. The build.xml will be fixed as soon as
we
> >> tag
> >>> the
> >>> release.
> >>>
> >>>> 2. Build targets
> >>>>
> >>>> The package target fails b/c the create-cppunit-configure target
> >> fails
> >>>> due to various problems w/ respect to autoconf. Are these
> >> dependencies
> >>>> documented somewhere ? I'd like to have a fully building system.
> >>>>
> >>>> create-cppunit-configure:
> >>>> [exec] Can't exec "libtoolize": No such file or directory at
> >>>> /usr/bin/autoreconf line 188.
> >>>> [exec] Use of uninitialized value $libtoolize in pattern
match
> >>>> (m//) at /usr/bin/autoreconf line 188.
> >>>> [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT' not
> >> found
> >>>> in library
> >>>> [exec] configure.ac:33: error: possibly undefined macro:
> >>>> AM_PATH_CPPUNIT
> >>>> [exec] If this token and others are legitimate, please
> >> use
> >>>> m4_pattern_allow.
> >>>> [exec] See the Autoconf documentation.
> >>>> [exec] configure.ac:53: error: possibly undefined macro:
> >>>> AC_PROG_LIBTOOL
> >>>> [exec] autoreconf: /usr/bin/autoconf failed with exit
status: 1
> >>>>
> >>> You need auto tools to run this. Please read the README for
building c
> >>> client library at src/c/ for the installation requirements.
> >>>> 3. Sync failure:
> >>>>
> >>>> This is still failing.
> >>>>
> >>>> svn: URL
> >>>>
'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch'
> >>>> doesn't exist
> >>>>
> >>> Yes this hasn't been fixed yet!
> >>>
> >>> Thanks
> >>> mahadev
> >>>> -Todd
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Todd Greenwood
> >>>>> Sent: Tuesday, August 04, 2009 11:26 AM
> >>>>> To: 'zookeeper-user@hadoop.apache.org'
> >>>>> Subject: RE: Unending Leader Elections in WAN deploy
> >>>>>
> >>>>> Great news. Thank you Mahadev. I'll report our findings later
> >> today.
> >>>>> -Todd
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> >>>>>> Sent: Tuesday, August 04, 2009 11:20 AM
> >>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>
> >>>>>> Hi Todd,
> >>>>>> I just committed 480 and 491. You can checkout the 3.2 branch
> >> now.
> >>>>>> Thanks
> >>>>>> mahadev
> >>>>>>
> >>>>>>
> >>>>>> On 8/3/09 4:29 PM, "Todd Greenwood" <to...@audiencescience.com>
> >>>> wrote:
> >>>>>>> That'd be perfect. Thanks!
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> >>>>>>>> Sent: Monday, August 03, 2009 4:24 PM
> >>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>
> >>>>>>>> Hi Todd,
> >>>>>>>> Most of the patches that you mention should be in the
branch
> >>>> 3.2 by
> >>>>>>> tomm
> >>>>>>>> or so. 481, 479 are already in. 480 and 491 should be in by
> >> tomm.
> >>>>>>> Would
> >>>>>>>> that
> >>>>>>>> suffice for you?
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>> mahadev
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 8/3/09 4:21 PM, "Todd Greenwood"
<to...@audiencescience.com>
> >>>>> wrote:
> >>>>>>>>> Another problem...I've reverted to the latest versions of
the
> >>>>>>> patches
> >>>>>>>>> that are not specific to branch-3.2, and I'm getting two
> >>>> compilation
> >>>>>>>>> errors:
> >>>>>>>>>
> >>>>>>>>> build-generated:
> >>>>>>>>> [javac] Compiling 44 source files to
> >>>>>>>>>
> >>
>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>>>>> atched/branch-3.2/build/classes
> >>>>>>>>>
> >>>>>>>>> compile-main:
> >>>>>>>>> [javac] Compiling 2 source files to
> >>>>>>>>>
> >>
>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>>>>> atched/branch-3.2/build/classes
> >>>>>>>>> [javac]
> >>>>>>>>>
> >>
>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>>> atched/branch-
> >>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> >>>>>>>>> mStats.java:30: name clash: getQuorumPeers() and
> >>>> getQuorumPeers()
> >>>>>>> have
> >>>>>>>>> the same erasure
> >>>>>>>>> [javac] public String[] getQuorumPeers();
> >>>>>>>>> [javac] ^
> >>>>>>>>> [javac]
> >>>>>>>>>
> >>
>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>>> atched/branch-
> >>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> >>>>>>>>> mStats.java:31: name clash: getServerState() and
> >>>> getServerState()
> >>>>>>> have
> >>>>>>>>> the same erasure
> >>>>>>>>> [javac] public String getServerState();
> >>>>>>>>> [javac] ^
> >>>>>>>>> [javac] 2 errors
> >>>>>>>>>
> >>>>>>>>> My build process is pretty simple:
> >>>>>>>>>
> >>>>>>>>> 1. copy the branch-3.2 source to a temp directory
> >>>>>>>>> (src/patched/branch-3.2)
> >>>>>>>>> 2. apply the ZOOKEEPER patches in my patches directory
> >>>>>>>>> 3. build zookeeper in the temp directory
> >>>>>>>>>
> >>>>>>>>> -Todd
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> >>>>>>>>>> Sent: Monday, August 03, 2009 4:09 PM
> >>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
> >>>>>>>>>>
> >>>>>>>>>> Flavio,
> >>>>>>>>>> I notice that you've updated the patches referenced for the
> >> WAN
> >>>>>>>>>> deployment. There appears to be an order dependency w/
respect
> >>>> to
> >>>>>>>>> these
> >>>>>>>>>> four patches...
> >>>>>>>>>>
> >>>>>>>>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
> >>>>>>>>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
> >>>>>>>>>>
> >>>>>>>>>> 473 -> 479 (479 fails)
> >>>>>>>>>>
> >>>>>>>>>>
> >>
>
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> >>>>>>>>>> /src/patched/branch-3.2$ patch -p0 <
> >>>>>>>>>> ../patches/ZOOKEEPER-479-branch3.2.patch
> >>>>>>>>>> patching file
> >>>>>>>>>>
> >>
>
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
> >>>>>>>>>> ical.java
> >>>>>>>>>> patching file
> >>>>>>>>>>
> >>
>
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
> >>>>>>>>>> patching file
> >>>>>>>>>>
> >>
>
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
> >>>>>>>>>> .java
> >>>>>>>>>> patching file
> >>>>>>>>>>
> >>>>
src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
> >>>>>>>>>> Hunk #1 FAILED at 93.
> >>>>>>>>>> Hunk #2 FAILED at 145.
> >>>>>>>>>> 2 out of 2 hunks FAILED -- saving rejects to file
> >>>>>>>>>>
> >>
src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
> >>
>
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> >>>>>>>>>> /src/patched/branch-3.2$ h ../patches/
> >>>>>>>>>>
> >>>>>>>>>> Could you advise as to which patches I need to apply, and
in
> >>>> what
> >>>>>>>>> order?
> >>>>>>>>>> -Todd
> >>>>>>>>>>
> >>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>>>>>>>> Sent: Friday, July 31, 2009 9:51 PM
> >>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>>>>
> >>>>>>>>>>> Perfect! Thanks for the update, Todd.
> >>>>>>>>>>>
> >>>>>>>>>>> -Flavio
> >>>>>>>>>>>
> >>>>>>>>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Thanks. You were right, I had a stale version of 479.
> >>>> Compilation
> >>>>>>>>>>>> succeeds and all tests pass on branch-3.2 with the latest
> >>>> patches
> >>>>>>>>>> 473,
> >>>>>>>>>>>> 479, 481, and 491.
> >>>>>>>>>>>>
> >>>>>>>>>>>> -Todd
> >>>>>>>>>>>>
> >>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:48 PM
> >>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> It should be in 479. Perhaps you have a stale version of
> >> the
> >>>>>>>>> patch.
> >>>>>>>>>>>>> -Flavio
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Flavio,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I'm getting a compilation error for patch 491:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> compile-main:
> >>>>>>>>>>>>> [javac] Compiling 1 source file to
> >>>>>>>>>>>>>
> >>>>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> >>>>>>>>>>>>> src/p
> >>>>>>>>>>>>> atched/branch-3.2/build/classes
> >>>>>>>>>>>>> [javac]
> >>>>>>>>>>>>>
> >>>>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> >>>>>>>>>>>>> src/p
> >>>>>>>>>>>>>
> >>>>
atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
> >>>>>>>>>>>>> FastL
> >>>>>>>>>>>>> eaderElection.java:601: cannot find symbol
> >>>>>>>>>>>>> [javac] symbol : method getWeight(long)
> >>>>>>>>>>>>> [javac] location: interface
> >>>>>>>>>>>>>
org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
> >>>>>>>>>>>>> [javac]
> >>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> >>>>>>>>>>>>> [javac]
> >>>> ^
> >>>>>>>>>>>>> [javac] 1 error
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I see a reference to getWeight in both
> >>>> FastLeaderElection.java
> >>>>>>>>> in
> >>>>>>>>>>>>> patch
> >>>>>>>>>>>>> 491:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> patches/ZOOKEEPER-491.patch:+
> >>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> >>>>>>>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
> >>>>>>>>>>>>> FastLeaderElection.java
> >>>>>>>>>>>>> :
> >>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
> >>>>>>>>>>>>> 0)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> However, I don't see a reference to this method in
patches
> >>>> 473,
> >>>>>>>>>> 479,
> >>>>>>>>>>>>> or
> >>>>>>>>>>>>> 481. I also don't see a reference to this method in the
> >>>>>>> trunk...
> >>>>>>>>>>>>> -Todd
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> >>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
> >>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Ok, I'll apply that patch and report back.
> >>>>>>>>>>>>> -Todd
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
> >>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> You're missing 491 from your set of patches.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Flavio
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This repro's in both branch-3.2, and
> >>>> branch-3.2+patches(473,
> >>>>>>>>>> 479,
> >>>>>>>>>>>>> 481).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Basically, it seems like the nodes are electing
> >>>> pd4-zook02
> >>>>>>> to
> >>>>>>>>>> be
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
> >>>>>>>>> supposed
> >>>>>>>>>> to
> >>>>>>>>>>>>> be
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>> then disconnects everyone. Then they re-elect it again,
> >>>> and
> >>>>>>>>> it
> >>>>>>>>>>>>> loops
> >>>>>>>>>>>>> over and over.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -------------
> >>>>>>>>>>>>> Server config
> >>>>>>>>>>>>> -------------
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
> >>>>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
> >>>>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
> >>>>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
> >>>>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
> >>>>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
> >>>>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
> >>>>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
> >>>>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> group.1:1:2:3:4:5
> >>>>>>>>>>>>> weight.1=1
> >>>>>>>>>>>>> weight.2=1
> >>>>>>>>>>>>> weight.3=1
> >>>>>>>>>>>>> weight.4=1
> >>>>>>>>>>>>> weight.5=1
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> group.2:6:7:8:9
> >>>>>>>>>>>>> weight.6=0
> >>>>>>>>>>>>> weight.7=0
> >>>>>>>>>>>>> weight.8=0
> >>>>>>>>>>>>> weight.9=0
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
> >>>>>>>>> different
> >>>>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
> >>>>>>> machines
> >>>>>>>>>> in
> >>>>>>>>>>>>> dc1
> >>>>>>>>>>>>> have voting rights, and the ability to become a leader.
> >>>> The
> >>>>>>>>>>>>> machines
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>> the pods all have a weight of zero, and are not expected
> >>>> to
> >>>>>>>>>>>> become
> >>>>>>>>>>>>> leaders, or to vote on transactions.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Let me know what I can do to help resolve this issue.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Todd
> >>>>>>>>>>>>>
> >
Re: Unending Leader Elections in WAN deploy
Posted by Patrick Hunt <ph...@apache.org>.
It would be better to create a JIRA with configs as well as logs.
Patrick
Mahadev Konar wrote:
> Hi Todd,
>
> What is the synclimit you are using? Can you post your config? For WAN's
> you will have to use much bigger values for synclimit and others.
>
> Thanks
> mahadev
>
>
> On 8/4/09 1:24 PM, "Todd Greenwood" <to...@audiencescience.com> wrote:
>
>> Mahadev,
>>
>> I just heard from IT that this build behaves in exactly the same way as
>> previous versions, e.g. we get continuous leader elections that
>> disconnect the followers and then get re-elected, and disconnect...etc.
>>
>> This is from a fresh sync to the 3.2 branch:
>>
>> svn co
>> http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2
>> ./branch-3.2
>>
>> CHANGES.TXT show the various fixes included:
>>
>> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>> /src/original$ head -n 50 branch-3.2/CHANGES.txt
>> Release 3.2.1
>>
>> Backward compatibile changes:
>>
>> BUGFIXES:
>> ZOOKEEPER-468. avoid compile warning in send_auth_info(). (chris via
>> flavio)
>>
>> ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten (chris via
>> mahadev)
>>
>> ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via mahadev)
>>
>> ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris via
>> mahadev)
>>
>> ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure)
>> (giri via mahadev)
>>
>> ZOOKEEPER-467. Change log level in BookieHandle (flavio via mahadev)
>>
>> ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent immediate
>> failure. (chris via mahadev)
>>
>> ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev via
>> phunt)
>>
>> ZOOKEEPER-457. Make ZookeeperMain public, support for HBase (and
>> other)
>> embedded clients (ryan rawson via phunt)
>>
>> ZOOKEEPER-481. Add lastMessageSent to QuorumCnxManager. (flavio via
>> mahadev)
>>
>> ZOOKEEPER-479. QuorumHierarchical does not count groups correctly
>> (flavio via mahadev)
>>
>> ZOOKEEPER-466. crash on zookeeper_close() when using auth with empty
>> cert
>> (Chris Darroch via phunt)
>>
>> ZOOKEEPER-480. FLE should perform leader check when node is not
>> leading and
>> add vote of follower (flavio via mahadev)
>>
>> ZOOKEEPER-491. Prevent zero-weight servers from being elected (flavio
>> via
>> mahadev)
>>
>> What can I do to assist you with this issue?
>>
>> -Todd
>>
>>> -----Original Message-----
>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>> Sent: Tuesday, August 04, 2009 12:43 PM
>>> To: zookeeper-dev@hadoop.apache.org
>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>
>>> Hi todd,
>>> comments in line
>>>
>>>
>>> On 8/4/09 12:38 PM, "Todd Greenwood" <to...@audiencescience.com>
>> wrote:
>>>> Mahadev,
>>>>
>>>> Some quick questions:
>>>>
>>>> 1. Version
>>>>
>>>> I see that the CHANGES.txt calls this 3.2.1, but the build.xml is
>> still
>>>> calling this 3.2.0. Should this be rev'd, and am I correct in
>> calling
>>>> this release 3.2.1?
>>> Yes the release is 3.2.1. The build.xml will be fixed as soon as we
>> tag
>>> the
>>> release.
>>>
>>>> 2. Build targets
>>>>
>>>> The package target fails b/c the create-cppunit-configure target
>> fails
>>>> due to various problems w/ respect to autoconf. Are these
>> dependencies
>>>> documented somewhere ? I'd like to have a fully building system.
>>>>
>>>> create-cppunit-configure:
>>>> [exec] Can't exec "libtoolize": No such file or directory at
>>>> /usr/bin/autoreconf line 188.
>>>> [exec] Use of uninitialized value $libtoolize in pattern match
>>>> (m//) at /usr/bin/autoreconf line 188.
>>>> [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT' not
>> found
>>>> in library
>>>> [exec] configure.ac:33: error: possibly undefined macro:
>>>> AM_PATH_CPPUNIT
>>>> [exec] If this token and others are legitimate, please
>> use
>>>> m4_pattern_allow.
>>>> [exec] See the Autoconf documentation.
>>>> [exec] configure.ac:53: error: possibly undefined macro:
>>>> AC_PROG_LIBTOOL
>>>> [exec] autoreconf: /usr/bin/autoconf failed with exit status: 1
>>>>
>>> You need auto tools to run this. Please read the README for building c
>>> client library at src/c/ for the installation requirements.
>>>> 3. Sync failure:
>>>>
>>>> This is still failing.
>>>>
>>>> svn: URL
>>>> 'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch'
>>>> doesn't exist
>>>>
>>> Yes this hasn't been fixed yet!
>>>
>>> Thanks
>>> mahadev
>>>> -Todd
>>>>
>>>>> -----Original Message-----
>>>>> From: Todd Greenwood
>>>>> Sent: Tuesday, August 04, 2009 11:26 AM
>>>>> To: 'zookeeper-user@hadoop.apache.org'
>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>
>>>>> Great news. Thank you Mahadev. I'll report our findings later
>> today.
>>>>> -Todd
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>>> Sent: Tuesday, August 04, 2009 11:20 AM
>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>
>>>>>> Hi Todd,
>>>>>> I just committed 480 and 491. You can checkout the 3.2 branch
>> now.
>>>>>> Thanks
>>>>>> mahadev
>>>>>>
>>>>>>
>>>>>> On 8/3/09 4:29 PM, "Todd Greenwood" <to...@audiencescience.com>
>>>> wrote:
>>>>>>> That'd be perfect. Thanks!
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>>>>> Sent: Monday, August 03, 2009 4:24 PM
>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>
>>>>>>>> Hi Todd,
>>>>>>>> Most of the patches that you mention should be in the branch
>>>> 3.2 by
>>>>>>> tomm
>>>>>>>> or so. 481, 479 are already in. 480 and 491 should be in by
>> tomm.
>>>>>>> Would
>>>>>>>> that
>>>>>>>> suffice for you?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> mahadev
>>>>>>>>
>>>>>>>>
>>>>>>>> On 8/3/09 4:21 PM, "Todd Greenwood" <to...@audiencescience.com>
>>>>> wrote:
>>>>>>>>> Another problem...I've reverted to the latest versions of the
>>>>>>> patches
>>>>>>>>> that are not specific to branch-3.2, and I'm getting two
>>>> compilation
>>>>>>>>> errors:
>>>>>>>>>
>>>>>>>>> build-generated:
>>>>>>>>> [javac] Compiling 44 source files to
>>>>>>>>>
>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>
>>>>>>>>> compile-main:
>>>>>>>>> [javac] Compiling 2 source files to
>>>>>>>>>
>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>> [javac]
>>>>>>>>>
>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>> atched/branch-
>>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>>>>> mStats.java:30: name clash: getQuorumPeers() and
>>>> getQuorumPeers()
>>>>>>> have
>>>>>>>>> the same erasure
>>>>>>>>> [javac] public String[] getQuorumPeers();
>>>>>>>>> [javac] ^
>>>>>>>>> [javac]
>>>>>>>>>
>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>> atched/branch-
>>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>>>>> mStats.java:31: name clash: getServerState() and
>>>> getServerState()
>>>>>>> have
>>>>>>>>> the same erasure
>>>>>>>>> [javac] public String getServerState();
>>>>>>>>> [javac] ^
>>>>>>>>> [javac] 2 errors
>>>>>>>>>
>>>>>>>>> My build process is pretty simple:
>>>>>>>>>
>>>>>>>>> 1. copy the branch-3.2 source to a temp directory
>>>>>>>>> (src/patched/branch-3.2)
>>>>>>>>> 2. apply the ZOOKEEPER patches in my patches directory
>>>>>>>>> 3. build zookeeper in the temp directory
>>>>>>>>>
>>>>>>>>> -Todd
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>>>>>>>> Sent: Monday, August 03, 2009 4:09 PM
>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>>
>>>>>>>>>> Flavio,
>>>>>>>>>> I notice that you've updated the patches referenced for the
>> WAN
>>>>>>>>>> deployment. There appears to be an order dependency w/ respect
>>>> to
>>>>>>>>> these
>>>>>>>>>> four patches...
>>>>>>>>>>
>>>>>>>>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
>>>>>>>>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
>>>>>>>>>>
>>>>>>>>>> 473 -> 479 (479 fails)
>>>>>>>>>>
>>>>>>>>>>
>> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>>>>> /src/patched/branch-3.2$ patch -p0 <
>>>>>>>>>> ../patches/ZOOKEEPER-479-branch3.2.patch
>>>>>>>>>> patching file
>>>>>>>>>>
>> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
>>>>>>>>>> ical.java
>>>>>>>>>> patching file
>>>>>>>>>>
>> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
>>>>>>>>>> patching file
>>>>>>>>>>
>> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
>>>>>>>>>> .java
>>>>>>>>>> patching file
>>>>>>>>>>
>>>> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
>>>>>>>>>> Hunk #1 FAILED at 93.
>>>>>>>>>> Hunk #2 FAILED at 145.
>>>>>>>>>> 2 out of 2 hunks FAILED -- saving rejects to file
>>>>>>>>>>
>> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
>> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>>>>> /src/patched/branch-3.2$ h ../patches/
>>>>>>>>>>
>>>>>>>>>> Could you advise as to which patches I need to apply, and in
>>>> what
>>>>>>>>> order?
>>>>>>>>>> -Todd
>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>> Sent: Friday, July 31, 2009 9:51 PM
>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>
>>>>>>>>>>> Perfect! Thanks for the update, Todd.
>>>>>>>>>>>
>>>>>>>>>>> -Flavio
>>>>>>>>>>>
>>>>>>>>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks. You were right, I had a stale version of 479.
>>>> Compilation
>>>>>>>>>>>> succeeds and all tests pass on branch-3.2 with the latest
>>>> patches
>>>>>>>>>> 473,
>>>>>>>>>>>> 479, 481, and 491.
>>>>>>>>>>>>
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:48 PM
>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>>
>>>>>>>>>>>>> It should be in 479. Perhaps you have a stale version of
>> the
>>>>>>>>> patch.
>>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Flavio,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm getting a compilation error for patch 491:
>>>>>>>>>>>>>
>>>>>>>>>>>>> compile-main:
>>>>>>>>>>>>> [javac] Compiling 1 source file to
>>>>>>>>>>>>>
>>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>>>> src/p
>>>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>>>> [javac]
>>>>>>>>>>>>>
>>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>>>> src/p
>>>>>>>>>>>>>
>>>> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>>>> FastL
>>>>>>>>>>>>> eaderElection.java:601: cannot find symbol
>>>>>>>>>>>>> [javac] symbol : method getWeight(long)
>>>>>>>>>>>>> [javac] location: interface
>>>>>>>>>>>>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
>>>>>>>>>>>>> [javac]
>>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>>>> [javac]
>>>> ^
>>>>>>>>>>>>> [javac] 1 error
>>>>>>>>>>>>>
>>>>>>>>>>>>> I see a reference to getWeight in both
>>>> FastLeaderElection.java
>>>>>>>>> in
>>>>>>>>>>>>> patch
>>>>>>>>>>>>> 491:
>>>>>>>>>>>>>
>>>>>>>>>>>>> patches/ZOOKEEPER-491.patch:+
>>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>>>> FastLeaderElection.java
>>>>>>>>>>>>> :
>>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
>>>>>>>>>>>>> 0)
>>>>>>>>>>>>>
>>>>>>>>>>>>> However, I don't see a reference to this method in patches
>>>> 473,
>>>>>>>>>> 479,
>>>>>>>>>>>>> or
>>>>>>>>>>>>> 481. I also don't see a reference to this method in the
>>>>>>> trunk...
>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ok, I'll apply that patch and report back.
>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
>>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>>
>>>>>>>>>>>>> You're missing 491 from your set of patches.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> This repro's in both branch-3.2, and
>>>> branch-3.2+patches(473,
>>>>>>>>>> 479,
>>>>>>>>>>>>> 481).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Basically, it seems like the nodes are electing
>>>> pd4-zook02
>>>>>>> to
>>>>>>>>>> be
>>>>>>>>>>>>> the
>>>>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
>>>>>>>>> supposed
>>>>>>>>>> to
>>>>>>>>>>>>> be
>>>>>>>>>>>>> and
>>>>>>>>>>>>> then disconnects everyone. Then they re-elect it again,
>>>> and
>>>>>>>>> it
>>>>>>>>>>>>> loops
>>>>>>>>>>>>> over and over.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -------------
>>>>>>>>>>>>> Server config
>>>>>>>>>>>>> -------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
>>>>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
>>>>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
>>>>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
>>>>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
>>>>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
>>>>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
>>>>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
>>>>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
>>>>>>>>>>>>>
>>>>>>>>>>>>> group.1:1:2:3:4:5
>>>>>>>>>>>>> weight.1=1
>>>>>>>>>>>>> weight.2=1
>>>>>>>>>>>>> weight.3=1
>>>>>>>>>>>>> weight.4=1
>>>>>>>>>>>>> weight.5=1
>>>>>>>>>>>>>
>>>>>>>>>>>>> group.2:6:7:8:9
>>>>>>>>>>>>> weight.6=0
>>>>>>>>>>>>> weight.7=0
>>>>>>>>>>>>> weight.8=0
>>>>>>>>>>>>> weight.9=0
>>>>>>>>>>>>>
>>>>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
>>>>>>>>> different
>>>>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
>>>>>>> machines
>>>>>>>>>> in
>>>>>>>>>>>>> dc1
>>>>>>>>>>>>> have voting rights, and the ability to become a leader.
>>>> The
>>>>>>>>>>>>> machines
>>>>>>>>>>>>> in
>>>>>>>>>>>>> the pods all have a weight of zero, and are not expected
>>>> to
>>>>>>>>>>>> become
>>>>>>>>>>>>> leaders, or to vote on transactions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Let me know what I can do to help resolve this issue.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>
>
Re: Unending Leader Elections in WAN deploy
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi Todd,
What is the synclimit you are using? Can you post your config? For WAN's
you will have to use much bigger values for synclimit and others.
Thanks
mahadev
On 8/4/09 1:24 PM, "Todd Greenwood" <to...@audiencescience.com> wrote:
> Mahadev,
>
> I just heard from IT that this build behaves in exactly the same way as
> previous versions, e.g. we get continuous leader elections that
> disconnect the followers and then get re-elected, and disconnect...etc.
>
> This is from a fresh sync to the 3.2 branch:
>
> svn co
> http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2
> ./branch-3.2
>
> CHANGES.TXT show the various fixes included:
>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> /src/original$ head -n 50 branch-3.2/CHANGES.txt
> Release 3.2.1
>
> Backward compatibile changes:
>
> BUGFIXES:
> ZOOKEEPER-468. avoid compile warning in send_auth_info(). (chris via
> flavio)
>
> ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten (chris via
> mahadev)
>
> ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via mahadev)
>
> ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris via
> mahadev)
>
> ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure)
> (giri via mahadev)
>
> ZOOKEEPER-467. Change log level in BookieHandle (flavio via mahadev)
>
> ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent immediate
> failure. (chris via mahadev)
>
> ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev via
> phunt)
>
> ZOOKEEPER-457. Make ZookeeperMain public, support for HBase (and
> other)
> embedded clients (ryan rawson via phunt)
>
> ZOOKEEPER-481. Add lastMessageSent to QuorumCnxManager. (flavio via
> mahadev)
>
> ZOOKEEPER-479. QuorumHierarchical does not count groups correctly
> (flavio via mahadev)
>
> ZOOKEEPER-466. crash on zookeeper_close() when using auth with empty
> cert
> (Chris Darroch via phunt)
>
> ZOOKEEPER-480. FLE should perform leader check when node is not
> leading and
> add vote of follower (flavio via mahadev)
>
> ZOOKEEPER-491. Prevent zero-weight servers from being elected (flavio
> via
> mahadev)
>
> What can I do to assist you with this issue?
>
> -Todd
>
>> -----Original Message-----
>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>> Sent: Tuesday, August 04, 2009 12:43 PM
>> To: zookeeper-dev@hadoop.apache.org
>> Subject: Re: Unending Leader Elections in WAN deploy
>>
>> Hi todd,
>> comments in line
>>
>>
>> On 8/4/09 12:38 PM, "Todd Greenwood" <to...@audiencescience.com>
> wrote:
>>
>>> Mahadev,
>>>
>>> Some quick questions:
>>>
>>> 1. Version
>>>
>>> I see that the CHANGES.txt calls this 3.2.1, but the build.xml is
> still
>>> calling this 3.2.0. Should this be rev'd, and am I correct in
> calling
>>> this release 3.2.1?
>> Yes the release is 3.2.1. The build.xml will be fixed as soon as we
> tag
>> the
>> release.
>>
>>>
>>> 2. Build targets
>>>
>>> The package target fails b/c the create-cppunit-configure target
> fails
>>> due to various problems w/ respect to autoconf. Are these
> dependencies
>>> documented somewhere ? I'd like to have a fully building system.
>>>
>>> create-cppunit-configure:
>>> [exec] Can't exec "libtoolize": No such file or directory at
>>> /usr/bin/autoreconf line 188.
>>> [exec] Use of uninitialized value $libtoolize in pattern match
>>> (m//) at /usr/bin/autoreconf line 188.
>>> [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT' not
> found
>>> in library
>>> [exec] configure.ac:33: error: possibly undefined macro:
>>> AM_PATH_CPPUNIT
>>> [exec] If this token and others are legitimate, please
> use
>>> m4_pattern_allow.
>>> [exec] See the Autoconf documentation.
>>> [exec] configure.ac:53: error: possibly undefined macro:
>>> AC_PROG_LIBTOOL
>>> [exec] autoreconf: /usr/bin/autoconf failed with exit status: 1
>>>
>> You need auto tools to run this. Please read the README for building c
>> client library at src/c/ for the installation requirements.
>>>
>>> 3. Sync failure:
>>>
>>> This is still failing.
>>>
>>> svn: URL
>>> 'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch'
>>> doesn't exist
>>>
>>
>> Yes this hasn't been fixed yet!
>>
>> Thanks
>> mahadev
>>> -Todd
>>>
>>>> -----Original Message-----
>>>> From: Todd Greenwood
>>>> Sent: Tuesday, August 04, 2009 11:26 AM
>>>> To: 'zookeeper-user@hadoop.apache.org'
>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>
>>>> Great news. Thank you Mahadev. I'll report our findings later
> today.
>>>> -Todd
>>>>
>>>>> -----Original Message-----
>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>> Sent: Tuesday, August 04, 2009 11:20 AM
>>>>> To: zookeeper-user@hadoop.apache.org
>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>
>>>>> Hi Todd,
>>>>> I just committed 480 and 491. You can checkout the 3.2 branch
> now.
>>>>>
>>>>> Thanks
>>>>> mahadev
>>>>>
>>>>>
>>>>> On 8/3/09 4:29 PM, "Todd Greenwood" <to...@audiencescience.com>
>>> wrote:
>>>>>
>>>>>> That'd be perfect. Thanks!
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>>>> Sent: Monday, August 03, 2009 4:24 PM
>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>
>>>>>>> Hi Todd,
>>>>>>> Most of the patches that you mention should be in the branch
>>> 3.2 by
>>>>>> tomm
>>>>>>> or so. 481, 479 are already in. 480 and 491 should be in by
> tomm.
>>>>>> Would
>>>>>>> that
>>>>>>> suffice for you?
>>>>>>>
>>>>>>> Thanks
>>>>>>> mahadev
>>>>>>>
>>>>>>>
>>>>>>> On 8/3/09 4:21 PM, "Todd Greenwood" <to...@audiencescience.com>
>>>> wrote:
>>>>>>>
>>>>>>>> Another problem...I've reverted to the latest versions of the
>>>>>> patches
>>>>>>>> that are not specific to branch-3.2, and I'm getting two
>>> compilation
>>>>>>>> errors:
>>>>>>>>
>>>>>>>> build-generated:
>>>>>>>> [javac] Compiling 44 source files to
>>>>>>>>
>>>>>>
>>>>
>>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>
>>>>>>>> compile-main:
>>>>>>>> [javac] Compiling 2 source files to
>>>>>>>>
>>>>>>
>>>>
>>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>> [javac]
>>>>>>>>
>>>>>>
>>>>
>>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>
>>>>>> atched/branch-
>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>>>> mStats.java:30: name clash: getQuorumPeers() and
>>> getQuorumPeers()
>>>>>> have
>>>>>>>> the same erasure
>>>>>>>> [javac] public String[] getQuorumPeers();
>>>>>>>> [javac] ^
>>>>>>>> [javac]
>>>>>>>>
>>>>>>
>>>>
>>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>>>
>>>>>> atched/branch-
>>>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>>>> mStats.java:31: name clash: getServerState() and
>>> getServerState()
>>>>>> have
>>>>>>>> the same erasure
>>>>>>>> [javac] public String getServerState();
>>>>>>>> [javac] ^
>>>>>>>> [javac] 2 errors
>>>>>>>>
>>>>>>>> My build process is pretty simple:
>>>>>>>>
>>>>>>>> 1. copy the branch-3.2 source to a temp directory
>>>>>>>> (src/patched/branch-3.2)
>>>>>>>> 2. apply the ZOOKEEPER patches in my patches directory
>>>>>>>> 3. build zookeeper in the temp directory
>>>>>>>>
>>>>>>>> -Todd
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>>>>>>> Sent: Monday, August 03, 2009 4:09 PM
>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>
>>>>>>>>> Flavio,
>>>>>>>>> I notice that you've updated the patches referenced for the
> WAN
>>>>>>>>> deployment. There appears to be an order dependency w/ respect
>>> to
>>>>>>>> these
>>>>>>>>> four patches...
>>>>>>>>>
>>>>>>>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
>>>>>>>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
>>>>>>>>>
>>>>>>>>> 473 -> 479 (479 fails)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>>>> /src/patched/branch-3.2$ patch -p0 <
>>>>>>>>> ../patches/ZOOKEEPER-479-branch3.2.patch
>>>>>>>>> patching file
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
>>>>>>>>> ical.java
>>>>>>>>> patching file
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
>>>>>>>>> patching file
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
>>>>>>>>> .java
>>>>>>>>> patching file
>>>>>>>>>
>>> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
>>>>>>>>> Hunk #1 FAILED at 93.
>>>>>>>>> Hunk #2 FAILED at 145.
>>>>>>>>> 2 out of 2 hunks FAILED -- saving rejects to file
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>>>> /src/patched/branch-3.2$ h ../patches/
>>>>>>>>>
>>>>>>>>> Could you advise as to which patches I need to apply, and in
>>> what
>>>>>>>> order?
>>>>>>>>>
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>> Sent: Friday, July 31, 2009 9:51 PM
>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>
>>>>>>>>>> Perfect! Thanks for the update, Todd.
>>>>>>>>>>
>>>>>>>>>> -Flavio
>>>>>>>>>>
>>>>>>>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks. You were right, I had a stale version of 479.
>>> Compilation
>>>>>>>>>>> succeeds and all tests pass on branch-3.2 with the latest
>>> patches
>>>>>>>>> 473,
>>>>>>>>>>> 479, 481, and 491.
>>>>>>>>>>>
>>>>>>>>>>> -Todd
>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:48 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> It should be in 479. Perhaps you have a stale version of
> the
>>>>>>>> patch.
>>>>>>>>>>>>
>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>
>>>>>>>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Flavio,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm getting a compilation error for patch 491:
>>>>>>>>>>>>
>>>>>>>>>>>> compile-main:
>>>>>>>>>>>> [javac] Compiling 1 source file to
>>>>>>>>>>>>
>>>>>>>>>
>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>>> src/p
>>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>>> [javac]
>>>>>>>>>>>>
>>>>>>>>>
>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>>> src/p
>>>>>>>>>>>>
>>>>>>>>>
>>> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>>> FastL
>>>>>>>>>>>> eaderElection.java:601: cannot find symbol
>>>>>>>>>>>> [javac] symbol : method getWeight(long)
>>>>>>>>>>>> [javac] location: interface
>>>>>>>>>>>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
>>>>>>>>>>>> [javac]
>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>>> [javac]
>>> ^
>>>>>>>>>>>> [javac] 1 error
>>>>>>>>>>>>
>>>>>>>>>>>> I see a reference to getWeight in both
>>> FastLeaderElection.java
>>>>>>>> in
>>>>>>>>>>>> patch
>>>>>>>>>>>> 491:
>>>>>>>>>>>>
>>>>>>>>>>>> patches/ZOOKEEPER-491.patch:+
>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>>> FastLeaderElection.java
>>>>>>>>>>>> :
>>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
>>>>>>>>>>>> 0)
>>>>>>>>>>>>
>>>>>>>>>>>> However, I don't see a reference to this method in patches
>>> 473,
>>>>>>>>> 479,
>>>>>>>>>>>> or
>>>>>>>>>>>> 481. I also don't see a reference to this method in the
>>>>>> trunk...
>>>>>>>>>>>>
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> Ok, I'll apply that patch and report back.
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> You're missing 491 from your set of patches.
>>>>>>>>>>>>
>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>
>>>>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> This repro's in both branch-3.2, and
>>> branch-3.2+patches(473,
>>>>>>>>> 479,
>>>>>>>>>>>> 481).
>>>>>>>>>>>>
>>>>>>>>>>>> Basically, it seems like the nodes are electing
>>> pd4-zook02
>>>>>> to
>>>>>>>>> be
>>>>>>>>>>>> the
>>>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
>>>>>>>> supposed
>>>>>>>>> to
>>>>>>>>>>>> be
>>>>>>>>>>>> and
>>>>>>>>>>>> then disconnects everyone. Then they re-elect it again,
>>> and
>>>>>>>> it
>>>>>>>>>>>> loops
>>>>>>>>>>>> over and over.
>>>>>>>>>>>>
>>>>>>>>>>>> -------------
>>>>>>>>>>>> Server config
>>>>>>>>>>>> -------------
>>>>>>>>>>>>
>>>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
>>>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
>>>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
>>>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
>>>>>>>>>>>>
>>>>>>>>>>>> group.1:1:2:3:4:5
>>>>>>>>>>>> weight.1=1
>>>>>>>>>>>> weight.2=1
>>>>>>>>>>>> weight.3=1
>>>>>>>>>>>> weight.4=1
>>>>>>>>>>>> weight.5=1
>>>>>>>>>>>>
>>>>>>>>>>>> group.2:6:7:8:9
>>>>>>>>>>>> weight.6=0
>>>>>>>>>>>> weight.7=0
>>>>>>>>>>>> weight.8=0
>>>>>>>>>>>> weight.9=0
>>>>>>>>>>>>
>>>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
>>>>>>>> different
>>>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
>>>>>> machines
>>>>>>>>> in
>>>>>>>>>>>> dc1
>>>>>>>>>>>> have voting rights, and the ability to become a leader.
>>> The
>>>>>>>>>>>> machines
>>>>>>>>>>>> in
>>>>>>>>>>>> the pods all have a weight of zero, and are not expected
>>> to
>>>>>>>>>>> become
>>>>>>>>>>>> leaders, or to vote on transactions.
>>>>>>>>>>>>
>>>>>>>>>>>> Let me know what I can do to help resolve this issue.
>>>>>>>>>>>>
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>
>
RE: Unending Leader Elections in WAN deploy
Posted by Todd Greenwood <to...@audiencescience.com>.
Mahadev,
I just heard from IT that this build behaves in exactly the same way as
previous versions, e.g. we get continuous leader elections that
disconnect the followers and then get re-elected, and disconnect...etc.
This is from a fresh sync to the 3.2 branch:
svn co
http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2
./branch-3.2
CHANGES.TXT show the various fixes included:
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
/src/original$ head -n 50 branch-3.2/CHANGES.txt
Release 3.2.1
Backward compatibile changes:
BUGFIXES:
ZOOKEEPER-468. avoid compile warning in send_auth_info(). (chris via
flavio)
ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten (chris via
mahadev)
ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via mahadev)
ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris via
mahadev)
ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure)
(giri via mahadev)
ZOOKEEPER-467. Change log level in BookieHandle (flavio via mahadev)
ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent immediate
failure. (chris via mahadev)
ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev via
phunt)
ZOOKEEPER-457. Make ZookeeperMain public, support for HBase (and
other)
embedded clients (ryan rawson via phunt)
ZOOKEEPER-481. Add lastMessageSent to QuorumCnxManager. (flavio via
mahadev)
ZOOKEEPER-479. QuorumHierarchical does not count groups correctly
(flavio via mahadev)
ZOOKEEPER-466. crash on zookeeper_close() when using auth with empty
cert
(Chris Darroch via phunt)
ZOOKEEPER-480. FLE should perform leader check when node is not
leading and
add vote of follower (flavio via mahadev)
ZOOKEEPER-491. Prevent zero-weight servers from being elected (flavio
via
mahadev)
What can I do to assist you with this issue?
-Todd
> -----Original Message-----
> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> Sent: Tuesday, August 04, 2009 12:43 PM
> To: zookeeper-dev@hadoop.apache.org
> Subject: Re: Unending Leader Elections in WAN deploy
>
> Hi todd,
> comments in line
>
>
> On 8/4/09 12:38 PM, "Todd Greenwood" <to...@audiencescience.com>
wrote:
>
> > Mahadev,
> >
> > Some quick questions:
> >
> > 1. Version
> >
> > I see that the CHANGES.txt calls this 3.2.1, but the build.xml is
still
> > calling this 3.2.0. Should this be rev'd, and am I correct in
calling
> > this release 3.2.1?
> Yes the release is 3.2.1. The build.xml will be fixed as soon as we
tag
> the
> release.
>
> >
> > 2. Build targets
> >
> > The package target fails b/c the create-cppunit-configure target
fails
> > due to various problems w/ respect to autoconf. Are these
dependencies
> > documented somewhere ? I'd like to have a fully building system.
> >
> > create-cppunit-configure:
> > [exec] Can't exec "libtoolize": No such file or directory at
> > /usr/bin/autoreconf line 188.
> > [exec] Use of uninitialized value $libtoolize in pattern match
> > (m//) at /usr/bin/autoreconf line 188.
> > [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT' not
found
> > in library
> > [exec] configure.ac:33: error: possibly undefined macro:
> > AM_PATH_CPPUNIT
> > [exec] If this token and others are legitimate, please
use
> > m4_pattern_allow.
> > [exec] See the Autoconf documentation.
> > [exec] configure.ac:53: error: possibly undefined macro:
> > AC_PROG_LIBTOOL
> > [exec] autoreconf: /usr/bin/autoconf failed with exit status: 1
> >
> You need auto tools to run this. Please read the README for building c
> client library at src/c/ for the installation requirements.
> >
> > 3. Sync failure:
> >
> > This is still failing.
> >
> > svn: URL
> > 'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch'
> > doesn't exist
> >
>
> Yes this hasn't been fixed yet!
>
> Thanks
> mahadev
> > -Todd
> >
> >> -----Original Message-----
> >> From: Todd Greenwood
> >> Sent: Tuesday, August 04, 2009 11:26 AM
> >> To: 'zookeeper-user@hadoop.apache.org'
> >> Subject: RE: Unending Leader Elections in WAN deploy
> >>
> >> Great news. Thank you Mahadev. I'll report our findings later
today.
> >> -Todd
> >>
> >>> -----Original Message-----
> >>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> >>> Sent: Tuesday, August 04, 2009 11:20 AM
> >>> To: zookeeper-user@hadoop.apache.org
> >>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>
> >>> Hi Todd,
> >>> I just committed 480 and 491. You can checkout the 3.2 branch
now.
> >>>
> >>> Thanks
> >>> mahadev
> >>>
> >>>
> >>> On 8/3/09 4:29 PM, "Todd Greenwood" <to...@audiencescience.com>
> > wrote:
> >>>
> >>>> That'd be perfect. Thanks!
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> >>>>> Sent: Monday, August 03, 2009 4:24 PM
> >>>>> To: zookeeper-user@hadoop.apache.org
> >>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>
> >>>>> Hi Todd,
> >>>>> Most of the patches that you mention should be in the branch
> > 3.2 by
> >>>> tomm
> >>>>> or so. 481, 479 are already in. 480 and 491 should be in by
tomm.
> >>>> Would
> >>>>> that
> >>>>> suffice for you?
> >>>>>
> >>>>> Thanks
> >>>>> mahadev
> >>>>>
> >>>>>
> >>>>> On 8/3/09 4:21 PM, "Todd Greenwood" <to...@audiencescience.com>
> >> wrote:
> >>>>>
> >>>>>> Another problem...I've reverted to the latest versions of the
> >>>> patches
> >>>>>> that are not specific to branch-3.2, and I'm getting two
> > compilation
> >>>>>> errors:
> >>>>>>
> >>>>>> build-generated:
> >>>>>> [javac] Compiling 44 source files to
> >>>>>>
> >>>>
> >>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>> atched/branch-3.2/build/classes
> >>>>>>
> >>>>>> compile-main:
> >>>>>> [javac] Compiling 2 source files to
> >>>>>>
> >>>>
> >>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>> atched/branch-3.2/build/classes
> >>>>>> [javac]
> >>>>>>
> >>>>
> >>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>>
> >>>> atched/branch-
> >> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> >>>>>> mStats.java:30: name clash: getQuorumPeers() and
> > getQuorumPeers()
> >>>> have
> >>>>>> the same erasure
> >>>>>> [javac] public String[] getQuorumPeers();
> >>>>>> [javac] ^
> >>>>>> [javac]
> >>>>>>
> >>>>
> >>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>>>>
> >>>> atched/branch-
> >> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> >>>>>> mStats.java:31: name clash: getServerState() and
> > getServerState()
> >>>> have
> >>>>>> the same erasure
> >>>>>> [javac] public String getServerState();
> >>>>>> [javac] ^
> >>>>>> [javac] 2 errors
> >>>>>>
> >>>>>> My build process is pretty simple:
> >>>>>>
> >>>>>> 1. copy the branch-3.2 source to a temp directory
> >>>>>> (src/patched/branch-3.2)
> >>>>>> 2. apply the ZOOKEEPER patches in my patches directory
> >>>>>> 3. build zookeeper in the temp directory
> >>>>>>
> >>>>>> -Todd
> >>>>>>> -----Original Message-----
> >>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> >>>>>>> Sent: Monday, August 03, 2009 4:09 PM
> >>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
> >>>>>>>
> >>>>>>> Flavio,
> >>>>>>> I notice that you've updated the patches referenced for the
WAN
> >>>>>>> deployment. There appears to be an order dependency w/ respect
> > to
> >>>>>> these
> >>>>>>> four patches...
> >>>>>>>
> >>>>>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
> >>>>>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
> >>>>>>>
> >>>>>>> 473 -> 479 (479 fails)
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> >
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> >>>>>>> /src/patched/branch-3.2$ patch -p0 <
> >>>>>>> ../patches/ZOOKEEPER-479-branch3.2.patch
> >>>>>>> patching file
> >>>>>>>
> >>>>>>
> >>>>
> >>
> >
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
> >>>>>>> ical.java
> >>>>>>> patching file
> >>>>>>>
> >>>>>>
> >>>>
> >>
> >
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
> >>>>>>> patching file
> >>>>>>>
> >>>>>>
> >>>>
> >>
> >
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
> >>>>>>> .java
> >>>>>>> patching file
> >>>>>>>
> > src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
> >>>>>>> Hunk #1 FAILED at 93.
> >>>>>>> Hunk #2 FAILED at 145.
> >>>>>>> 2 out of 2 hunks FAILED -- saving rejects to file
> >>>>>>>
> >>>>>>
> >>>>
> >>
> >
src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
> >>>>>>>
> >>>>>>
> >>>>
> >>
> >
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> >>>>>>> /src/patched/branch-3.2$ h ../patches/
> >>>>>>>
> >>>>>>> Could you advise as to which patches I need to apply, and in
> > what
> >>>>>> order?
> >>>>>>>
> >>>>>>> -Todd
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>>>>> Sent: Friday, July 31, 2009 9:51 PM
> >>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>
> >>>>>>>> Perfect! Thanks for the update, Todd.
> >>>>>>>>
> >>>>>>>> -Flavio
> >>>>>>>>
> >>>>>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
> >>>>>>>>
> >>>>>>>>> Thanks. You were right, I had a stale version of 479.
> > Compilation
> >>>>>>>>> succeeds and all tests pass on branch-3.2 with the latest
> > patches
> >>>>>>> 473,
> >>>>>>>>> 479, 481, and 491.
> >>>>>>>>>
> >>>>>>>>> -Todd
> >>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>>>>>>> Sent: Friday, July 31, 2009 7:48 PM
> >>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>>>
> >>>>>>>>>> It should be in 479. Perhaps you have a stale version of
the
> >>>>>> patch.
> >>>>>>>>>>
> >>>>>>>>>> -Flavio
> >>>>>>>>>>
> >>>>>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Flavio,
> >>>>>>>>>>>
> >>>>>>>>>>> I'm getting a compilation error for patch 491:
> >>>>>>>>>>>
> >>>>>>>>>>> compile-main:
> >>>>>>>>>>> [javac] Compiling 1 source file to
> >>>>>>>>>>>
> >>>>>>>
> > /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> >>>>>>>>>>> src/p
> >>>>>>>>>>> atched/branch-3.2/build/classes
> >>>>>>>>>>> [javac]
> >>>>>>>>>>>
> >>>>>>>
> > /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> >>>>>>>>>>> src/p
> >>>>>>>>>>>
> >>>>>>>
> > atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
> >>>>>>>>>>> FastL
> >>>>>>>>>>> eaderElection.java:601: cannot find symbol
> >>>>>>>>>>> [javac] symbol : method getWeight(long)
> >>>>>>>>>>> [javac] location: interface
> >>>>>>>>>>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
> >>>>>>>>>>> [javac]
> >>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> >>>>>>>>>>> [javac]
> > ^
> >>>>>>>>>>> [javac] 1 error
> >>>>>>>>>>>
> >>>>>>>>>>> I see a reference to getWeight in both
> > FastLeaderElection.java
> >>>>>> in
> >>>>>>>>>>> patch
> >>>>>>>>>>> 491:
> >>>>>>>>>>>
> >>>>>>>>>>> patches/ZOOKEEPER-491.patch:+
> >>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> >>>>>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
> >>>>>>>>>>> FastLeaderElection.java
> >>>>>>>>>>> :
> >>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
> >>>>>>>>>>> 0)
> >>>>>>>>>>>
> >>>>>>>>>>> However, I don't see a reference to this method in patches
> > 473,
> >>>>>>> 479,
> >>>>>>>>>>> or
> >>>>>>>>>>> 481. I also don't see a reference to this method in the
> >>>> trunk...
> >>>>>>>>>>>
> >>>>>>>>>>> -Todd
> >>>>>>>>>>>
> >>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> >>>>>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
> >>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
> >>>>>>>>>>>>
> >>>>>>>>>>>> Ok, I'll apply that patch and report back.
> >>>>>>>>>>>> -Todd
> >>>>>>>>>>>>
> >>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
> >>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>>>>>
> >>>>>>>>>>>> You're missing 491 from your set of patches.
> >>>>>>>>>>>>
> >>>>>>>>>>>> -Flavio
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> This repro's in both branch-3.2, and
> > branch-3.2+patches(473,
> >>>>>>> 479,
> >>>>>>>>>>>> 481).
> >>>>>>>>>>>>
> >>>>>>>>>>>> Basically, it seems like the nodes are electing
> > pd4-zook02
> >>>> to
> >>>>>>> be
> >>>>>>>>>>> the
> >>>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
> >>>>>> supposed
> >>>>>>> to
> >>>>>>>>>>> be
> >>>>>>>>>>>> and
> >>>>>>>>>>>> then disconnects everyone. Then they re-elect it again,
> > and
> >>>>>> it
> >>>>>>>>>>> loops
> >>>>>>>>>>>> over and over.
> >>>>>>>>>>>>
> >>>>>>>>>>>> -------------
> >>>>>>>>>>>> Server config
> >>>>>>>>>>>> -------------
> >>>>>>>>>>>>
> >>>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
> >>>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
> >>>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
> >>>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
> >>>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
> >>>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
> >>>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
> >>>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
> >>>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
> >>>>>>>>>>>>
> >>>>>>>>>>>> group.1:1:2:3:4:5
> >>>>>>>>>>>> weight.1=1
> >>>>>>>>>>>> weight.2=1
> >>>>>>>>>>>> weight.3=1
> >>>>>>>>>>>> weight.4=1
> >>>>>>>>>>>> weight.5=1
> >>>>>>>>>>>>
> >>>>>>>>>>>> group.2:6:7:8:9
> >>>>>>>>>>>> weight.6=0
> >>>>>>>>>>>> weight.7=0
> >>>>>>>>>>>> weight.8=0
> >>>>>>>>>>>> weight.9=0
> >>>>>>>>>>>>
> >>>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
> >>>>>> different
> >>>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
> >>>> machines
> >>>>>>> in
> >>>>>>>>>>> dc1
> >>>>>>>>>>>> have voting rights, and the ability to become a leader.
> > The
> >>>>>>>>>>> machines
> >>>>>>>>>>>> in
> >>>>>>>>>>>> the pods all have a weight of zero, and are not expected
> > to
> >>>>>>>>> become
> >>>>>>>>>>>> leaders, or to vote on transactions.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Let me know what I can do to help resolve this issue.
> >>>>>>>>>>>>
> >>>>>>>>>>>> -Todd
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> >
Re: Unending Leader Elections in WAN deploy
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi todd,
comments in line
On 8/4/09 12:38 PM, "Todd Greenwood" <to...@audiencescience.com> wrote:
> Mahadev,
>
> Some quick questions:
>
> 1. Version
>
> I see that the CHANGES.txt calls this 3.2.1, but the build.xml is still
> calling this 3.2.0. Should this be rev'd, and am I correct in calling
> this release 3.2.1?
Yes the release is 3.2.1. The build.xml will be fixed as soon as we tag the
release.
>
> 2. Build targets
>
> The package target fails b/c the create-cppunit-configure target fails
> due to various problems w/ respect to autoconf. Are these dependencies
> documented somewhere ? I'd like to have a fully building system.
>
> create-cppunit-configure:
> [exec] Can't exec "libtoolize": No such file or directory at
> /usr/bin/autoreconf line 188.
> [exec] Use of uninitialized value $libtoolize in pattern match
> (m//) at /usr/bin/autoreconf line 188.
> [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT' not found
> in library
> [exec] configure.ac:33: error: possibly undefined macro:
> AM_PATH_CPPUNIT
> [exec] If this token and others are legitimate, please use
> m4_pattern_allow.
> [exec] See the Autoconf documentation.
> [exec] configure.ac:53: error: possibly undefined macro:
> AC_PROG_LIBTOOL
> [exec] autoreconf: /usr/bin/autoconf failed with exit status: 1
>
You need auto tools to run this. Please read the README for building c
client library at src/c/ for the installation requirements.
>
> 3. Sync failure:
>
> This is still failing.
>
> svn: URL
> 'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch'
> doesn't exist
>
Yes this hasn't been fixed yet!
Thanks
mahadev
> -Todd
>
>> -----Original Message-----
>> From: Todd Greenwood
>> Sent: Tuesday, August 04, 2009 11:26 AM
>> To: 'zookeeper-user@hadoop.apache.org'
>> Subject: RE: Unending Leader Elections in WAN deploy
>>
>> Great news. Thank you Mahadev. I'll report our findings later today.
>> -Todd
>>
>>> -----Original Message-----
>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>> Sent: Tuesday, August 04, 2009 11:20 AM
>>> To: zookeeper-user@hadoop.apache.org
>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>
>>> Hi Todd,
>>> I just committed 480 and 491. You can checkout the 3.2 branch now.
>>>
>>> Thanks
>>> mahadev
>>>
>>>
>>> On 8/3/09 4:29 PM, "Todd Greenwood" <to...@audiencescience.com>
> wrote:
>>>
>>>> That'd be perfect. Thanks!
>>>>
>>>>> -----Original Message-----
>>>>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>>>>> Sent: Monday, August 03, 2009 4:24 PM
>>>>> To: zookeeper-user@hadoop.apache.org
>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>
>>>>> Hi Todd,
>>>>> Most of the patches that you mention should be in the branch
> 3.2 by
>>>> tomm
>>>>> or so. 481, 479 are already in. 480 and 491 should be in by tomm.
>>>> Would
>>>>> that
>>>>> suffice for you?
>>>>>
>>>>> Thanks
>>>>> mahadev
>>>>>
>>>>>
>>>>> On 8/3/09 4:21 PM, "Todd Greenwood" <to...@audiencescience.com>
>> wrote:
>>>>>
>>>>>> Another problem...I've reverted to the latest versions of the
>>>> patches
>>>>>> that are not specific to branch-3.2, and I'm getting two
> compilation
>>>>>> errors:
>>>>>>
>>>>>> build-generated:
>>>>>> [javac] Compiling 44 source files to
>>>>>>
>>>>
>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>> atched/branch-3.2/build/classes
>>>>>>
>>>>>> compile-main:
>>>>>> [javac] Compiling 2 source files to
>>>>>>
>>>>
>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>> atched/branch-3.2/build/classes
>>>>>> [javac]
>>>>>>
>>>>
>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>
>>>> atched/branch-
>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>> mStats.java:30: name clash: getQuorumPeers() and
> getQuorumPeers()
>>>> have
>>>>>> the same erasure
>>>>>> [javac] public String[] getQuorumPeers();
>>>>>> [javac] ^
>>>>>> [javac]
>>>>>>
>>>>
>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>>>>
>>>> atched/branch-
>> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>>>>> mStats.java:31: name clash: getServerState() and
> getServerState()
>>>> have
>>>>>> the same erasure
>>>>>> [javac] public String getServerState();
>>>>>> [javac] ^
>>>>>> [javac] 2 errors
>>>>>>
>>>>>> My build process is pretty simple:
>>>>>>
>>>>>> 1. copy the branch-3.2 source to a temp directory
>>>>>> (src/patched/branch-3.2)
>>>>>> 2. apply the ZOOKEEPER patches in my patches directory
>>>>>> 3. build zookeeper in the temp directory
>>>>>>
>>>>>> -Todd
>>>>>>> -----Original Message-----
>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>>>>> Sent: Monday, August 03, 2009 4:09 PM
>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>
>>>>>>> Flavio,
>>>>>>> I notice that you've updated the patches referenced for the WAN
>>>>>>> deployment. There appears to be an order dependency w/ respect
> to
>>>>>> these
>>>>>>> four patches...
>>>>>>>
>>>>>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
>>>>>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
>>>>>>>
>>>>>>> 473 -> 479 (479 fails)
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>> /src/patched/branch-3.2$ patch -p0 <
>>>>>>> ../patches/ZOOKEEPER-479-branch3.2.patch
>>>>>>> patching file
>>>>>>>
>>>>>>
>>>>
>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
>>>>>>> ical.java
>>>>>>> patching file
>>>>>>>
>>>>>>
>>>>
>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
>>>>>>> patching file
>>>>>>>
>>>>>>
>>>>
>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
>>>>>>> .java
>>>>>>> patching file
>>>>>>>
> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
>>>>>>> Hunk #1 FAILED at 93.
>>>>>>> Hunk #2 FAILED at 145.
>>>>>>> 2 out of 2 hunks FAILED -- saving rejects to file
>>>>>>>
>>>>>>
>>>>
>>
> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
>>>>>>>
>>>>>>
>>>>
>>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>>>>> /src/patched/branch-3.2$ h ../patches/
>>>>>>>
>>>>>>> Could you advise as to which patches I need to apply, and in
> what
>>>>>> order?
>>>>>>>
>>>>>>> -Todd
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>> Sent: Friday, July 31, 2009 9:51 PM
>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>
>>>>>>>> Perfect! Thanks for the update, Todd.
>>>>>>>>
>>>>>>>> -Flavio
>>>>>>>>
>>>>>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
>>>>>>>>
>>>>>>>>> Thanks. You were right, I had a stale version of 479.
> Compilation
>>>>>>>>> succeeds and all tests pass on branch-3.2 with the latest
> patches
>>>>>>> 473,
>>>>>>>>> 479, 481, and 491.
>>>>>>>>>
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>> Sent: Friday, July 31, 2009 7:48 PM
>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>
>>>>>>>>>> It should be in 479. Perhaps you have a stale version of the
>>>>>> patch.
>>>>>>>>>>
>>>>>>>>>> -Flavio
>>>>>>>>>>
>>>>>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
>>>>>>>>>>
>>>>>>>>>>> Flavio,
>>>>>>>>>>>
>>>>>>>>>>> I'm getting a compilation error for patch 491:
>>>>>>>>>>>
>>>>>>>>>>> compile-main:
>>>>>>>>>>> [javac] Compiling 1 source file to
>>>>>>>>>>>
>>>>>>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>> src/p
>>>>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>>>>> [javac]
>>>>>>>>>>>
>>>>>>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>>>>> src/p
>>>>>>>>>>>
>>>>>>>
> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>> FastL
>>>>>>>>>>> eaderElection.java:601: cannot find symbol
>>>>>>>>>>> [javac] symbol : method getWeight(long)
>>>>>>>>>>> [javac] location: interface
>>>>>>>>>>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
>>>>>>>>>>> [javac]
>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>> [javac]
> ^
>>>>>>>>>>> [javac] 1 error
>>>>>>>>>>>
>>>>>>>>>>> I see a reference to getWeight in both
> FastLeaderElection.java
>>>>>> in
>>>>>>>>>>> patch
>>>>>>>>>>> 491:
>>>>>>>>>>>
>>>>>>>>>>> patches/ZOOKEEPER-491.patch:+
>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>>>>> FastLeaderElection.java
>>>>>>>>>>> :
>>>>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
>>>>>>>>>>> 0)
>>>>>>>>>>>
>>>>>>>>>>> However, I don't see a reference to this method in patches
> 473,
>>>>>>> 479,
>>>>>>>>>>> or
>>>>>>>>>>> 481. I also don't see a reference to this method in the
>>>> trunk...
>>>>>>>>>>>
>>>>>>>>>>> -Todd
>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> Ok, I'll apply that patch and report back.
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
>>>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>>>
>>>>>>>>>>>> You're missing 491 from your set of patches.
>>>>>>>>>>>>
>>>>>>>>>>>> -Flavio
>>>>>>>>>>>>
>>>>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> This repro's in both branch-3.2, and
> branch-3.2+patches(473,
>>>>>>> 479,
>>>>>>>>>>>> 481).
>>>>>>>>>>>>
>>>>>>>>>>>> Basically, it seems like the nodes are electing
> pd4-zook02
>>>> to
>>>>>>> be
>>>>>>>>>>> the
>>>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
>>>>>> supposed
>>>>>>> to
>>>>>>>>>>> be
>>>>>>>>>>>> and
>>>>>>>>>>>> then disconnects everyone. Then they re-elect it again,
> and
>>>>>> it
>>>>>>>>>>> loops
>>>>>>>>>>>> over and over.
>>>>>>>>>>>>
>>>>>>>>>>>> -------------
>>>>>>>>>>>> Server config
>>>>>>>>>>>> -------------
>>>>>>>>>>>>
>>>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
>>>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
>>>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
>>>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
>>>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
>>>>>>>>>>>>
>>>>>>>>>>>> group.1:1:2:3:4:5
>>>>>>>>>>>> weight.1=1
>>>>>>>>>>>> weight.2=1
>>>>>>>>>>>> weight.3=1
>>>>>>>>>>>> weight.4=1
>>>>>>>>>>>> weight.5=1
>>>>>>>>>>>>
>>>>>>>>>>>> group.2:6:7:8:9
>>>>>>>>>>>> weight.6=0
>>>>>>>>>>>> weight.7=0
>>>>>>>>>>>> weight.8=0
>>>>>>>>>>>> weight.9=0
>>>>>>>>>>>>
>>>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
>>>>>> different
>>>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
>>>> machines
>>>>>>> in
>>>>>>>>>>> dc1
>>>>>>>>>>>> have voting rights, and the ability to become a leader.
> The
>>>>>>>>>>> machines
>>>>>>>>>>>> in
>>>>>>>>>>>> the pods all have a weight of zero, and are not expected
> to
>>>>>>>>> become
>>>>>>>>>>>> leaders, or to vote on transactions.
>>>>>>>>>>>>
>>>>>>>>>>>> Let me know what I can do to help resolve this issue.
>>>>>>>>>>>>
>>>>>>>>>>>> -Todd
>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>
RE: Unending Leader Elections in WAN deploy
Posted by Todd Greenwood <to...@audiencescience.com>.
Mahadev,
Some quick questions:
1. Version
I see that the CHANGES.txt calls this 3.2.1, but the build.xml is still
calling this 3.2.0. Should this be rev'd, and am I correct in calling
this release 3.2.1?
2. Build targets
The package target fails b/c the create-cppunit-configure target fails
due to various problems w/ respect to autoconf. Are these dependencies
documented somewhere ? I'd like to have a fully building system.
create-cppunit-configure:
[exec] Can't exec "libtoolize": No such file or directory at
/usr/bin/autoreconf line 188.
[exec] Use of uninitialized value $libtoolize in pattern match
(m//) at /usr/bin/autoreconf line 188.
[exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT' not found
in library
[exec] configure.ac:33: error: possibly undefined macro:
AM_PATH_CPPUNIT
[exec] If this token and others are legitimate, please use
m4_pattern_allow.
[exec] See the Autoconf documentation.
[exec] configure.ac:53: error: possibly undefined macro:
AC_PROG_LIBTOOL
[exec] autoreconf: /usr/bin/autoconf failed with exit status: 1
3. Sync failure:
This is still failing.
svn: URL
'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch'
doesn't exist
-Todd
> -----Original Message-----
> From: Todd Greenwood
> Sent: Tuesday, August 04, 2009 11:26 AM
> To: 'zookeeper-user@hadoop.apache.org'
> Subject: RE: Unending Leader Elections in WAN deploy
>
> Great news. Thank you Mahadev. I'll report our findings later today.
> -Todd
>
> > -----Original Message-----
> > From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> > Sent: Tuesday, August 04, 2009 11:20 AM
> > To: zookeeper-user@hadoop.apache.org
> > Subject: Re: Unending Leader Elections in WAN deploy
> >
> > Hi Todd,
> > I just committed 480 and 491. You can checkout the 3.2 branch now.
> >
> > Thanks
> > mahadev
> >
> >
> > On 8/3/09 4:29 PM, "Todd Greenwood" <to...@audiencescience.com>
wrote:
> >
> > > That'd be perfect. Thanks!
> > >
> > >> -----Original Message-----
> > >> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> > >> Sent: Monday, August 03, 2009 4:24 PM
> > >> To: zookeeper-user@hadoop.apache.org
> > >> Subject: Re: Unending Leader Elections in WAN deploy
> > >>
> > >> Hi Todd,
> > >> Most of the patches that you mention should be in the branch
3.2 by
> > > tomm
> > >> or so. 481, 479 are already in. 480 and 491 should be in by tomm.
> > > Would
> > >> that
> > >> suffice for you?
> > >>
> > >> Thanks
> > >> mahadev
> > >>
> > >>
> > >> On 8/3/09 4:21 PM, "Todd Greenwood" <to...@audiencescience.com>
> wrote:
> > >>
> > >>> Another problem...I've reverted to the latest versions of the
> > > patches
> > >>> that are not specific to branch-3.2, and I'm getting two
compilation
> > >>> errors:
> > >>>
> > >>> build-generated:
> > >>> [javac] Compiling 44 source files to
> > >>>
> > >
>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> > >>> atched/branch-3.2/build/classes
> > >>>
> > >>> compile-main:
> > >>> [javac] Compiling 2 source files to
> > >>>
> > >
>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> > >>> atched/branch-3.2/build/classes
> > >>> [javac]
> > >>>
> > >
>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> > >>>
> > > atched/branch-
> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> > >>> mStats.java:30: name clash: getQuorumPeers() and
getQuorumPeers()
> > > have
> > >>> the same erasure
> > >>> [javac] public String[] getQuorumPeers();
> > >>> [javac] ^
> > >>> [javac]
> > >>>
> > >
>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> > >>>
> > > atched/branch-
> 3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> > >>> mStats.java:31: name clash: getServerState() and
getServerState()
> > > have
> > >>> the same erasure
> > >>> [javac] public String getServerState();
> > >>> [javac] ^
> > >>> [javac] 2 errors
> > >>>
> > >>> My build process is pretty simple:
> > >>>
> > >>> 1. copy the branch-3.2 source to a temp directory
> > >>> (src/patched/branch-3.2)
> > >>> 2. apply the ZOOKEEPER patches in my patches directory
> > >>> 3. build zookeeper in the temp directory
> > >>>
> > >>> -Todd
> > >>>> -----Original Message-----
> > >>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> > >>>> Sent: Monday, August 03, 2009 4:09 PM
> > >>>> To: zookeeper-user@hadoop.apache.org
> > >>>> Subject: RE: Unending Leader Elections in WAN deploy
> > >>>>
> > >>>> Flavio,
> > >>>> I notice that you've updated the patches referenced for the WAN
> > >>>> deployment. There appears to be an order dependency w/ respect
to
> > >>> these
> > >>>> four patches...
> > >>>>
> > >>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
> > >>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
> > >>>>
> > >>>> 473 -> 479 (479 fails)
> > >>>>
> > >>>>
> > >>>
> > >
>
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> > >>>> /src/patched/branch-3.2$ patch -p0 <
> > >>>> ../patches/ZOOKEEPER-479-branch3.2.patch
> > >>>> patching file
> > >>>>
> > >>>
> > >
>
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
> > >>>> ical.java
> > >>>> patching file
> > >>>>
> > >>>
> > >
>
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
> > >>>> patching file
> > >>>>
> > >>>
> > >
>
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
> > >>>> .java
> > >>>> patching file
> > >>>>
src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
> > >>>> Hunk #1 FAILED at 93.
> > >>>> Hunk #2 FAILED at 145.
> > >>>> 2 out of 2 hunks FAILED -- saving rejects to file
> > >>>>
> > >>>
> > >
>
src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
> > >>>>
> > >>>
> > >
>
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> > >>>> /src/patched/branch-3.2$ h ../patches/
> > >>>>
> > >>>> Could you advise as to which patches I need to apply, and in
what
> > >>> order?
> > >>>>
> > >>>> -Todd
> > >>>>
> > >>>>> -----Original Message-----
> > >>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> > >>>>> Sent: Friday, July 31, 2009 9:51 PM
> > >>>>> To: zookeeper-user@hadoop.apache.org
> > >>>>> Subject: Re: Unending Leader Elections in WAN deploy
> > >>>>>
> > >>>>> Perfect! Thanks for the update, Todd.
> > >>>>>
> > >>>>> -Flavio
> > >>>>>
> > >>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
> > >>>>>
> > >>>>>> Thanks. You were right, I had a stale version of 479.
Compilation
> > >>>>>> succeeds and all tests pass on branch-3.2 with the latest
patches
> > >>>> 473,
> > >>>>>> 479, 481, and 491.
> > >>>>>>
> > >>>>>> -Todd
> > >>>>>>
> > >>>>>>> -----Original Message-----
> > >>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> > >>>>>>> Sent: Friday, July 31, 2009 7:48 PM
> > >>>>>>> To: zookeeper-user@hadoop.apache.org
> > >>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> > >>>>>>>
> > >>>>>>> It should be in 479. Perhaps you have a stale version of the
> > >>> patch.
> > >>>>>>>
> > >>>>>>> -Flavio
> > >>>>>>>
> > >>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
> > >>>>>>>
> > >>>>>>>> Flavio,
> > >>>>>>>>
> > >>>>>>>> I'm getting a compilation error for patch 491:
> > >>>>>>>>
> > >>>>>>>> compile-main:
> > >>>>>>>> [javac] Compiling 1 source file to
> > >>>>>>>>
> > >>>>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> > >>>>>>>> src/p
> > >>>>>>>> atched/branch-3.2/build/classes
> > >>>>>>>> [javac]
> > >>>>>>>>
> > >>>>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> > >>>>>>>> src/p
> > >>>>>>>>
> > >>>>
atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
> > >>>>>>>> FastL
> > >>>>>>>> eaderElection.java:601: cannot find symbol
> > >>>>>>>> [javac] symbol : method getWeight(long)
> > >>>>>>>> [javac] location: interface
> > >>>>>>>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
> > >>>>>>>> [javac]
> > >>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> > >>>>>>>> [javac]
^
> > >>>>>>>> [javac] 1 error
> > >>>>>>>>
> > >>>>>>>> I see a reference to getWeight in both
FastLeaderElection.java
> > >>> in
> > >>>>>>>> patch
> > >>>>>>>> 491:
> > >>>>>>>>
> > >>>>>>>> patches/ZOOKEEPER-491.patch:+
> > >>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> > >>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
> > >>>>>>>> FastLeaderElection.java
> > >>>>>>>> :
> > >>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
> > >>>>>>>> 0)
> > >>>>>>>>
> > >>>>>>>> However, I don't see a reference to this method in patches
473,
> > >>>> 479,
> > >>>>>>>> or
> > >>>>>>>> 481. I also don't see a reference to this method in the
> > > trunk...
> > >>>>>>>>
> > >>>>>>>> -Todd
> > >>>>>>>>
> > >>>>>>>>> -----Original Message-----
> > >>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> > >>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
> > >>>>>>>>> To: zookeeper-user@hadoop.apache.org
> > >>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
> > >>>>>>>>>
> > >>>>>>>>> Ok, I'll apply that patch and report back.
> > >>>>>>>>> -Todd
> > >>>>>>>>>
> > >>>>>>>>>> -----Original Message-----
> > >>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> > >>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
> > >>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> > >>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> > >>>>>>>>>>
> > >>>>>>>>>> You're missing 491 from your set of patches.
> > >>>>>>>>>>
> > >>>>>>>>>> -Flavio
> > >>>>>>>>>>
> > >>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> This repro's in both branch-3.2, and
branch-3.2+patches(473,
> > >>>> 479,
> > >>>>>>>>>>> 481).
> > >>>>>>>>>>>
> > >>>>>>>>>>> Basically, it seems like the nodes are electing
pd4-zook02
> > > to
> > >>>> be
> > >>>>>>>> the
> > >>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
> > >>> supposed
> > >>>> to
> > >>>>>>>> be
> > >>>>>>>>>>> and
> > >>>>>>>>>>> then disconnects everyone. Then they re-elect it again,
and
> > >>> it
> > >>>>>>>> loops
> > >>>>>>>>>>> over and over.
> > >>>>>>>>>>>
> > >>>>>>>>>>> -------------
> > >>>>>>>>>>> Server config
> > >>>>>>>>>>> -------------
> > >>>>>>>>>>>
> > >>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
> > >>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
> > >>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
> > >>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
> > >>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
> > >>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
> > >>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
> > >>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
> > >>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
> > >>>>>>>>>>>
> > >>>>>>>>>>> group.1:1:2:3:4:5
> > >>>>>>>>>>> weight.1=1
> > >>>>>>>>>>> weight.2=1
> > >>>>>>>>>>> weight.3=1
> > >>>>>>>>>>> weight.4=1
> > >>>>>>>>>>> weight.5=1
> > >>>>>>>>>>>
> > >>>>>>>>>>> group.2:6:7:8:9
> > >>>>>>>>>>> weight.6=0
> > >>>>>>>>>>> weight.7=0
> > >>>>>>>>>>> weight.8=0
> > >>>>>>>>>>> weight.9=0
> > >>>>>>>>>>>
> > >>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
> > >>> different
> > >>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
> > > machines
> > >>>> in
> > >>>>>>>> dc1
> > >>>>>>>>>>> have voting rights, and the ability to become a leader.
The
> > >>>>>>>> machines
> > >>>>>>>>>>> in
> > >>>>>>>>>>> the pods all have a weight of zero, and are not expected
to
> > >>>>>> become
> > >>>>>>>>>>> leaders, or to vote on transactions.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Let me know what I can do to help resolve this issue.
> > >>>>>>>>>>>
> > >>>>>>>>>>> -Todd
> > >>>>>>>>
> > >>>>>>
> > >>>
> > >
RE: Unending Leader Elections in WAN deploy
Posted by Todd Greenwood <to...@audiencescience.com>.
Great news. Thank you Mahadev. I'll report our findings later today.
-Todd
> -----Original Message-----
> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> Sent: Tuesday, August 04, 2009 11:20 AM
> To: zookeeper-user@hadoop.apache.org
> Subject: Re: Unending Leader Elections in WAN deploy
>
> Hi Todd,
> I just committed 480 and 491. You can checkout the 3.2 branch now.
>
> Thanks
> mahadev
>
>
> On 8/3/09 4:29 PM, "Todd Greenwood" <to...@audiencescience.com> wrote:
>
> > That'd be perfect. Thanks!
> >
> >> -----Original Message-----
> >> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> >> Sent: Monday, August 03, 2009 4:24 PM
> >> To: zookeeper-user@hadoop.apache.org
> >> Subject: Re: Unending Leader Elections in WAN deploy
> >>
> >> Hi Todd,
> >> Most of the patches that you mention should be in the branch 3.2
by
> > tomm
> >> or so. 481, 479 are already in. 480 and 491 should be in by tomm.
> > Would
> >> that
> >> suffice for you?
> >>
> >> Thanks
> >> mahadev
> >>
> >>
> >> On 8/3/09 4:21 PM, "Todd Greenwood" <to...@audiencescience.com>
wrote:
> >>
> >>> Another problem...I've reverted to the latest versions of the
> > patches
> >>> that are not specific to branch-3.2, and I'm getting two
compilation
> >>> errors:
> >>>
> >>> build-generated:
> >>> [javac] Compiling 44 source files to
> >>>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>> atched/branch-3.2/build/classes
> >>>
> >>> compile-main:
> >>> [javac] Compiling 2 source files to
> >>>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>> atched/branch-3.2/build/classes
> >>> [javac]
> >>>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>
> >
atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> >>> mStats.java:30: name clash: getQuorumPeers() and getQuorumPeers()
> > have
> >>> the same erasure
> >>> [javac] public String[] getQuorumPeers();
> >>> [javac] ^
> >>> [javac]
> >>>
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >>>
> >
atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> >>> mStats.java:31: name clash: getServerState() and getServerState()
> > have
> >>> the same erasure
> >>> [javac] public String getServerState();
> >>> [javac] ^
> >>> [javac] 2 errors
> >>>
> >>> My build process is pretty simple:
> >>>
> >>> 1. copy the branch-3.2 source to a temp directory
> >>> (src/patched/branch-3.2)
> >>> 2. apply the ZOOKEEPER patches in my patches directory
> >>> 3. build zookeeper in the temp directory
> >>>
> >>> -Todd
> >>>> -----Original Message-----
> >>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> >>>> Sent: Monday, August 03, 2009 4:09 PM
> >>>> To: zookeeper-user@hadoop.apache.org
> >>>> Subject: RE: Unending Leader Elections in WAN deploy
> >>>>
> >>>> Flavio,
> >>>> I notice that you've updated the patches referenced for the WAN
> >>>> deployment. There appears to be an order dependency w/ respect to
> >>> these
> >>>> four patches...
> >>>>
> >>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
> >>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
> >>>>
> >>>> 473 -> 479 (479 fails)
> >>>>
> >>>>
> >>>
> >
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> >>>> /src/patched/branch-3.2$ patch -p0 <
> >>>> ../patches/ZOOKEEPER-479-branch3.2.patch
> >>>> patching file
> >>>>
> >>>
> >
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
> >>>> ical.java
> >>>> patching file
> >>>>
> >>>
> >
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
> >>>> patching file
> >>>>
> >>>
> >
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
> >>>> .java
> >>>> patching file
> >>>>
src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
> >>>> Hunk #1 FAILED at 93.
> >>>> Hunk #2 FAILED at 145.
> >>>> 2 out of 2 hunks FAILED -- saving rejects to file
> >>>>
> >>>
> >
src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
> >>>>
> >>>
> >
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> >>>> /src/patched/branch-3.2$ h ../patches/
> >>>>
> >>>> Could you advise as to which patches I need to apply, and in what
> >>> order?
> >>>>
> >>>> -Todd
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>> Sent: Friday, July 31, 2009 9:51 PM
> >>>>> To: zookeeper-user@hadoop.apache.org
> >>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>
> >>>>> Perfect! Thanks for the update, Todd.
> >>>>>
> >>>>> -Flavio
> >>>>>
> >>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
> >>>>>
> >>>>>> Thanks. You were right, I had a stale version of 479.
Compilation
> >>>>>> succeeds and all tests pass on branch-3.2 with the latest
patches
> >>>> 473,
> >>>>>> 479, 481, and 491.
> >>>>>>
> >>>>>> -Todd
> >>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>>>> Sent: Friday, July 31, 2009 7:48 PM
> >>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>
> >>>>>>> It should be in 479. Perhaps you have a stale version of the
> >>> patch.
> >>>>>>>
> >>>>>>> -Flavio
> >>>>>>>
> >>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
> >>>>>>>
> >>>>>>>> Flavio,
> >>>>>>>>
> >>>>>>>> I'm getting a compilation error for patch 491:
> >>>>>>>>
> >>>>>>>> compile-main:
> >>>>>>>> [javac] Compiling 1 source file to
> >>>>>>>>
> >>>>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> >>>>>>>> src/p
> >>>>>>>> atched/branch-3.2/build/classes
> >>>>>>>> [javac]
> >>>>>>>>
> >>>>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> >>>>>>>> src/p
> >>>>>>>>
> >>>>
atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
> >>>>>>>> FastL
> >>>>>>>> eaderElection.java:601: cannot find symbol
> >>>>>>>> [javac] symbol : method getWeight(long)
> >>>>>>>> [javac] location: interface
> >>>>>>>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
> >>>>>>>> [javac]
> >>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> >>>>>>>> [javac]
^
> >>>>>>>> [javac] 1 error
> >>>>>>>>
> >>>>>>>> I see a reference to getWeight in both
FastLeaderElection.java
> >>> in
> >>>>>>>> patch
> >>>>>>>> 491:
> >>>>>>>>
> >>>>>>>> patches/ZOOKEEPER-491.patch:+
> >>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> >>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
> >>>>>>>> FastLeaderElection.java
> >>>>>>>> :
> >>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
> >>>>>>>> 0)
> >>>>>>>>
> >>>>>>>> However, I don't see a reference to this method in patches
473,
> >>>> 479,
> >>>>>>>> or
> >>>>>>>> 481. I also don't see a reference to this method in the
> > trunk...
> >>>>>>>>
> >>>>>>>> -Todd
> >>>>>>>>
> >>>>>>>>> -----Original Message-----
> >>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> >>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
> >>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
> >>>>>>>>>
> >>>>>>>>> Ok, I'll apply that patch and report back.
> >>>>>>>>> -Todd
> >>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
> >>>>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>>>
> >>>>>>>>>> You're missing 491 from your set of patches.
> >>>>>>>>>>
> >>>>>>>>>> -Flavio
> >>>>>>>>>>
> >>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
> >>>>>>>>>>
> >>>>>>>>>>> This repro's in both branch-3.2, and
branch-3.2+patches(473,
> >>>> 479,
> >>>>>>>>>>> 481).
> >>>>>>>>>>>
> >>>>>>>>>>> Basically, it seems like the nodes are electing pd4-zook02
> > to
> >>>> be
> >>>>>>>> the
> >>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
> >>> supposed
> >>>> to
> >>>>>>>> be
> >>>>>>>>>>> and
> >>>>>>>>>>> then disconnects everyone. Then they re-elect it again,
and
> >>> it
> >>>>>>>> loops
> >>>>>>>>>>> over and over.
> >>>>>>>>>>>
> >>>>>>>>>>> -------------
> >>>>>>>>>>> Server config
> >>>>>>>>>>> -------------
> >>>>>>>>>>>
> >>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
> >>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
> >>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
> >>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
> >>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
> >>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
> >>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
> >>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
> >>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
> >>>>>>>>>>>
> >>>>>>>>>>> group.1:1:2:3:4:5
> >>>>>>>>>>> weight.1=1
> >>>>>>>>>>> weight.2=1
> >>>>>>>>>>> weight.3=1
> >>>>>>>>>>> weight.4=1
> >>>>>>>>>>> weight.5=1
> >>>>>>>>>>>
> >>>>>>>>>>> group.2:6:7:8:9
> >>>>>>>>>>> weight.6=0
> >>>>>>>>>>> weight.7=0
> >>>>>>>>>>> weight.8=0
> >>>>>>>>>>> weight.9=0
> >>>>>>>>>>>
> >>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
> >>> different
> >>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
> > machines
> >>>> in
> >>>>>>>> dc1
> >>>>>>>>>>> have voting rights, and the ability to become a leader.
The
> >>>>>>>> machines
> >>>>>>>>>>> in
> >>>>>>>>>>> the pods all have a weight of zero, and are not expected
to
> >>>>>> become
> >>>>>>>>>>> leaders, or to vote on transactions.
> >>>>>>>>>>>
> >>>>>>>>>>> Let me know what I can do to help resolve this issue.
> >>>>>>>>>>>
> >>>>>>>>>>> -Todd
> >>>>>>>>
> >>>>>>
> >>>
> >
Re: Unending Leader Elections in WAN deploy
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi Todd,
I just committed 480 and 491. You can checkout the 3.2 branch now.
Thanks
mahadev
On 8/3/09 4:29 PM, "Todd Greenwood" <to...@audiencescience.com> wrote:
> That'd be perfect. Thanks!
>
>> -----Original Message-----
>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>> Sent: Monday, August 03, 2009 4:24 PM
>> To: zookeeper-user@hadoop.apache.org
>> Subject: Re: Unending Leader Elections in WAN deploy
>>
>> Hi Todd,
>> Most of the patches that you mention should be in the branch 3.2 by
> tomm
>> or so. 481, 479 are already in. 480 and 491 should be in by tomm.
> Would
>> that
>> suffice for you?
>>
>> Thanks
>> mahadev
>>
>>
>> On 8/3/09 4:21 PM, "Todd Greenwood" <to...@audiencescience.com> wrote:
>>
>>> Another problem...I've reverted to the latest versions of the
> patches
>>> that are not specific to branch-3.2, and I'm getting two compilation
>>> errors:
>>>
>>> build-generated:
>>> [javac] Compiling 44 source files to
>>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>> atched/branch-3.2/build/classes
>>>
>>> compile-main:
>>> [javac] Compiling 2 source files to
>>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>> atched/branch-3.2/build/classes
>>> [javac]
>>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>
> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>> mStats.java:30: name clash: getQuorumPeers() and getQuorumPeers()
> have
>>> the same erasure
>>> [javac] public String[] getQuorumPeers();
>>> [javac] ^
>>> [javac]
>>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
>>>
> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
>>> mStats.java:31: name clash: getServerState() and getServerState()
> have
>>> the same erasure
>>> [javac] public String getServerState();
>>> [javac] ^
>>> [javac] 2 errors
>>>
>>> My build process is pretty simple:
>>>
>>> 1. copy the branch-3.2 source to a temp directory
>>> (src/patched/branch-3.2)
>>> 2. apply the ZOOKEEPER patches in my patches directory
>>> 3. build zookeeper in the temp directory
>>>
>>> -Todd
>>>> -----Original Message-----
>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>> Sent: Monday, August 03, 2009 4:09 PM
>>>> To: zookeeper-user@hadoop.apache.org
>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>
>>>> Flavio,
>>>> I notice that you've updated the patches referenced for the WAN
>>>> deployment. There appears to be an order dependency w/ respect to
>>> these
>>>> four patches...
>>>>
>>>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
>>>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
>>>>
>>>> 473 -> 479 (479 fails)
>>>>
>>>>
>>>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>> /src/patched/branch-3.2$ patch -p0 <
>>>> ../patches/ZOOKEEPER-479-branch3.2.patch
>>>> patching file
>>>>
>>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
>>>> ical.java
>>>> patching file
>>>>
>>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
>>>> patching file
>>>>
>>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
>>>> .java
>>>> patching file
>>>> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
>>>> Hunk #1 FAILED at 93.
>>>> Hunk #2 FAILED at 145.
>>>> 2 out of 2 hunks FAILED -- saving rejects to file
>>>>
>>>
> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
>>>>
>>>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>>>> /src/patched/branch-3.2$ h ../patches/
>>>>
>>>> Could you advise as to which patches I need to apply, and in what
>>> order?
>>>>
>>>> -Todd
>>>>
>>>>> -----Original Message-----
>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>> Sent: Friday, July 31, 2009 9:51 PM
>>>>> To: zookeeper-user@hadoop.apache.org
>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>
>>>>> Perfect! Thanks for the update, Todd.
>>>>>
>>>>> -Flavio
>>>>>
>>>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
>>>>>
>>>>>> Thanks. You were right, I had a stale version of 479. Compilation
>>>>>> succeeds and all tests pass on branch-3.2 with the latest patches
>>>> 473,
>>>>>> 479, 481, and 491.
>>>>>>
>>>>>> -Todd
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>> Sent: Friday, July 31, 2009 7:48 PM
>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>
>>>>>>> It should be in 479. Perhaps you have a stale version of the
>>> patch.
>>>>>>>
>>>>>>> -Flavio
>>>>>>>
>>>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
>>>>>>>
>>>>>>>> Flavio,
>>>>>>>>
>>>>>>>> I'm getting a compilation error for patch 491:
>>>>>>>>
>>>>>>>> compile-main:
>>>>>>>> [javac] Compiling 1 source file to
>>>>>>>>
>>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>> src/p
>>>>>>>> atched/branch-3.2/build/classes
>>>>>>>> [javac]
>>>>>>>>
>>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>>>> src/p
>>>>>>>>
>>>> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>> FastL
>>>>>>>> eaderElection.java:601: cannot find symbol
>>>>>>>> [javac] symbol : method getWeight(long)
>>>>>>>> [javac] location: interface
>>>>>>>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
>>>>>>>> [javac]
>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>> [javac] ^
>>>>>>>> [javac] 1 error
>>>>>>>>
>>>>>>>> I see a reference to getWeight in both FastLeaderElection.java
>>> in
>>>>>>>> patch
>>>>>>>> 491:
>>>>>>>>
>>>>>>>> patches/ZOOKEEPER-491.patch:+
>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
>>>>>>>> FastLeaderElection.java
>>>>>>>> :
>>>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
>>>>>>>> 0)
>>>>>>>>
>>>>>>>> However, I don't see a reference to this method in patches 473,
>>>> 479,
>>>>>>>> or
>>>>>>>> 481. I also don't see a reference to this method in the
> trunk...
>>>>>>>>
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>>>
>>>>>>>>> Ok, I'll apply that patch and report back.
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
>>>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>>>
>>>>>>>>>> You're missing 491 from your set of patches.
>>>>>>>>>>
>>>>>>>>>> -Flavio
>>>>>>>>>>
>>>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
>>>>>>>>>>
>>>>>>>>>>> This repro's in both branch-3.2, and branch-3.2+patches(473,
>>>> 479,
>>>>>>>>>>> 481).
>>>>>>>>>>>
>>>>>>>>>>> Basically, it seems like the nodes are electing pd4-zook02
> to
>>>> be
>>>>>>>> the
>>>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
>>> supposed
>>>> to
>>>>>>>> be
>>>>>>>>>>> and
>>>>>>>>>>> then disconnects everyone. Then they re-elect it again, and
>>> it
>>>>>>>> loops
>>>>>>>>>>> over and over.
>>>>>>>>>>>
>>>>>>>>>>> -------------
>>>>>>>>>>> Server config
>>>>>>>>>>> -------------
>>>>>>>>>>>
>>>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
>>>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
>>>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
>>>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
>>>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
>>>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
>>>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
>>>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
>>>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
>>>>>>>>>>>
>>>>>>>>>>> group.1:1:2:3:4:5
>>>>>>>>>>> weight.1=1
>>>>>>>>>>> weight.2=1
>>>>>>>>>>> weight.3=1
>>>>>>>>>>> weight.4=1
>>>>>>>>>>> weight.5=1
>>>>>>>>>>>
>>>>>>>>>>> group.2:6:7:8:9
>>>>>>>>>>> weight.6=0
>>>>>>>>>>> weight.7=0
>>>>>>>>>>> weight.8=0
>>>>>>>>>>> weight.9=0
>>>>>>>>>>>
>>>>>>>>>>> Note that we have 2 groups, composed of machines in 3
>>> different
>>>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
> machines
>>>> in
>>>>>>>> dc1
>>>>>>>>>>> have voting rights, and the ability to become a leader. The
>>>>>>>> machines
>>>>>>>>>>> in
>>>>>>>>>>> the pods all have a weight of zero, and are not expected to
>>>>>> become
>>>>>>>>>>> leaders, or to vote on transactions.
>>>>>>>>>>>
>>>>>>>>>>> Let me know what I can do to help resolve this issue.
>>>>>>>>>>>
>>>>>>>>>>> -Todd
>>>>>>>>
>>>>>>
>>>
>
RE: Unending Leader Elections in WAN deploy
Posted by Todd Greenwood <to...@audiencescience.com>.
That'd be perfect. Thanks!
> -----Original Message-----
> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
> Sent: Monday, August 03, 2009 4:24 PM
> To: zookeeper-user@hadoop.apache.org
> Subject: Re: Unending Leader Elections in WAN deploy
>
> Hi Todd,
> Most of the patches that you mention should be in the branch 3.2 by
tomm
> or so. 481, 479 are already in. 480 and 491 should be in by tomm.
Would
> that
> suffice for you?
>
> Thanks
> mahadev
>
>
> On 8/3/09 4:21 PM, "Todd Greenwood" <to...@audiencescience.com> wrote:
>
> > Another problem...I've reverted to the latest versions of the
patches
> > that are not specific to branch-3.2, and I'm getting two compilation
> > errors:
> >
> > build-generated:
> > [javac] Compiling 44 source files to
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> > atched/branch-3.2/build/classes
> >
> > compile-main:
> > [javac] Compiling 2 source files to
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> > atched/branch-3.2/build/classes
> > [javac]
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >
atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> > mStats.java:30: name clash: getQuorumPeers() and getQuorumPeers()
have
> > the same erasure
> > [javac] public String[] getQuorumPeers();
> > [javac] ^
> > [javac]
> >
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> >
atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> > mStats.java:31: name clash: getServerState() and getServerState()
have
> > the same erasure
> > [javac] public String getServerState();
> > [javac] ^
> > [javac] 2 errors
> >
> > My build process is pretty simple:
> >
> > 1. copy the branch-3.2 source to a temp directory
> > (src/patched/branch-3.2)
> > 2. apply the ZOOKEEPER patches in my patches directory
> > 3. build zookeeper in the temp directory
> >
> > -Todd
> >> -----Original Message-----
> >> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> >> Sent: Monday, August 03, 2009 4:09 PM
> >> To: zookeeper-user@hadoop.apache.org
> >> Subject: RE: Unending Leader Elections in WAN deploy
> >>
> >> Flavio,
> >> I notice that you've updated the patches referenced for the WAN
> >> deployment. There appears to be an order dependency w/ respect to
> > these
> >> four patches...
> >>
> >> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
> >> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
> >>
> >> 473 -> 479 (479 fails)
> >>
> >>
> >
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> >> /src/patched/branch-3.2$ patch -p0 <
> >> ../patches/ZOOKEEPER-479-branch3.2.patch
> >> patching file
> >>
> >
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
> >> ical.java
> >> patching file
> >>
> >
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
> >> patching file
> >>
> >
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
> >> .java
> >> patching file
> >> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
> >> Hunk #1 FAILED at 93.
> >> Hunk #2 FAILED at 145.
> >> 2 out of 2 hunks FAILED -- saving rejects to file
> >>
> >
src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
> >>
> >
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> >> /src/patched/branch-3.2$ h ../patches/
> >>
> >> Could you advise as to which patches I need to apply, and in what
> > order?
> >>
> >> -Todd
> >>
> >>> -----Original Message-----
> >>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>> Sent: Friday, July 31, 2009 9:51 PM
> >>> To: zookeeper-user@hadoop.apache.org
> >>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>
> >>> Perfect! Thanks for the update, Todd.
> >>>
> >>> -Flavio
> >>>
> >>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
> >>>
> >>>> Thanks. You were right, I had a stale version of 479. Compilation
> >>>> succeeds and all tests pass on branch-3.2 with the latest patches
> >> 473,
> >>>> 479, 481, and 491.
> >>>>
> >>>> -Todd
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>> Sent: Friday, July 31, 2009 7:48 PM
> >>>>> To: zookeeper-user@hadoop.apache.org
> >>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>
> >>>>> It should be in 479. Perhaps you have a stale version of the
> > patch.
> >>>>>
> >>>>> -Flavio
> >>>>>
> >>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
> >>>>>
> >>>>>> Flavio,
> >>>>>>
> >>>>>> I'm getting a compilation error for patch 491:
> >>>>>>
> >>>>>> compile-main:
> >>>>>> [javac] Compiling 1 source file to
> >>>>>>
> >> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> >>>>>> src/p
> >>>>>> atched/branch-3.2/build/classes
> >>>>>> [javac]
> >>>>>>
> >> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> >>>>>> src/p
> >>>>>>
> >> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
> >>>>>> FastL
> >>>>>> eaderElection.java:601: cannot find symbol
> >>>>>> [javac] symbol : method getWeight(long)
> >>>>>> [javac] location: interface
> >>>>>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
> >>>>>> [javac]
> >>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> >>>>>> [javac] ^
> >>>>>> [javac] 1 error
> >>>>>>
> >>>>>> I see a reference to getWeight in both FastLeaderElection.java
> > in
> >>>>>> patch
> >>>>>> 491:
> >>>>>>
> >>>>>> patches/ZOOKEEPER-491.patch:+
> >>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> >>>>>> src/java/main/org/apache/zookeeper/server/quorum/
> >>>>>> FastLeaderElection.java
> >>>>>> :
> >>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
> >>>>>> 0)
> >>>>>>
> >>>>>> However, I don't see a reference to this method in patches 473,
> >> 479,
> >>>>>> or
> >>>>>> 481. I also don't see a reference to this method in the
trunk...
> >>>>>>
> >>>>>> -Todd
> >>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> >>>>>>> Sent: Friday, July 31, 2009 7:30 PM
> >>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
> >>>>>>>
> >>>>>>> Ok, I'll apply that patch and report back.
> >>>>>>> -Todd
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
> >>>>>>>> To: zookeeper-user@hadoop.apache.org
> >>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>>>>
> >>>>>>>> You're missing 491 from your set of patches.
> >>>>>>>>
> >>>>>>>> -Flavio
> >>>>>>>>
> >>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
> >>>>>>>>
> >>>>>>>>> This repro's in both branch-3.2, and branch-3.2+patches(473,
> >> 479,
> >>>>>>>>> 481).
> >>>>>>>>>
> >>>>>>>>> Basically, it seems like the nodes are electing pd4-zook02
to
> >> be
> >>>>>> the
> >>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
> > supposed
> >> to
> >>>>>> be
> >>>>>>>>> and
> >>>>>>>>> then disconnects everyone. Then they re-elect it again, and
> > it
> >>>>>> loops
> >>>>>>>>> over and over.
> >>>>>>>>>
> >>>>>>>>> -------------
> >>>>>>>>> Server config
> >>>>>>>>> -------------
> >>>>>>>>>
> >>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
> >>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
> >>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
> >>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
> >>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
> >>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
> >>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
> >>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
> >>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
> >>>>>>>>>
> >>>>>>>>> group.1:1:2:3:4:5
> >>>>>>>>> weight.1=1
> >>>>>>>>> weight.2=1
> >>>>>>>>> weight.3=1
> >>>>>>>>> weight.4=1
> >>>>>>>>> weight.5=1
> >>>>>>>>>
> >>>>>>>>> group.2:6:7:8:9
> >>>>>>>>> weight.6=0
> >>>>>>>>> weight.7=0
> >>>>>>>>> weight.8=0
> >>>>>>>>> weight.9=0
> >>>>>>>>>
> >>>>>>>>> Note that we have 2 groups, composed of machines in 3
> > different
> >>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only
machines
> >> in
> >>>>>> dc1
> >>>>>>>>> have voting rights, and the ability to become a leader. The
> >>>>>> machines
> >>>>>>>>> in
> >>>>>>>>> the pods all have a weight of zero, and are not expected to
> >>>> become
> >>>>>>>>> leaders, or to vote on transactions.
> >>>>>>>>>
> >>>>>>>>> Let me know what I can do to help resolve this issue.
> >>>>>>>>>
> >>>>>>>>> -Todd
> >>>>>>
> >>>>
> >
Re: Unending Leader Elections in WAN deploy
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi Todd,
Most of the patches that you mention should be in the branch 3.2 by tomm
or so. 481, 479 are already in. 480 and 491 should be in by tomm. Would that
suffice for you?
Thanks
mahadev
On 8/3/09 4:21 PM, "Todd Greenwood" <to...@audiencescience.com> wrote:
> Another problem...I've reverted to the latest versions of the patches
> that are not specific to branch-3.2, and I'm getting two compilation
> errors:
>
> build-generated:
> [javac] Compiling 44 source files to
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> atched/branch-3.2/build/classes
>
> compile-main:
> [javac] Compiling 2 source files to
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> atched/branch-3.2/build/classes
> [javac]
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> mStats.java:30: name clash: getQuorumPeers() and getQuorumPeers() have
> the same erasure
> [javac] public String[] getQuorumPeers();
> [javac] ^
> [javac]
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
> mStats.java:31: name clash: getServerState() and getServerState() have
> the same erasure
> [javac] public String getServerState();
> [javac] ^
> [javac] 2 errors
>
> My build process is pretty simple:
>
> 1. copy the branch-3.2 source to a temp directory
> (src/patched/branch-3.2)
> 2. apply the ZOOKEEPER patches in my patches directory
> 3. build zookeeper in the temp directory
>
> -Todd
>> -----Original Message-----
>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>> Sent: Monday, August 03, 2009 4:09 PM
>> To: zookeeper-user@hadoop.apache.org
>> Subject: RE: Unending Leader Elections in WAN deploy
>>
>> Flavio,
>> I notice that you've updated the patches referenced for the WAN
>> deployment. There appears to be an order dependency w/ respect to
> these
>> four patches...
>>
>> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
>> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
>>
>> 473 -> 479 (479 fails)
>>
>>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>> /src/patched/branch-3.2$ patch -p0 <
>> ../patches/ZOOKEEPER-479-branch3.2.patch
>> patching file
>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
>> ical.java
>> patching file
>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
>> patching file
>>
> src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
>> .java
>> patching file
>> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
>> Hunk #1 FAILED at 93.
>> Hunk #2 FAILED at 145.
>> 2 out of 2 hunks FAILED -- saving rejects to file
>>
> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
>>
> toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
>> /src/patched/branch-3.2$ h ../patches/
>>
>> Could you advise as to which patches I need to apply, and in what
> order?
>>
>> -Todd
>>
>>> -----Original Message-----
>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>> Sent: Friday, July 31, 2009 9:51 PM
>>> To: zookeeper-user@hadoop.apache.org
>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>
>>> Perfect! Thanks for the update, Todd.
>>>
>>> -Flavio
>>>
>>> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
>>>
>>>> Thanks. You were right, I had a stale version of 479. Compilation
>>>> succeeds and all tests pass on branch-3.2 with the latest patches
>> 473,
>>>> 479, 481, and 491.
>>>>
>>>> -Todd
>>>>
>>>>> -----Original Message-----
>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>> Sent: Friday, July 31, 2009 7:48 PM
>>>>> To: zookeeper-user@hadoop.apache.org
>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>
>>>>> It should be in 479. Perhaps you have a stale version of the
> patch.
>>>>>
>>>>> -Flavio
>>>>>
>>>>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
>>>>>
>>>>>> Flavio,
>>>>>>
>>>>>> I'm getting a compilation error for patch 491:
>>>>>>
>>>>>> compile-main:
>>>>>> [javac] Compiling 1 source file to
>>>>>>
>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>> src/p
>>>>>> atched/branch-3.2/build/classes
>>>>>> [javac]
>>>>>>
>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>>>>> src/p
>>>>>>
>> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
>>>>>> FastL
>>>>>> eaderElection.java:601: cannot find symbol
>>>>>> [javac] symbol : method getWeight(long)
>>>>>> [javac] location: interface
>>>>>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
>>>>>> [javac]
>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>> [javac] ^
>>>>>> [javac] 1 error
>>>>>>
>>>>>> I see a reference to getWeight in both FastLeaderElection.java
> in
>>>>>> patch
>>>>>> 491:
>>>>>>
>>>>>> patches/ZOOKEEPER-491.patch:+
>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>>>>> src/java/main/org/apache/zookeeper/server/quorum/
>>>>>> FastLeaderElection.java
>>>>>> :
>>>>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
>>>>>> 0)
>>>>>>
>>>>>> However, I don't see a reference to this method in patches 473,
>> 479,
>>>>>> or
>>>>>> 481. I also don't see a reference to this method in the trunk...
>>>>>>
>>>>>> -Todd
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>>>>> Sent: Friday, July 31, 2009 7:30 PM
>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>>>>
>>>>>>> Ok, I'll apply that patch and report back.
>>>>>>> -Todd
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>>>>> Sent: Friday, July 31, 2009 7:18 PM
>>>>>>>> To: zookeeper-user@hadoop.apache.org
>>>>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>>>>
>>>>>>>> You're missing 491 from your set of patches.
>>>>>>>>
>>>>>>>> -Flavio
>>>>>>>>
>>>>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
>>>>>>>>
>>>>>>>>> This repro's in both branch-3.2, and branch-3.2+patches(473,
>> 479,
>>>>>>>>> 481).
>>>>>>>>>
>>>>>>>>> Basically, it seems like the nodes are electing pd4-zook02 to
>> be
>>>>>> the
>>>>>>>>> leader. However, pd4-zook02 seems to realize it's not
> supposed
>> to
>>>>>> be
>>>>>>>>> and
>>>>>>>>> then disconnects everyone. Then they re-elect it again, and
> it
>>>>>> loops
>>>>>>>>> over and over.
>>>>>>>>>
>>>>>>>>> -------------
>>>>>>>>> Server config
>>>>>>>>> -------------
>>>>>>>>>
>>>>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
>>>>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
>>>>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
>>>>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
>>>>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
>>>>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
>>>>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
>>>>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
>>>>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
>>>>>>>>>
>>>>>>>>> group.1:1:2:3:4:5
>>>>>>>>> weight.1=1
>>>>>>>>> weight.2=1
>>>>>>>>> weight.3=1
>>>>>>>>> weight.4=1
>>>>>>>>> weight.5=1
>>>>>>>>>
>>>>>>>>> group.2:6:7:8:9
>>>>>>>>> weight.6=0
>>>>>>>>> weight.7=0
>>>>>>>>> weight.8=0
>>>>>>>>> weight.9=0
>>>>>>>>>
>>>>>>>>> Note that we have 2 groups, composed of machines in 3
> different
>>>>>>>>> locations (dc1, pd1, and pd4). The idea is that only machines
>> in
>>>>>> dc1
>>>>>>>>> have voting rights, and the ability to become a leader. The
>>>>>> machines
>>>>>>>>> in
>>>>>>>>> the pods all have a weight of zero, and are not expected to
>>>> become
>>>>>>>>> leaders, or to vote on transactions.
>>>>>>>>>
>>>>>>>>> Let me know what I can do to help resolve this issue.
>>>>>>>>>
>>>>>>>>> -Todd
>>>>>>
>>>>
>
RE: Unending Leader Elections in WAN deploy
Posted by Todd Greenwood <to...@audiencescience.com>.
Another problem...I've reverted to the latest versions of the patches
that are not specific to branch-3.2, and I'm getting two compilation
errors:
build-generated:
[javac] Compiling 44 source files to
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
atched/branch-3.2/build/classes
compile-main:
[javac] Compiling 2 source files to
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
atched/branch-3.2/build/classes
[javac]
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
mStats.java:30: name clash: getQuorumPeers() and getQuorumPeers() have
the same erasure
[javac] public String[] getQuorumPeers();
[javac] ^
[javac]
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/Quoru
mStats.java:31: name clash: getServerState() and getServerState() have
the same erasure
[javac] public String getServerState();
[javac] ^
[javac] 2 errors
My build process is pretty simple:
1. copy the branch-3.2 source to a temp directory
(src/patched/branch-3.2)
2. apply the ZOOKEEPER patches in my patches directory
3. build zookeeper in the temp directory
-Todd
> -----Original Message-----
> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> Sent: Monday, August 03, 2009 4:09 PM
> To: zookeeper-user@hadoop.apache.org
> Subject: RE: Unending Leader Elections in WAN deploy
>
> Flavio,
> I notice that you've updated the patches referenced for the WAN
> deployment. There appears to be an order dependency w/ respect to
these
> four patches...
>
> ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
> ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
>
> 473 -> 479 (479 fails)
>
>
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> /src/patched/branch-3.2$ patch -p0 <
> ../patches/ZOOKEEPER-479-branch3.2.patch
> patching file
>
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
> ical.java
> patching file
>
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
> patching file
>
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
> .java
> patching file
> src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
> Hunk #1 FAILED at 93.
> Hunk #2 FAILED at 145.
> 2 out of 2 hunks FAILED -- saving rejects to file
>
src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
>
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
> /src/patched/branch-3.2$ h ../patches/
>
> Could you advise as to which patches I need to apply, and in what
order?
>
> -Todd
>
> > -----Original Message-----
> > From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> > Sent: Friday, July 31, 2009 9:51 PM
> > To: zookeeper-user@hadoop.apache.org
> > Subject: Re: Unending Leader Elections in WAN deploy
> >
> > Perfect! Thanks for the update, Todd.
> >
> > -Flavio
> >
> > On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
> >
> > > Thanks. You were right, I had a stale version of 479. Compilation
> > > succeeds and all tests pass on branch-3.2 with the latest patches
> 473,
> > > 479, 481, and 491.
> > >
> > > -Todd
> > >
> > >> -----Original Message-----
> > >> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> > >> Sent: Friday, July 31, 2009 7:48 PM
> > >> To: zookeeper-user@hadoop.apache.org
> > >> Subject: Re: Unending Leader Elections in WAN deploy
> > >>
> > >> It should be in 479. Perhaps you have a stale version of the
patch.
> > >>
> > >> -Flavio
> > >>
> > >> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
> > >>
> > >>> Flavio,
> > >>>
> > >>> I'm getting a compilation error for patch 491:
> > >>>
> > >>> compile-main:
> > >>> [javac] Compiling 1 source file to
> > >>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> > >>> src/p
> > >>> atched/branch-3.2/build/classes
> > >>> [javac]
> > >>>
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> > >>> src/p
> > >>>
> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
> > >>> FastL
> > >>> eaderElection.java:601: cannot find symbol
> > >>> [javac] symbol : method getWeight(long)
> > >>> [javac] location: interface
> > >>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
> > >>> [javac]
> > >>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> > >>> [javac] ^
> > >>> [javac] 1 error
> > >>>
> > >>> I see a reference to getWeight in both FastLeaderElection.java
in
> > >>> patch
> > >>> 491:
> > >>>
> > >>> patches/ZOOKEEPER-491.patch:+
> > >>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> > >>> src/java/main/org/apache/zookeeper/server/quorum/
> > >>> FastLeaderElection.java
> > >>> :
> > >>> if(self.getQuorumVerifier().getWeight(n.sid) !=
> > >>> 0)
> > >>>
> > >>> However, I don't see a reference to this method in patches 473,
> 479,
> > >>> or
> > >>> 481. I also don't see a reference to this method in the trunk...
> > >>>
> > >>> -Todd
> > >>>
> > >>>> -----Original Message-----
> > >>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> > >>>> Sent: Friday, July 31, 2009 7:30 PM
> > >>>> To: zookeeper-user@hadoop.apache.org
> > >>>> Subject: RE: Unending Leader Elections in WAN deploy
> > >>>>
> > >>>> Ok, I'll apply that patch and report back.
> > >>>> -Todd
> > >>>>
> > >>>>> -----Original Message-----
> > >>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> > >>>>> Sent: Friday, July 31, 2009 7:18 PM
> > >>>>> To: zookeeper-user@hadoop.apache.org
> > >>>>> Subject: Re: Unending Leader Elections in WAN deploy
> > >>>>>
> > >>>>> You're missing 491 from your set of patches.
> > >>>>>
> > >>>>> -Flavio
> > >>>>>
> > >>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
> > >>>>>
> > >>>>>> This repro's in both branch-3.2, and branch-3.2+patches(473,
> 479,
> > >>>>>> 481).
> > >>>>>>
> > >>>>>> Basically, it seems like the nodes are electing pd4-zook02 to
> be
> > >>> the
> > >>>>>> leader. However, pd4-zook02 seems to realize it's not
supposed
> to
> > >>> be
> > >>>>>> and
> > >>>>>> then disconnects everyone. Then they re-elect it again, and
it
> > >>> loops
> > >>>>>> over and over.
> > >>>>>>
> > >>>>>> -------------
> > >>>>>> Server config
> > >>>>>> -------------
> > >>>>>>
> > >>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
> > >>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
> > >>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
> > >>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
> > >>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
> > >>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
> > >>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
> > >>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
> > >>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
> > >>>>>>
> > >>>>>> group.1:1:2:3:4:5
> > >>>>>> weight.1=1
> > >>>>>> weight.2=1
> > >>>>>> weight.3=1
> > >>>>>> weight.4=1
> > >>>>>> weight.5=1
> > >>>>>>
> > >>>>>> group.2:6:7:8:9
> > >>>>>> weight.6=0
> > >>>>>> weight.7=0
> > >>>>>> weight.8=0
> > >>>>>> weight.9=0
> > >>>>>>
> > >>>>>> Note that we have 2 groups, composed of machines in 3
different
> > >>>>>> locations (dc1, pd1, and pd4). The idea is that only machines
> in
> > >>> dc1
> > >>>>>> have voting rights, and the ability to become a leader. The
> > >>> machines
> > >>>>>> in
> > >>>>>> the pods all have a weight of zero, and are not expected to
> > > become
> > >>>>>> leaders, or to vote on transactions.
> > >>>>>>
> > >>>>>> Let me know what I can do to help resolve this issue.
> > >>>>>>
> > >>>>>> -Todd
> > >>>
> > >
RE: Unending Leader Elections in WAN deploy
Posted by Todd Greenwood <to...@audiencescience.com>.
Flavio,
I notice that you've updated the patches referenced for the WAN
deployment. There appears to be an order dependency w/ respect to these
four patches...
ZOOKEEPER-473.patch ZOOKEEPER-479-branch3.2.patch
ZOOKEEPER-481-branch3.2.patch ZOOKEEPER-491.patch
473 -> 479 (479 fails)
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
/src/patched/branch-3.2$ patch -p0 <
../patches/ZOOKEEPER-479-branch3.2.patch
patching file
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarch
ical.java
patching file
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumMaj.java
patching file
src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumVerifier
.java
patching file
src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
Hunk #1 FAILED at 93.
Hunk #2 FAILED at 145.
2 out of 2 hunks FAILED -- saving rejects to file
src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java.rej
toddg@TODDG01LT:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper
/src/patched/branch-3.2$ h ../patches/
Could you advise as to which patches I need to apply, and in what order?
-Todd
> -----Original Message-----
> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> Sent: Friday, July 31, 2009 9:51 PM
> To: zookeeper-user@hadoop.apache.org
> Subject: Re: Unending Leader Elections in WAN deploy
>
> Perfect! Thanks for the update, Todd.
>
> -Flavio
>
> On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
>
> > Thanks. You were right, I had a stale version of 479. Compilation
> > succeeds and all tests pass on branch-3.2 with the latest patches
473,
> > 479, 481, and 491.
> >
> > -Todd
> >
> >> -----Original Message-----
> >> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >> Sent: Friday, July 31, 2009 7:48 PM
> >> To: zookeeper-user@hadoop.apache.org
> >> Subject: Re: Unending Leader Elections in WAN deploy
> >>
> >> It should be in 479. Perhaps you have a stale version of the patch.
> >>
> >> -Flavio
> >>
> >> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
> >>
> >>> Flavio,
> >>>
> >>> I'm getting a compilation error for patch 491:
> >>>
> >>> compile-main:
> >>> [javac] Compiling 1 source file to
> >>>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> >>> src/p
> >>> atched/branch-3.2/build/classes
> >>> [javac]
> >>>
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> >>> src/p
> >>>
atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
> >>> FastL
> >>> eaderElection.java:601: cannot find symbol
> >>> [javac] symbol : method getWeight(long)
> >>> [javac] location: interface
> >>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
> >>> [javac]
> >>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> >>> [javac] ^
> >>> [javac] 1 error
> >>>
> >>> I see a reference to getWeight in both FastLeaderElection.java in
> >>> patch
> >>> 491:
> >>>
> >>> patches/ZOOKEEPER-491.patch:+
> >>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> >>> src/java/main/org/apache/zookeeper/server/quorum/
> >>> FastLeaderElection.java
> >>> :
> >>> if(self.getQuorumVerifier().getWeight(n.sid) !=
> >>> 0)
> >>>
> >>> However, I don't see a reference to this method in patches 473,
479,
> >>> or
> >>> 481. I also don't see a reference to this method in the trunk...
> >>>
> >>> -Todd
> >>>
> >>>> -----Original Message-----
> >>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> >>>> Sent: Friday, July 31, 2009 7:30 PM
> >>>> To: zookeeper-user@hadoop.apache.org
> >>>> Subject: RE: Unending Leader Elections in WAN deploy
> >>>>
> >>>> Ok, I'll apply that patch and report back.
> >>>> -Todd
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>>>> Sent: Friday, July 31, 2009 7:18 PM
> >>>>> To: zookeeper-user@hadoop.apache.org
> >>>>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>>>
> >>>>> You're missing 491 from your set of patches.
> >>>>>
> >>>>> -Flavio
> >>>>>
> >>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
> >>>>>
> >>>>>> This repro's in both branch-3.2, and branch-3.2+patches(473,
479,
> >>>>>> 481).
> >>>>>>
> >>>>>> Basically, it seems like the nodes are electing pd4-zook02 to
be
> >>> the
> >>>>>> leader. However, pd4-zook02 seems to realize it's not supposed
to
> >>> be
> >>>>>> and
> >>>>>> then disconnects everyone. Then they re-elect it again, and it
> >>> loops
> >>>>>> over and over.
> >>>>>>
> >>>>>> -------------
> >>>>>> Server config
> >>>>>> -------------
> >>>>>>
> >>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
> >>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
> >>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
> >>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
> >>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
> >>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
> >>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
> >>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
> >>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
> >>>>>>
> >>>>>> group.1:1:2:3:4:5
> >>>>>> weight.1=1
> >>>>>> weight.2=1
> >>>>>> weight.3=1
> >>>>>> weight.4=1
> >>>>>> weight.5=1
> >>>>>>
> >>>>>> group.2:6:7:8:9
> >>>>>> weight.6=0
> >>>>>> weight.7=0
> >>>>>> weight.8=0
> >>>>>> weight.9=0
> >>>>>>
> >>>>>> Note that we have 2 groups, composed of machines in 3 different
> >>>>>> locations (dc1, pd1, and pd4). The idea is that only machines
in
> >>> dc1
> >>>>>> have voting rights, and the ability to become a leader. The
> >>> machines
> >>>>>> in
> >>>>>> the pods all have a weight of zero, and are not expected to
> > become
> >>>>>> leaders, or to vote on transactions.
> >>>>>>
> >>>>>> Let me know what I can do to help resolve this issue.
> >>>>>>
> >>>>>> -Todd
> >>>
> >
Re: Unending Leader Elections in WAN deploy
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
Perfect! Thanks for the update, Todd.
-Flavio
On Jul 31, 2009, at 8:17 PM, Todd Greenwood wrote:
> Thanks. You were right, I had a stale version of 479. Compilation
> succeeds and all tests pass on branch-3.2 with the latest patches 473,
> 479, 481, and 491.
>
> -Todd
>
>> -----Original Message-----
>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>> Sent: Friday, July 31, 2009 7:48 PM
>> To: zookeeper-user@hadoop.apache.org
>> Subject: Re: Unending Leader Elections in WAN deploy
>>
>> It should be in 479. Perhaps you have a stale version of the patch.
>>
>> -Flavio
>>
>> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
>>
>>> Flavio,
>>>
>>> I'm getting a compilation error for patch 491:
>>>
>>> compile-main:
>>> [javac] Compiling 1 source file to
>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>> src/p
>>> atched/branch-3.2/build/classes
>>> [javac]
>>> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
>>> src/p
>>> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
>>> FastL
>>> eaderElection.java:601: cannot find symbol
>>> [javac] symbol : method getWeight(long)
>>> [javac] location: interface
>>> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
>>> [javac]
>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>> [javac] ^
>>> [javac] 1 error
>>>
>>> I see a reference to getWeight in both FastLeaderElection.java in
>>> patch
>>> 491:
>>>
>>> patches/ZOOKEEPER-491.patch:+
>>> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
>>> src/java/main/org/apache/zookeeper/server/quorum/
>>> FastLeaderElection.java
>>> :
>>> if(self.getQuorumVerifier().getWeight(n.sid) !=
>>> 0)
>>>
>>> However, I don't see a reference to this method in patches 473, 479,
>>> or
>>> 481. I also don't see a reference to this method in the trunk...
>>>
>>> -Todd
>>>
>>>> -----Original Message-----
>>>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>>>> Sent: Friday, July 31, 2009 7:30 PM
>>>> To: zookeeper-user@hadoop.apache.org
>>>> Subject: RE: Unending Leader Elections in WAN deploy
>>>>
>>>> Ok, I'll apply that patch and report back.
>>>> -Todd
>>>>
>>>>> -----Original Message-----
>>>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>>>> Sent: Friday, July 31, 2009 7:18 PM
>>>>> To: zookeeper-user@hadoop.apache.org
>>>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>>>
>>>>> You're missing 491 from your set of patches.
>>>>>
>>>>> -Flavio
>>>>>
>>>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
>>>>>
>>>>>> This repro's in both branch-3.2, and branch-3.2+patches(473, 479,
>>>>>> 481).
>>>>>>
>>>>>> Basically, it seems like the nodes are electing pd4-zook02 to be
>>> the
>>>>>> leader. However, pd4-zook02 seems to realize it's not supposed to
>>> be
>>>>>> and
>>>>>> then disconnects everyone. Then they re-elect it again, and it
>>> loops
>>>>>> over and over.
>>>>>>
>>>>>> -------------
>>>>>> Server config
>>>>>> -------------
>>>>>>
>>>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
>>>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
>>>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
>>>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
>>>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
>>>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
>>>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
>>>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
>>>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
>>>>>>
>>>>>> group.1:1:2:3:4:5
>>>>>> weight.1=1
>>>>>> weight.2=1
>>>>>> weight.3=1
>>>>>> weight.4=1
>>>>>> weight.5=1
>>>>>>
>>>>>> group.2:6:7:8:9
>>>>>> weight.6=0
>>>>>> weight.7=0
>>>>>> weight.8=0
>>>>>> weight.9=0
>>>>>>
>>>>>> Note that we have 2 groups, composed of machines in 3 different
>>>>>> locations (dc1, pd1, and pd4). The idea is that only machines in
>>> dc1
>>>>>> have voting rights, and the ability to become a leader. The
>>> machines
>>>>>> in
>>>>>> the pods all have a weight of zero, and are not expected to
> become
>>>>>> leaders, or to vote on transactions.
>>>>>>
>>>>>> Let me know what I can do to help resolve this issue.
>>>>>>
>>>>>> -Todd
>>>
>
RE: Unending Leader Elections in WAN deploy
Posted by Todd Greenwood <to...@audiencescience.com>.
Thanks. You were right, I had a stale version of 479. Compilation
succeeds and all tests pass on branch-3.2 with the latest patches 473,
479, 481, and 491.
-Todd
> -----Original Message-----
> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> Sent: Friday, July 31, 2009 7:48 PM
> To: zookeeper-user@hadoop.apache.org
> Subject: Re: Unending Leader Elections in WAN deploy
>
> It should be in 479. Perhaps you have a stale version of the patch.
>
> -Flavio
>
> On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
>
> > Flavio,
> >
> > I'm getting a compilation error for patch 491:
> >
> > compile-main:
> > [javac] Compiling 1 source file to
> > /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> > src/p
> > atched/branch-3.2/build/classes
> > [javac]
> > /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> > src/p
> > atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
> > FastL
> > eaderElection.java:601: cannot find symbol
> > [javac] symbol : method getWeight(long)
> > [javac] location: interface
> > org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
> > [javac]
> > if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> > [javac] ^
> > [javac] 1 error
> >
> > I see a reference to getWeight in both FastLeaderElection.java in
> > patch
> > 491:
> >
> > patches/ZOOKEEPER-491.patch:+
> > if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> > src/java/main/org/apache/zookeeper/server/quorum/
> > FastLeaderElection.java
> > :
> > if(self.getQuorumVerifier().getWeight(n.sid) !=
> > 0)
> >
> > However, I don't see a reference to this method in patches 473, 479,
> > or
> > 481. I also don't see a reference to this method in the trunk...
> >
> > -Todd
> >
> >> -----Original Message-----
> >> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> >> Sent: Friday, July 31, 2009 7:30 PM
> >> To: zookeeper-user@hadoop.apache.org
> >> Subject: RE: Unending Leader Elections in WAN deploy
> >>
> >> Ok, I'll apply that patch and report back.
> >> -Todd
> >>
> >>> -----Original Message-----
> >>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> >>> Sent: Friday, July 31, 2009 7:18 PM
> >>> To: zookeeper-user@hadoop.apache.org
> >>> Subject: Re: Unending Leader Elections in WAN deploy
> >>>
> >>> You're missing 491 from your set of patches.
> >>>
> >>> -Flavio
> >>>
> >>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
> >>>
> >>>> This repro's in both branch-3.2, and branch-3.2+patches(473, 479,
> >>>> 481).
> >>>>
> >>>> Basically, it seems like the nodes are electing pd4-zook02 to be
> > the
> >>>> leader. However, pd4-zook02 seems to realize it's not supposed to
> > be
> >>>> and
> >>>> then disconnects everyone. Then they re-elect it again, and it
> > loops
> >>>> over and over.
> >>>>
> >>>> -------------
> >>>> Server config
> >>>> -------------
> >>>>
> >>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
> >>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
> >>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
> >>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
> >>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
> >>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
> >>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
> >>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
> >>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
> >>>>
> >>>> group.1:1:2:3:4:5
> >>>> weight.1=1
> >>>> weight.2=1
> >>>> weight.3=1
> >>>> weight.4=1
> >>>> weight.5=1
> >>>>
> >>>> group.2:6:7:8:9
> >>>> weight.6=0
> >>>> weight.7=0
> >>>> weight.8=0
> >>>> weight.9=0
> >>>>
> >>>> Note that we have 2 groups, composed of machines in 3 different
> >>>> locations (dc1, pd1, and pd4). The idea is that only machines in
> > dc1
> >>>> have voting rights, and the ability to become a leader. The
> > machines
> >>>> in
> >>>> the pods all have a weight of zero, and are not expected to
become
> >>>> leaders, or to vote on transactions.
> >>>>
> >>>> Let me know what I can do to help resolve this issue.
> >>>>
> >>>> -Todd
> >
Re: Unending Leader Elections in WAN deploy
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
It should be in 479. Perhaps you have a stale version of the patch.
-Flavio
On Jul 31, 2009, at 7:46 PM, Todd Greenwood wrote:
> Flavio,
>
> I'm getting a compilation error for patch 491:
>
> compile-main:
> [javac] Compiling 1 source file to
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> src/p
> atched/branch-3.2/build/classes
> [javac]
> /home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/
> src/p
> atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/
> FastL
> eaderElection.java:601: cannot find symbol
> [javac] symbol : method getWeight(long)
> [javac] location: interface
> org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
> [javac]
> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> [javac] ^
> [javac] 1 error
>
> I see a reference to getWeight in both FastLeaderElection.java in
> patch
> 491:
>
> patches/ZOOKEEPER-491.patch:+
> if(self.getQuorumVerifier().getWeight(n.sid) != 0)
> src/java/main/org/apache/zookeeper/server/quorum/
> FastLeaderElection.java
> :
> if(self.getQuorumVerifier().getWeight(n.sid) !=
> 0)
>
> However, I don't see a reference to this method in patches 473, 479,
> or
> 481. I also don't see a reference to this method in the trunk...
>
> -Todd
>
>> -----Original Message-----
>> From: Todd Greenwood [mailto:toddg@audiencescience.com]
>> Sent: Friday, July 31, 2009 7:30 PM
>> To: zookeeper-user@hadoop.apache.org
>> Subject: RE: Unending Leader Elections in WAN deploy
>>
>> Ok, I'll apply that patch and report back.
>> -Todd
>>
>>> -----Original Message-----
>>> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
>>> Sent: Friday, July 31, 2009 7:18 PM
>>> To: zookeeper-user@hadoop.apache.org
>>> Subject: Re: Unending Leader Elections in WAN deploy
>>>
>>> You're missing 491 from your set of patches.
>>>
>>> -Flavio
>>>
>>> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
>>>
>>>> This repro's in both branch-3.2, and branch-3.2+patches(473, 479,
>>>> 481).
>>>>
>>>> Basically, it seems like the nodes are electing pd4-zook02 to be
> the
>>>> leader. However, pd4-zook02 seems to realize it's not supposed to
> be
>>>> and
>>>> then disconnects everyone. Then they re-elect it again, and it
> loops
>>>> over and over.
>>>>
>>>> -------------
>>>> Server config
>>>> -------------
>>>>
>>>> server.1=dc1-zook01.dc01.revsci.net:2888:3888
>>>> server.2=dc1-zook02.dc01.revsci.net:2888:3888
>>>> server.3=dc1-zook03.dc01.revsci.net:2888:3888
>>>> server.4=dc1-zook04.dc01.revsci.net:2888:3888
>>>> server.5=dc1-zook05.dc01.revsci.net:2888:3888
>>>> server.6=pd1-zook01.pd01.revsci.net:2888:3888
>>>> server.7=pd1-zook02.pd01.revsci.net:2888:3888
>>>> server.8=pd4-zook01.iad1.audsci.net:2888:3888
>>>> server.9=pd4-zook02.iad1.audsci.net:2888:3888
>>>>
>>>> group.1:1:2:3:4:5
>>>> weight.1=1
>>>> weight.2=1
>>>> weight.3=1
>>>> weight.4=1
>>>> weight.5=1
>>>>
>>>> group.2:6:7:8:9
>>>> weight.6=0
>>>> weight.7=0
>>>> weight.8=0
>>>> weight.9=0
>>>>
>>>> Note that we have 2 groups, composed of machines in 3 different
>>>> locations (dc1, pd1, and pd4). The idea is that only machines in
> dc1
>>>> have voting rights, and the ability to become a leader. The
> machines
>>>> in
>>>> the pods all have a weight of zero, and are not expected to become
>>>> leaders, or to vote on transactions.
>>>>
>>>> Let me know what I can do to help resolve this issue.
>>>>
>>>> -Todd
>
RE: Unending Leader Elections in WAN deploy
Posted by Todd Greenwood <to...@audiencescience.com>.
Flavio,
I'm getting a compilation error for patch 491:
compile-main:
[javac] Compiling 1 source file to
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
atched/branch-3.2/build/classes
[javac]
/home/toddg/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper/src/p
atched/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/FastL
eaderElection.java:601: cannot find symbol
[javac] symbol : method getWeight(long)
[javac] location: interface
org.apache.zookeeper.server.quorum.flexible.QuorumVerifier
[javac]
if(self.getQuorumVerifier().getWeight(n.sid) != 0)
[javac] ^
[javac] 1 error
I see a reference to getWeight in both FastLeaderElection.java in patch
491:
patches/ZOOKEEPER-491.patch:+
if(self.getQuorumVerifier().getWeight(n.sid) != 0)
src/java/main/org/apache/zookeeper/server/quorum/FastLeaderElection.java
: if(self.getQuorumVerifier().getWeight(n.sid) !=
0)
However, I don't see a reference to this method in patches 473, 479, or
481. I also don't see a reference to this method in the trunk...
-Todd
> -----Original Message-----
> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> Sent: Friday, July 31, 2009 7:30 PM
> To: zookeeper-user@hadoop.apache.org
> Subject: RE: Unending Leader Elections in WAN deploy
>
> Ok, I'll apply that patch and report back.
> -Todd
>
> > -----Original Message-----
> > From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> > Sent: Friday, July 31, 2009 7:18 PM
> > To: zookeeper-user@hadoop.apache.org
> > Subject: Re: Unending Leader Elections in WAN deploy
> >
> > You're missing 491 from your set of patches.
> >
> > -Flavio
> >
> > On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
> >
> > > This repro's in both branch-3.2, and branch-3.2+patches(473, 479,
> > > 481).
> > >
> > > Basically, it seems like the nodes are electing pd4-zook02 to be
the
> > > leader. However, pd4-zook02 seems to realize it's not supposed to
be
> > > and
> > > then disconnects everyone. Then they re-elect it again, and it
loops
> > > over and over.
> > >
> > > -------------
> > > Server config
> > > -------------
> > >
> > > server.1=dc1-zook01.dc01.revsci.net:2888:3888
> > > server.2=dc1-zook02.dc01.revsci.net:2888:3888
> > > server.3=dc1-zook03.dc01.revsci.net:2888:3888
> > > server.4=dc1-zook04.dc01.revsci.net:2888:3888
> > > server.5=dc1-zook05.dc01.revsci.net:2888:3888
> > > server.6=pd1-zook01.pd01.revsci.net:2888:3888
> > > server.7=pd1-zook02.pd01.revsci.net:2888:3888
> > > server.8=pd4-zook01.iad1.audsci.net:2888:3888
> > > server.9=pd4-zook02.iad1.audsci.net:2888:3888
> > >
> > > group.1:1:2:3:4:5
> > > weight.1=1
> > > weight.2=1
> > > weight.3=1
> > > weight.4=1
> > > weight.5=1
> > >
> > > group.2:6:7:8:9
> > > weight.6=0
> > > weight.7=0
> > > weight.8=0
> > > weight.9=0
> > >
> > > Note that we have 2 groups, composed of machines in 3 different
> > > locations (dc1, pd1, and pd4). The idea is that only machines in
dc1
> > > have voting rights, and the ability to become a leader. The
machines
> > > in
> > > the pods all have a weight of zero, and are not expected to become
> > > leaders, or to vote on transactions.
> > >
> > > Let me know what I can do to help resolve this issue.
> > >
> > > -Todd
RE: Unending Leader Elections in WAN deploy
Posted by Todd Greenwood <to...@audiencescience.com>.
Ok, I'll apply that patch and report back.
-Todd
> -----Original Message-----
> From: Flavio Junqueira [mailto:fpj@yahoo-inc.com]
> Sent: Friday, July 31, 2009 7:18 PM
> To: zookeeper-user@hadoop.apache.org
> Subject: Re: Unending Leader Elections in WAN deploy
>
> You're missing 491 from your set of patches.
>
> -Flavio
>
> On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
>
> > This repro's in both branch-3.2, and branch-3.2+patches(473, 479,
> > 481).
> >
> > Basically, it seems like the nodes are electing pd4-zook02 to be the
> > leader. However, pd4-zook02 seems to realize it's not supposed to be
> > and
> > then disconnects everyone. Then they re-elect it again, and it loops
> > over and over.
> >
> > -------------
> > Server config
> > -------------
> >
> > server.1=dc1-zook01.dc01.revsci.net:2888:3888
> > server.2=dc1-zook02.dc01.revsci.net:2888:3888
> > server.3=dc1-zook03.dc01.revsci.net:2888:3888
> > server.4=dc1-zook04.dc01.revsci.net:2888:3888
> > server.5=dc1-zook05.dc01.revsci.net:2888:3888
> > server.6=pd1-zook01.pd01.revsci.net:2888:3888
> > server.7=pd1-zook02.pd01.revsci.net:2888:3888
> > server.8=pd4-zook01.iad1.audsci.net:2888:3888
> > server.9=pd4-zook02.iad1.audsci.net:2888:3888
> >
> > group.1:1:2:3:4:5
> > weight.1=1
> > weight.2=1
> > weight.3=1
> > weight.4=1
> > weight.5=1
> >
> > group.2:6:7:8:9
> > weight.6=0
> > weight.7=0
> > weight.8=0
> > weight.9=0
> >
> > Note that we have 2 groups, composed of machines in 3 different
> > locations (dc1, pd1, and pd4). The idea is that only machines in dc1
> > have voting rights, and the ability to become a leader. The machines
> > in
> > the pods all have a weight of zero, and are not expected to become
> > leaders, or to vote on transactions.
> >
> > Let me know what I can do to help resolve this issue.
> >
> > -Todd
Re: Unending Leader Elections in WAN deploy
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
You're missing 491 from your set of patches.
-Flavio
On Jul 31, 2009, at 7:15 PM, Todd Greenwood wrote:
> This repro's in both branch-3.2, and branch-3.2+patches(473, 479,
> 481).
>
> Basically, it seems like the nodes are electing pd4-zook02 to be the
> leader. However, pd4-zook02 seems to realize it's not supposed to be
> and
> then disconnects everyone. Then they re-elect it again, and it loops
> over and over.
>
> -------------
> Server config
> -------------
>
> server.1=dc1-zook01.dc01.revsci.net:2888:3888
> server.2=dc1-zook02.dc01.revsci.net:2888:3888
> server.3=dc1-zook03.dc01.revsci.net:2888:3888
> server.4=dc1-zook04.dc01.revsci.net:2888:3888
> server.5=dc1-zook05.dc01.revsci.net:2888:3888
> server.6=pd1-zook01.pd01.revsci.net:2888:3888
> server.7=pd1-zook02.pd01.revsci.net:2888:3888
> server.8=pd4-zook01.iad1.audsci.net:2888:3888
> server.9=pd4-zook02.iad1.audsci.net:2888:3888
>
> group.1:1:2:3:4:5
> weight.1=1
> weight.2=1
> weight.3=1
> weight.4=1
> weight.5=1
>
> group.2:6:7:8:9
> weight.6=0
> weight.7=0
> weight.8=0
> weight.9=0
>
> Note that we have 2 groups, composed of machines in 3 different
> locations (dc1, pd1, and pd4). The idea is that only machines in dc1
> have voting rights, and the ability to become a leader. The machines
> in
> the pods all have a weight of zero, and are not expected to become
> leaders, or to vote on transactions.
>
> Let me know what I can do to help resolve this issue.
>
> -Todd
RE: Unending Leader Elections in WAN deploy
Posted by Todd Greenwood <to...@audiencescience.com>.
Some how the logs did not attach. Zookeeper logs should be attached.
> -----Original Message-----
> From: Todd Greenwood [mailto:toddg@audiencescience.com]
> Sent: Friday, July 31, 2009 7:15 PM
> To: zookeeper-user@hadoop.apache.org
> Subject: Unending Leader Elections in WAN deploy
>
> This repro's in both branch-3.2, and branch-3.2+patches(473, 479,
481).
>
> Basically, it seems like the nodes are electing pd4-zook02 to be the
> leader. However, pd4-zook02 seems to realize it's not supposed to be
and
> then disconnects everyone. Then they re-elect it again, and it loops
> over and over.
>
> -------------
> Server config
> -------------
>
> server.1=dc1-zook01.dc01.revsci.net:2888:3888
> server.2=dc1-zook02.dc01.revsci.net:2888:3888
> server.3=dc1-zook03.dc01.revsci.net:2888:3888
> server.4=dc1-zook04.dc01.revsci.net:2888:3888
> server.5=dc1-zook05.dc01.revsci.net:2888:3888
> server.6=pd1-zook01.pd01.revsci.net:2888:3888
> server.7=pd1-zook02.pd01.revsci.net:2888:3888
> server.8=pd4-zook01.iad1.audsci.net:2888:3888
> server.9=pd4-zook02.iad1.audsci.net:2888:3888
>
> group.1:1:2:3:4:5
> weight.1=1
> weight.2=1
> weight.3=1
> weight.4=1
> weight.5=1
>
> group.2:6:7:8:9
> weight.6=0
> weight.7=0
> weight.8=0
> weight.9=0
>
> Note that we have 2 groups, composed of machines in 3 different
> locations (dc1, pd1, and pd4). The idea is that only machines in dc1
> have voting rights, and the ability to become a leader. The machines
in
> the pods all have a weight of zero, and are not expected to become
> leaders, or to vote on transactions.
>
> Let me know what I can do to help resolve this issue.
>
> -Todd