You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Adrian Liew <ad...@avanade.com> on 2015/10/07 06:22:06 UTC
If zookeeper is down, SolrCloud nodes will not start correctly, even
if zookeeper is started later
Changing subject header.
I am encountering this issue in Solr 5.3.0 whereby I am getting haywired leader election using SolrCloud. I am using NSSM 2.24 to startup my solr services with zookeeper set as a dependency.
For example, if I have three servers marked as 10.0.0.4, 10.0.0.5 and 10.0.0.6, both 10.0.0.4 and 10.0.0.5 show up as Leaders in Solr Admin Panel.
I then decided to manually stop all services, and ensure all Zookeeper services are booted up first prior to starting Solr services on all machines. Then I refreshed my Solr Admin Panel to observe the correct leaders and followers and test node recovery. Everything turned out fine.
Hence, the issue is that upon startup of three machines, the startup of ZK and Solr is out of sequence that causes SolrCloud to behave unexpectedly. Noting there is Jira ticket addressed here for Solr 4.9 above to include an improvement to the issue above. (https://issues.apache.org/jira/browse/SOLR-5129)
Can someone please advise.
Best regards,
Adrian
-----Original Message-----
From: Alessandro Benedetti [mailto:benedetti.alex85@gmail.com]
Sent: Tuesday, October 6, 2015 7:14 PM
To: solr-user@lucene.apache.org
Subject: Re: Zookeeper HA with 3x ZK with Solr server nodes
When you have a ZK Ensemble a quorum of active nodes is necessary to have the entire Ensemble to work ( elect leaders, manage the cluster topology etc etc) .
The quorum is 50% living nodes +1 .
If you have an ensemble of 3 nodes, the quorum is 3/2 +1 = 2 nodes .
With an ensemble of 3 nodes, you can lose 1 and the ZK ensemble will continue to work.
If you have an ensemble of 5 nodes, the quorum is 5/2 +1 = 3 nodes With an ensemble of 5 nodes, you can lose 2 and the ZK ensemble will continue to work.
ect ect
Cheers
2015-10-06 10:55 GMT+01:00 Adrian Liew <ad...@avanade.com>:
> Hi there,
>
>
>
> I have 3 Solr server Azure VM nodes participating in SolrCloud with ZK
> installed on each of these nodes (to avoid a single point of failure
> with ZK for leader election). Each Solr server is hosted in a Windows
> Server
> 2012 R2 environment. I was told by my peer that if one zookeeper
> service fails, the entire quorum fails. Hence if a quorum fails, does
> that mean it will not be able to elect the leader from the remaining 2
> alive Solr servers, even if ZK services are installed in each node?
>
>
>
> I am yet to this out as this defeats the purpose of having a ZK
> installed on each server. I am afraid if one node fails, a leader
> cannot be elected with the remaining two available nodes. Correct me if I am wrong.
>
>
>
> Regards,
>
> Adrian
>
>
>
--
--------------------------
Benedetti Alessandro
Visiting card - http://about.me/alessandro_benedetti
Blog - http://alexbenedetti.blogspot.co.uk
"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"
William Blake - Songs of Experience -1794 England
Re: If zookeeper is down, SolrCloud nodes will not start correctly,
even if zookeeper is started later
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/7/2015 3:06 AM, Adrian Liew wrote:
> Thanks for informing me. I guess the worst case scenario is that all 3 ZK services are down and that may be unlikely the case. At this juncture, as you said the viable workaround is a manual approach to start up the services in sequence in ensuring a quorum can take place. So the proper sequence in a 3 ZK + Solr (both ZK and Solr in each server) server setup will be as follows:
>
> Downed situation with one or mode ZK services
> 1. Restart all ZK Services first on all three machines
> 2. Restart all Solr Services on all three machines
>
> Please do clarify if the above is correct and I will be happy to take this approach and communicate to my customer.
If zookeeper is external (not embedded in Solr), your procedure is
correct -- ensure enough ZK nodes are started to reach quorum, then
start Solr. If using the embedded zookeeper (-DzkRun) then you would
want to follow the procedure I outlined in my last message.
Thanks,
Shawn
RE: If zookeeper is down, SolrCloud nodes will not start correctly,
even if zookeeper is started later
Posted by Adrian Liew <ad...@avanade.com>.
Hi Shawn
Thanks for informing me. I guess the worst case scenario is that all 3 ZK services are down and that may be unlikely the case. At this juncture, as you said the viable workaround is a manual approach to start up the services in sequence in ensuring a quorum can take place. So the proper sequence in a 3 ZK + Solr (both ZK and Solr in each server) server setup will be as follows:
Downed situation with one or mode ZK services
1. Restart all ZK Services first on all three machines
2. Restart all Solr Services on all three machines
Please do clarify if the above is correct and I will be happy to take this approach and communicate to my customer.
Many thanks.
Regards,
Adrian
-----Original Message-----
From: Shawn Heisey [mailto:apache@elyograg.org]
Sent: Wednesday, October 7, 2015 4:09 PM
To: solr-user@lucene.apache.org
Subject: Re: If zookeeper is down, SolrCloud nodes will not start correctly, even if zookeeper is started later
On 10/6/2015 10:22 PM, Adrian Liew wrote:
> Hence, the issue is that upon startup of three machines, the startup
> of ZK and Solr is out of sequence that causes SolrCloud to behave
> unexpectedly. Noting there is Jira ticket addressed here for Solr 4.9
> above to include an improvement to the issue above.
> (https://issues.apache.org/jira/browse/SOLR-5129)
That issue is unresolved, so it has not been fixed in any Solr version.
At this time, if you do not have Zookeeper quorum (a majority of your ZK nodes fully operational), you will not be able to successfully start SolrCloud nodes. The issue has low priority because there is a viable workaround -- ensure that ZK has quorum before starting or restarting any Solr node.
Thinking out loud: Until this issue is fixed, I think this means that a 3-node setup where all three nodes use the zookeeper embedded in Solr will require a strange startup sequence if none of the nodes are running:
* Start node 1. Solr will not start correctly -- no ZK quorum.
* Start node 2. Solr might start correctly, not sure.
* Start node 3. This should start correctly.
* Restart node 1. With ZK nodes 2 and 3 running, this will work.
* Restart node 2 if it did not start properly the first time.
I really have no idea whether the second node startup will work properly.
Thanks,
Shawn
Re: If zookeeper is down, SolrCloud nodes will not start correctly,
even if zookeeper is started later
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/6/2015 10:22 PM, Adrian Liew wrote:
> Hence, the issue is that upon startup of three machines, the startup of ZK and Solr is out of sequence that causes SolrCloud to behave unexpectedly. Noting there is Jira ticket addressed here for Solr 4.9 above to include an improvement to the issue above. (https://issues.apache.org/jira/browse/SOLR-5129)
That issue is unresolved, so it has not been fixed in any Solr version.
At this time, if you do not have Zookeeper quorum (a majority of your ZK
nodes fully operational), you will not be able to successfully start
SolrCloud nodes. The issue has low priority because there is a viable
workaround -- ensure that ZK has quorum before starting or restarting
any Solr node.
Thinking out loud: Until this issue is fixed, I think this means that a
3-node setup where all three nodes use the zookeeper embedded in Solr
will require a strange startup sequence if none of the nodes are running:
* Start node 1. Solr will not start correctly -- no ZK quorum.
* Start node 2. Solr might start correctly, not sure.
* Start node 3. This should start correctly.
* Restart node 1. With ZK nodes 2 and 3 running, this will work.
* Restart node 2 if it did not start properly the first time.
I really have no idea whether the second node startup will work properly.
Thanks,
Shawn