You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Tim Chen <Ti...@sbs.com.au> on 2016/07/27 04:27:36 UTC
Is it possible to force a Shard Leader change?
Hi Guys,
I am running a Solr Cloud 4.10, with 4 Solr servers and 5 Zookeeper setup.
Solr servers:
solr01, solr02, solr03, solr04
I have around 20 collections in Solr cloud, and there are 4 Shards for each Collection. For each Shard, I have 4 Replicas, and sitting on each Solr server, with one of them is the Shard Leader.
The issue I am having right now is all the Shard Leader are pointing to the same server, eg: solr01. When there are documents update, they are all pushed to the Leader. I really want to distribute the Shard Leader across all 4 Solr servers.
I noticed Solr 6 has a "REBALANCELEADERS" command to do that, but not available in Solr 4.
Questions:
1, Is my setup OK? with 4 Shards for each Collection and 4 Replicas for each Shard. Each Solr server has full set of documents.
2, To distribute the Shard Leader to different Solr servers, can I somehow shutdown a single Replica that is currently a Shard Leader and force Solr to elect a different replica to be new Shard Leader?
Thanks guys!
Regards,
Tim
[Roots Wednesday 27 July 8.30pm]<http://www.sbs.com.au/programs/roots/>
Re: Is it possible to force a Shard Leader change?
Posted by Anshum Gupta <an...@anshumgupta.net>.
I understand there could be many reasons, but if it is at all possible, I'd
suggest you upgrade to a more recent version of Solr.
With that, you'd get a ton of bug fixes, and also a bunch of APIs that
would help you with triggering a leader election.
On Tue, Jul 26, 2016 at 9:27 PM, Tim Chen <Ti...@sbs.com.au> wrote:
> Hi Guys,
>
> I am running a Solr Cloud 4.10, with 4 Solr servers and 5 Zookeeper setup.
>
> Solr servers:
> solr01, solr02, solr03, solr04
>
> I have around 20 collections in Solr cloud, and there are 4 Shards for
> each Collection. For each Shard, I have 4 Replicas, and sitting on each
> Solr server, with one of them is the Shard Leader.
>
> The issue I am having right now is all the Shard Leader are pointing to
> the same server, eg: solr01. When there are documents update, they are all
> pushed to the Leader. I really want to distribute the Shard Leader across
> all 4 Solr servers.
>
> I noticed Solr 6 has a "REBALANCELEADERS" command to do that, but not
> available in Solr 4.
>
> Questions:
>
> 1, Is my setup OK? with 4 Shards for each Collection and 4 Replicas for
> each Shard. Each Solr server has full set of documents.
> 2, To distribute the Shard Leader to different Solr servers, can I somehow
> shutdown a single Replica that is currently a Shard Leader and force Solr
> to elect a different replica to be new Shard Leader?
>
> Thanks guys!
>
> Regards,
> Tim
>
>
> [Roots Wednesday 27 July 8.30pm]<http://www.sbs.com.au/programs/roots/>
>
--
Anshum Gupta
RE: Is it possible to force a Shard Leader change?
Posted by Tim Chen <Ti...@sbs.com.au>.
Thanks Erick. You made the point, the CPU usage is not high on my Leader server, so I guess I will leave it.
And if I want to, as you suggested, I guess I could remove some replica from certain collections and add them back in to break the election order, then take JVM down to force an election.
Thanks again.
Cheers,
Tim
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Thursday, 28 July 2016 12:10 AM
To: solr-user
Subject: Re: Is it possible to force a Shard Leader change?
The REBALANCELEADERS stuff was put in to deal with 100s of leaders winding up on a single machine in a case where extremely high throughput was required. Until you get into pretty high scale the additional "work" on a leader is minimal. So unless your CPU usage is consistently significantly higher on the machine with all the leaders, I wouldn't worry about it.
Otherwise there isn't much you can do I'm afraid. If you have asymmetric replica placement leaders will tend to different machines.
You could try to take the JVM down on the machine with all the leaders and let leader election redistribute, but I that's not a long-term solution.
Best,
Erick
On Tue, Jul 26, 2016 at 9:27 PM, Tim Chen <Ti...@sbs.com.au> wrote:
> Hi Guys,
>
> I am running a Solr Cloud 4.10, with 4 Solr servers and 5 Zookeeper setup.
>
> Solr servers:
> solr01, solr02, solr03, solr04
>
> I have around 20 collections in Solr cloud, and there are 4 Shards for each Collection. For each Shard, I have 4 Replicas, and sitting on each Solr server, with one of them is the Shard Leader.
>
> The issue I am having right now is all the Shard Leader are pointing to the same server, eg: solr01. When there are documents update, they are all pushed to the Leader. I really want to distribute the Shard Leader across all 4 Solr servers.
>
> I noticed Solr 6 has a "REBALANCELEADERS" command to do that, but not available in Solr 4.
>
> Questions:
>
> 1, Is my setup OK? with 4 Shards for each Collection and 4 Replicas for each Shard. Each Solr server has full set of documents.
> 2, To distribute the Shard Leader to different Solr servers, can I somehow shutdown a single Replica that is currently a Shard Leader and force Solr to elect a different replica to be new Shard Leader?
>
> Thanks guys!
>
> Regards,
> Tim
>
>
> [Roots Wednesday 27 July
> 8.30pm]<http://www.sbs.com.au/programs/roots/>
[Premiere League Starts Saturday 13 August 9.30pm on SBS]<http://theworldgame.sbs.com.au/>
Re: Is it possible to force a Shard Leader change?
Posted by Erick Erickson <er...@gmail.com>.
The REBALANCELEADERS stuff was put in to deal with 100s of leaders
winding up on a single machine in a case where extremely high
throughput was required. Until you get into pretty high scale the
additional "work" on a leader is minimal. So unless your CPU usage is
consistently significantly higher on the machine with all the leaders,
I wouldn't worry about it.
Otherwise there isn't much you can do I'm afraid. If you have
asymmetric replica placement leaders will tend to different machines.
You could try to take the JVM down on the machine with all the leaders
and let leader election redistribute, but I that's not a long-term
solution.
Best,
Erick
On Tue, Jul 26, 2016 at 9:27 PM, Tim Chen <Ti...@sbs.com.au> wrote:
> Hi Guys,
>
> I am running a Solr Cloud 4.10, with 4 Solr servers and 5 Zookeeper setup.
>
> Solr servers:
> solr01, solr02, solr03, solr04
>
> I have around 20 collections in Solr cloud, and there are 4 Shards for each Collection. For each Shard, I have 4 Replicas, and sitting on each Solr server, with one of them is the Shard Leader.
>
> The issue I am having right now is all the Shard Leader are pointing to the same server, eg: solr01. When there are documents update, they are all pushed to the Leader. I really want to distribute the Shard Leader across all 4 Solr servers.
>
> I noticed Solr 6 has a "REBALANCELEADERS" command to do that, but not available in Solr 4.
>
> Questions:
>
> 1, Is my setup OK? with 4 Shards for each Collection and 4 Replicas for each Shard. Each Solr server has full set of documents.
> 2, To distribute the Shard Leader to different Solr servers, can I somehow shutdown a single Replica that is currently a Shard Leader and force Solr to elect a different replica to be new Shard Leader?
>
> Thanks guys!
>
> Regards,
> Tim
>
>
> [Roots Wednesday 27 July 8.30pm]<http://www.sbs.com.au/programs/roots/>