collections replicas still in Recovery Mode after restarting Solr

[Derek Poh <dp...@globalsources.com>]

Hi
We have a setup of 2 servers, running Solr 6.6.2, on production.
There are 5 collections.
All collection are created as 1 shard x 2 replicas.

4 of the collections have this issue.
A replica of each of this 4 collections is in Recovery Mode. The 
affected replicas are on the same server or node.
I noticed there is no Leader node indicated for this 4 collections in 
the Solr Admin. This is the screenshot of the Solr Admin 
http://imagebucket.net/pmndqkijla5c/solr_admin.PNG This is the commands 
I used to stop and start the solr process, bin/solr stop -p 8983 
bin/solr start -cloud -p 8983 -s "/apps/search/solr-6.6.2/home" -z 
hktszk1:2181,hktszk2:2181,hktszk3:2181 May I know how can I bring up 
this replicas? Derek

----------------------
CONFIDENTIALITY NOTICE 

This e-mail (including any attachments) may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error, please inform the sender immediately and delete this e-mail (including any attachments) from your computer, and you must not use, disclose to anyone else or copy this e-mail (including any attachments), whether in whole or in part. 

This e-mail and any reply to it may be monitored for security, legal, regulatory compliance and/or other appropriate reasons.

Re: collections replicas still in Recovery Mode after restarting Solr

[Shawn Heisey <ap...@elyograg.org>]

On 8/15/2018 1:26 AM, Derek Poh wrote:
> We have a setup of 2 servers, running Solr 6.6.2, on production.
> There are 5 collections.
> All collection are created as 1 shard x 2 replicas.
>
> 4 of the collections have this issue.
> A replica of each of this 4 collections is in Recovery Mode. The 
> affected replicas are on the same server or node.
> I noticed there is no Leader node indicated for this 4 collections in 
> the Solr Admin. This is the screenshot of the Solr Admin 
> http://imagebucket.net/pmndqkijla5c/solr_admin.PNG This is the 
> commands I used to stop and start the solr process, bin/solr stop -p 
> 8983 bin/solr start -cloud -p 8983 -s "/apps/search/solr-6.6.2/home" 
> -z hktszk1:2181,hktszk2:2181,hktszk3:2181 May I know how can I bring 
> up this replicas? Derek 

We see reports of this happening occasionally on different versions.

 From what I've seen, you may need to restart ALL of your Solr servers 
at least once to clear the problem, even the ones that seem to be 
working.  It may take more than one restart.

If there continue to be problems, share the solr.log file from each 
server so we can look at what the errors are.

Thanks,
Shawn