You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Jan Høydahl <ja...@cominvent.com> on 2024/04/24 08:51:12 UTC

Re: [EXTERNAL] SOLR Shard nodes are unhealthy

Hi,

Indeed, Solr 8.11 used zk client v3.6.2, but I believe it will also work with newer 3.x versions,  you can test this locally on your laptop first to be sure.

Note also that network resolution should work between all nodes on the FQDN.
So if your ZK1 node is called ZK1.region3.example.com and is on ip 10.0.1.1 then it should be routable on that FQDN address from all solr and zk nodes, and a reverse lookup on IP should also return the same host name.

Jan

> 24. apr. 2024 kl. 10:38 skrev Ritesh Kumar (Accenture International Limite) <v-...@microsoft.com.INVALID>:
> 
> Hi Jan,
> 
> Your assumptions are correct as those are setup correctly. I will provide the logs.
> 
> Meanwhile I can give a try with SOLR 8.11.3 and see how it behaves. Moreover just wanted to check one thing:
> 
> SOLR 8.11 document says "Solr currently uses Apache ZooKeeper v3.6.2."
> Setting Up an External ZooKeeper Ensemble | Apache Solr Reference Guide 8.11<https://solr.apache.org/guide/8_11/setting-up-an-external-zookeeper-ensemble.html>
> 
> However looking at Apache ZooKeeper<https://zookeeper.apache.org/releases.html>, it says "Apache ZooKeeper 3.9.2 is our current release, and 3.8.4 our latest stable release."
> 
> Hence i wanted to check if i can install zookeeper 3.8.4 along with solr 8.11.3 or i have to use ZooKeeper v3.6.2 only.
> 
> 
> Thanks
> 
> Ritesh
> 
> ________________________________
> From: Jan H?ydahl <ja...@cominvent.com>
> Sent: Wednesday, April 24, 2024 01:41 PM
> To: users@solr.apache.org <us...@solr.apache.org>
> Subject: [EXTERNAL] Re: SOLR Shard nodes are unhealthy
> 
> Hi,
> 
> First, you should use the latest 8.11.3 which has a lot of fixes.
> 
> Your setup could work if the regions are connected with really low latency and high bandwidth.
> 
> Your description suggests that your Solr-ZK connection is not as it should.
> 
> A few assumptions for this to work:
> * You ave installed ZK correctly as one five-node ensemble, and validated it works flawlessly
> * All your Solr nodes lists all five ZK nodes in ZK_HOST, e.g. ZK_HOST=zk1:2181,zk2:2182,zk3:2181,zk4:2181,zk5:2181
> 
> You'll have to provide more information regarding your exact configuration and/or error log messages to get further on this.
> 
> Jan
> 
>> 24. apr. 2024 kl. 04:32 skrev Ritesh Kumar (Accenture International Limite) <v-...@microsoft.com.INVALID>:
>> 
>> Hello All,
>> 
>> I have setup SOLR-ZK ensemble (Windows) with 6 SOLR VMs across two regions and 5 ZK VMs across three regions.
>> 
>> SOLR Version - 8.11.2
>> ZK Version - 3.6.2
>> 
>> Region1:
>> SOLR1
>> SOLR2
>> SOLR3
>> ZK2
>> ZK3
>> 
>> Region2:
>> SOLR4
>> SOLR5
>> SOLR6
>> ZK4
>> ZK5
>> 
>> Region3:
>> ZK1
>> 
>> I am facing issue after restarting ZK and SOLR VMs . The SOLR UI is showing all the 6 SOLR nodes and 5 ZK nodes are active and with leader.
>> 
>> All the 3 Solr nodes (1,2,3) of each shard is in recovery and later down and remaining 3 solr nodes (4,5,6) are healthy. Now if i restart solr nodes 4,5,6 they go to recovery/down and nodes 1,2,3 become healthy - but all 6 nodes are never becoming healthy together. I have tried restarting Solr and ZK multiple times.
>> 
>> Please let me know if anyone has faced this issue or has suggestions for a fix.
>> 
>> 
>> 
>> 
>> Note: I cannot upgrade to latest SOLR version as this is recommened by Sitecore.
>> 
>> Thanks
>> Ritesh
>

Re: [EXTERNAL] SOLR Shard nodes are unhealthy

Posted by "Ritesh Kumar (Accenture International Limite)" <v-...@microsoft.com.INVALID>.
I have used the FQDN in -h parameter.


Thanks

Ritesh

________________________________
From: Jan Høydahl <ja...@cominvent.com>
Sent: Thursday, April 25, 2024 08:03 PM
To: users@solr.apache.org <us...@solr.apache.org>
Subject: Re: [EXTERNAL] SOLR Shard nodes are unhealthy

Make sure you use fully qualified domain names everywhere, and that they are routable everywhere. If you start Solr with -h my-short-name then it will be persisted in zk state and will not be routable from all other solr hosts.

Jan Høydahl

> 25. apr. 2024 kl. 13:34 skrev Ritesh Kumar (Accenture International Limite) <v-...@microsoft.com.invalid>:
>
> Hello,
>
> One additional observation, I do not face the issue if I use IP addresses for the solr nodes but I observe this issue as soon as I modify solr service to use hostname instead of the IP address.
>
> start -f -h XX.XX.XX.XX -p 8983 -cloud -Dsolr.ssl.checkPeerName=false -m 8g -z "XX.X.X.XX:2181,YY.Y.Y.YY:2181,ZZ.Z.Z.ZZ:2181,AA.A.A.AA:2181,BB.B.B.BB:2181"
>
> start -f -h HOSTNAME -p 8983 -cloud -Dsolr.ssl.checkPeerName=false -m 8g -z "XX.X.X.XX:2181,YY.Y.Y.YY:2181,ZZ.Z.Z.ZZ:2181,AA.A.A.AA:2181,BB.B.B.BB:2181"
>
>
> Thanks
>
> Ritesh
>
> ________________________________
> From: Jan H?ydahl <ja...@cominvent.com>
> Sent: Wednesday, April 24, 2024 02:21 PM
> To: users@solr.apache.org <us...@solr.apache.org>
> Subject: Re: [EXTERNAL] SOLR Shard nodes are unhealthy
>
> Hi,
>
> Indeed, Solr 8.11 used zk client v3.6.2, but I believe it will also work with newer 3.x versions,  you can test this locally on your laptop first to be sure.
>
> Note also that network resolution should work between all nodes on the FQDN.
> So if your ZK1 node is called ZK1.region3.example.com and is on ip 10.0.1.1 then it should be routable on that FQDN address from all solr and zk nodes, and a reverse lookup on IP should also return the same host name.
>
> Jan
>
>> 24. apr. 2024 kl. 10:38 skrev Ritesh Kumar (Accenture International Limite) <v-...@microsoft.com.INVALID>:
>>
>> Hi Jan,
>>
>> Your assumptions are correct as those are setup correctly. I will provide the logs.
>>
>> Meanwhile I can give a try with SOLR 8.11.3 and see how it behaves. Moreover just wanted to check one thing:
>>
>> SOLR 8.11 document says "Solr currently uses Apache ZooKeeper v3.6.2."
>> Setting Up an External ZooKeeper Ensemble | Apache Solr Reference Guide 8.11<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fguide%2F8_11%2Fsetting-up-an-external-zookeeper-ensemble.html&data=05%7C02%7Cv-kurite%40microsoft.com%7Cc9baee72f81f47bfa1e308dc6534b8e0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638496524360921223%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=iw5bQ9hymThEG%2FUSmrN83NMjPNxNr%2Bxhz9HUr8R97pw%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fguide%2F8_11%2Fsetting-up-an-external-zookeeper-ensemble.html&data=05%7C02%7Cv-kurite%40microsoft.com%7Cc9baee72f81f47bfa1e308dc6534b8e0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638496524360929843%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=eWvihYZuxcH1Z2yjkIEbv3LZCdWs2nzd5lgyN7dsmT0%3D&reserved=0<https://solr.apache.org/guide/8_11/setting-up-an-external-zookeeper-ensemble.html>>>
>>
>> However looking at Apache ZooKeeper<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fzookeeper.apache.org%2Freleases.html&data=05%7C02%7Cv-kurite%40microsoft.com%7Cc9baee72f81f47bfa1e308dc6534b8e0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638496524360933765%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=gI9A10jbDogJhoE%2B4LKftTikXpEecLR8Wb8ilo7v5zI%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fzookeeper.apache.org%2Freleases.html&data=05%7C02%7Cv-kurite%40microsoft.com%7Cc9baee72f81f47bfa1e308dc6534b8e0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638496524360937470%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=wurzI5%2F3sfkpGDH7h6lDAj4d3UCebpzm66IIOtcx4LI%3D&reserved=0<https://zookeeper.apache.org/releases.html>>>, it says "Apache ZooKeeper 3.9.2 is our current release, and 3.8.4 our latest stable release."
>>
>> Hence i wanted to check if i can install zookeeper 3.8.4 along with solr 8.11.3 or i have to use ZooKeeper v3.6.2 only.
>>
>>
>> Thanks
>>
>> Ritesh
>>
>> ________________________________
>> From: Jan H?ydahl <ja...@cominvent.com>
>> Sent: Wednesday, April 24, 2024 01:41 PM
>> To: users@solr.apache.org <us...@solr.apache.org>
>> Subject: [EXTERNAL] Re: SOLR Shard nodes are unhealthy
>>
>> Hi,
>>
>> First, you should use the latest 8.11.3 which has a lot of fixes.
>>
>> Your setup could work if the regions are connected with really low latency and high bandwidth.
>>
>> Your description suggests that your Solr-ZK connection is not as it should.
>>
>> A few assumptions for this to work:
>> * You ave installed ZK correctly as one five-node ensemble, and validated it works flawlessly
>> * All your Solr nodes lists all five ZK nodes in ZK_HOST, e.g. ZK_HOST=zk1:2181,zk2:2182,zk3:2181,zk4:2181,zk5:2181
>>
>> You'll have to provide more information regarding your exact configuration and/or error log messages to get further on this.
>>
>> Jan
>>
>>>> 24. apr. 2024 kl. 04:32 skrev Ritesh Kumar (Accenture International Limite) <v-...@microsoft.com.INVALID>:
>>>
>>> Hello All,
>>>
>>> I have setup SOLR-ZK ensemble (Windows) with 6 SOLR VMs across two regions and 5 ZK VMs across three regions.
>>>
>>> SOLR Version - 8.11.2
>>> ZK Version - 3.6.2
>>>
>>> Region1:
>>> SOLR1
>>> SOLR2
>>> SOLR3
>>> ZK2
>>> ZK3
>>>
>>> Region2:
>>> SOLR4
>>> SOLR5
>>> SOLR6
>>> ZK4
>>> ZK5
>>>
>>> Region3:
>>> ZK1
>>>
>>> I am facing issue after restarting ZK and SOLR VMs . The SOLR UI is showing all the 6 SOLR nodes and 5 ZK nodes are active and with leader.
>>>
>>> All the 3 Solr nodes (1,2,3) of each shard is in recovery and later down and remaining 3 solr nodes (4,5,6) are healthy. Now if i restart solr nodes 4,5,6 they go to recovery/down and nodes 1,2,3 become healthy - but all 6 nodes are never becoming healthy together. I have tried restarting Solr and ZK multiple times.
>>>
>>> Please let me know if anyone has faced this issue or has suggestions for a fix.
>>>
>>>
>>>
>>>
>>> Note: I cannot upgrade to latest SOLR version as this is recommened by Sitecore.
>>>
>>> Thanks
>>> Ritesh
>>
>

Re: [EXTERNAL] SOLR Shard nodes are unhealthy

Posted by Jan Høydahl <ja...@cominvent.com>.
Make sure you use fully qualified domain names everywhere, and that they are routable everywhere. If you start Solr with -h my-short-name then it will be persisted in zk state and will not be routable from all other solr hosts.

Jan Høydahl

> 25. apr. 2024 kl. 13:34 skrev Ritesh Kumar (Accenture International Limite) <v-...@microsoft.com.invalid>:
> 
> Hello,
> 
> One additional observation, I do not face the issue if I use IP addresses for the solr nodes but I observe this issue as soon as I modify solr service to use hostname instead of the IP address.
> 
> start -f -h XX.XX.XX.XX -p 8983 -cloud -Dsolr.ssl.checkPeerName=false -m 8g -z "XX.X.X.XX:2181,YY.Y.Y.YY:2181,ZZ.Z.Z.ZZ:2181,AA.A.A.AA:2181,BB.B.B.BB:2181"
> 
> start -f -h HOSTNAME -p 8983 -cloud -Dsolr.ssl.checkPeerName=false -m 8g -z "XX.X.X.XX:2181,YY.Y.Y.YY:2181,ZZ.Z.Z.ZZ:2181,AA.A.A.AA:2181,BB.B.B.BB:2181"
> 
> 
> Thanks
> 
> Ritesh
> 
> ________________________________
> From: Jan H?ydahl <ja...@cominvent.com>
> Sent: Wednesday, April 24, 2024 02:21 PM
> To: users@solr.apache.org <us...@solr.apache.org>
> Subject: Re: [EXTERNAL] SOLR Shard nodes are unhealthy
> 
> Hi,
> 
> Indeed, Solr 8.11 used zk client v3.6.2, but I believe it will also work with newer 3.x versions,  you can test this locally on your laptop first to be sure.
> 
> Note also that network resolution should work between all nodes on the FQDN.
> So if your ZK1 node is called ZK1.region3.example.com and is on ip 10.0.1.1 then it should be routable on that FQDN address from all solr and zk nodes, and a reverse lookup on IP should also return the same host name.
> 
> Jan
> 
>> 24. apr. 2024 kl. 10:38 skrev Ritesh Kumar (Accenture International Limite) <v-...@microsoft.com.INVALID>:
>> 
>> Hi Jan,
>> 
>> Your assumptions are correct as those are setup correctly. I will provide the logs.
>> 
>> Meanwhile I can give a try with SOLR 8.11.3 and see how it behaves. Moreover just wanted to check one thing:
>> 
>> SOLR 8.11 document says "Solr currently uses Apache ZooKeeper v3.6.2."
>> Setting Up an External ZooKeeper Ensemble | Apache Solr Reference Guide 8.11<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fguide%2F8_11%2Fsetting-up-an-external-zookeeper-ensemble.html&data=05%7C02%7Cv-kurite%40microsoft.com%7Ca075331631884b3cfb1808dc643bc41a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638495455093288541%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=2bN%2B4oq9MJcmNv315lnR4Op9OvPG4yP0cF%2FQDye9%2Ffs%3D&reserved=0<https://solr.apache.org/guide/8_11/setting-up-an-external-zookeeper-ensemble.html>>
>> 
>> However looking at Apache ZooKeeper<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fzookeeper.apache.org%2Freleases.html&data=05%7C02%7Cv-kurite%40microsoft.com%7Ca075331631884b3cfb1808dc643bc41a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638495455093299320%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=78iVlW3%2BRilLuGrFr%2FzqdnYYRdDvM5H891djm4ZFI%2B0%3D&reserved=0<https://zookeeper.apache.org/releases.html>>, it says "Apache ZooKeeper 3.9.2 is our current release, and 3.8.4 our latest stable release."
>> 
>> Hence i wanted to check if i can install zookeeper 3.8.4 along with solr 8.11.3 or i have to use ZooKeeper v3.6.2 only.
>> 
>> 
>> Thanks
>> 
>> Ritesh
>> 
>> ________________________________
>> From: Jan H?ydahl <ja...@cominvent.com>
>> Sent: Wednesday, April 24, 2024 01:41 PM
>> To: users@solr.apache.org <us...@solr.apache.org>
>> Subject: [EXTERNAL] Re: SOLR Shard nodes are unhealthy
>> 
>> Hi,
>> 
>> First, you should use the latest 8.11.3 which has a lot of fixes.
>> 
>> Your setup could work if the regions are connected with really low latency and high bandwidth.
>> 
>> Your description suggests that your Solr-ZK connection is not as it should.
>> 
>> A few assumptions for this to work:
>> * You ave installed ZK correctly as one five-node ensemble, and validated it works flawlessly
>> * All your Solr nodes lists all five ZK nodes in ZK_HOST, e.g. ZK_HOST=zk1:2181,zk2:2182,zk3:2181,zk4:2181,zk5:2181
>> 
>> You'll have to provide more information regarding your exact configuration and/or error log messages to get further on this.
>> 
>> Jan
>> 
>>>> 24. apr. 2024 kl. 04:32 skrev Ritesh Kumar (Accenture International Limite) <v-...@microsoft.com.INVALID>:
>>> 
>>> Hello All,
>>> 
>>> I have setup SOLR-ZK ensemble (Windows) with 6 SOLR VMs across two regions and 5 ZK VMs across three regions.
>>> 
>>> SOLR Version - 8.11.2
>>> ZK Version - 3.6.2
>>> 
>>> Region1:
>>> SOLR1
>>> SOLR2
>>> SOLR3
>>> ZK2
>>> ZK3
>>> 
>>> Region2:
>>> SOLR4
>>> SOLR5
>>> SOLR6
>>> ZK4
>>> ZK5
>>> 
>>> Region3:
>>> ZK1
>>> 
>>> I am facing issue after restarting ZK and SOLR VMs . The SOLR UI is showing all the 6 SOLR nodes and 5 ZK nodes are active and with leader.
>>> 
>>> All the 3 Solr nodes (1,2,3) of each shard is in recovery and later down and remaining 3 solr nodes (4,5,6) are healthy. Now if i restart solr nodes 4,5,6 they go to recovery/down and nodes 1,2,3 become healthy - but all 6 nodes are never becoming healthy together. I have tried restarting Solr and ZK multiple times.
>>> 
>>> Please let me know if anyone has faced this issue or has suggestions for a fix.
>>> 
>>> 
>>> 
>>> 
>>> Note: I cannot upgrade to latest SOLR version as this is recommened by Sitecore.
>>> 
>>> Thanks
>>> Ritesh
>> 
>

Re: [EXTERNAL] SOLR Shard nodes are unhealthy

Posted by "Ritesh Kumar (Accenture International Limite)" <v-...@microsoft.com.INVALID>.
Hello,

One additional observation, I do not face the issue if I use IP addresses for the solr nodes but I observe this issue as soon as I modify solr service to use hostname instead of the IP address.

start -f -h XX.XX.XX.XX -p 8983 -cloud -Dsolr.ssl.checkPeerName=false -m 8g -z "XX.X.X.XX:2181,YY.Y.Y.YY:2181,ZZ.Z.Z.ZZ:2181,AA.A.A.AA:2181,BB.B.B.BB:2181"

start -f -h HOSTNAME -p 8983 -cloud -Dsolr.ssl.checkPeerName=false -m 8g -z "XX.X.X.XX:2181,YY.Y.Y.YY:2181,ZZ.Z.Z.ZZ:2181,AA.A.A.AA:2181,BB.B.B.BB:2181"


Thanks

Ritesh

________________________________
From: Jan H?ydahl <ja...@cominvent.com>
Sent: Wednesday, April 24, 2024 02:21 PM
To: users@solr.apache.org <us...@solr.apache.org>
Subject: Re: [EXTERNAL] SOLR Shard nodes are unhealthy

Hi,

Indeed, Solr 8.11 used zk client v3.6.2, but I believe it will also work with newer 3.x versions,  you can test this locally on your laptop first to be sure.

Note also that network resolution should work between all nodes on the FQDN.
So if your ZK1 node is called ZK1.region3.example.com and is on ip 10.0.1.1 then it should be routable on that FQDN address from all solr and zk nodes, and a reverse lookup on IP should also return the same host name.

Jan

> 24. apr. 2024 kl. 10:38 skrev Ritesh Kumar (Accenture International Limite) <v-...@microsoft.com.INVALID>:
>
> Hi Jan,
>
> Your assumptions are correct as those are setup correctly. I will provide the logs.
>
> Meanwhile I can give a try with SOLR 8.11.3 and see how it behaves. Moreover just wanted to check one thing:
>
> SOLR 8.11 document says "Solr currently uses Apache ZooKeeper v3.6.2."
> Setting Up an External ZooKeeper Ensemble | Apache Solr Reference Guide 8.11<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fguide%2F8_11%2Fsetting-up-an-external-zookeeper-ensemble.html&data=05%7C02%7Cv-kurite%40microsoft.com%7Ca075331631884b3cfb1808dc643bc41a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638495455093288541%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=2bN%2B4oq9MJcmNv315lnR4Op9OvPG4yP0cF%2FQDye9%2Ffs%3D&reserved=0<https://solr.apache.org/guide/8_11/setting-up-an-external-zookeeper-ensemble.html>>
>
> However looking at Apache ZooKeeper<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fzookeeper.apache.org%2Freleases.html&data=05%7C02%7Cv-kurite%40microsoft.com%7Ca075331631884b3cfb1808dc643bc41a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638495455093299320%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=78iVlW3%2BRilLuGrFr%2FzqdnYYRdDvM5H891djm4ZFI%2B0%3D&reserved=0<https://zookeeper.apache.org/releases.html>>, it says "Apache ZooKeeper 3.9.2 is our current release, and 3.8.4 our latest stable release."
>
> Hence i wanted to check if i can install zookeeper 3.8.4 along with solr 8.11.3 or i have to use ZooKeeper v3.6.2 only.
>
>
> Thanks
>
> Ritesh
>
> ________________________________
> From: Jan H?ydahl <ja...@cominvent.com>
> Sent: Wednesday, April 24, 2024 01:41 PM
> To: users@solr.apache.org <us...@solr.apache.org>
> Subject: [EXTERNAL] Re: SOLR Shard nodes are unhealthy
>
> Hi,
>
> First, you should use the latest 8.11.3 which has a lot of fixes.
>
> Your setup could work if the regions are connected with really low latency and high bandwidth.
>
> Your description suggests that your Solr-ZK connection is not as it should.
>
> A few assumptions for this to work:
> * You ave installed ZK correctly as one five-node ensemble, and validated it works flawlessly
> * All your Solr nodes lists all five ZK nodes in ZK_HOST, e.g. ZK_HOST=zk1:2181,zk2:2182,zk3:2181,zk4:2181,zk5:2181
>
> You'll have to provide more information regarding your exact configuration and/or error log messages to get further on this.
>
> Jan
>
>> 24. apr. 2024 kl. 04:32 skrev Ritesh Kumar (Accenture International Limite) <v-...@microsoft.com.INVALID>:
>>
>> Hello All,
>>
>> I have setup SOLR-ZK ensemble (Windows) with 6 SOLR VMs across two regions and 5 ZK VMs across three regions.
>>
>> SOLR Version - 8.11.2
>> ZK Version - 3.6.2
>>
>> Region1:
>> SOLR1
>> SOLR2
>> SOLR3
>> ZK2
>> ZK3
>>
>> Region2:
>> SOLR4
>> SOLR5
>> SOLR6
>> ZK4
>> ZK5
>>
>> Region3:
>> ZK1
>>
>> I am facing issue after restarting ZK and SOLR VMs . The SOLR UI is showing all the 6 SOLR nodes and 5 ZK nodes are active and with leader.
>>
>> All the 3 Solr nodes (1,2,3) of each shard is in recovery and later down and remaining 3 solr nodes (4,5,6) are healthy. Now if i restart solr nodes 4,5,6 they go to recovery/down and nodes 1,2,3 become healthy - but all 6 nodes are never becoming healthy together. I have tried restarting Solr and ZK multiple times.
>>
>> Please let me know if anyone has faced this issue or has suggestions for a fix.
>>
>>
>>
>>
>> Note: I cannot upgrade to latest SOLR version as this is recommened by Sitecore.
>>
>> Thanks
>> Ritesh
>