Solr 4.8 Overseer/Queue Processing

["Durham, Russell" <rd...@medassets.com>]

Hello,

So I’m running SolrCloud 4.8 on a two node cluster and 3 zookeeper instances on a separate set of machines. The cluster has roughly 150 collections, each running as 1 shard with a replication factor of 2. There is also a management tool we have running that will periodically request cluster status from solr to update some other monitoring services we have in place. We’re also running the instances with Tomcat.

For the most part everything seems to be working great, however every now and then the cluster will start timing out when the management tool requests the cluster status. When I go to check the /overseer/queue node I can see that the work queue is starting to build up and the leader is no longer processing the queue. I’ve been bouncing the Tomcat service which fixes the issue but it will eventually come back and continuing to bounce the Tomcat service is not really an option. I tried searching and found some old threads with the same issue but it looked as if whatever it was got resolved in an earlier 4.X version.

We do have plans to upgrade to 5.X early next year at which point will also switch to using Jetty instead of Tomcat. There is still some time before that happens though so I’d like to figure out what is going on now if possible. I tried looking at the solr logs but I can’t seem to find anything that says why it’s not processing the queue.

Thanks in advance for any thoughts/suggestions.

Russell

Russell Durham | Senior Software Engineer | MedAssets
5543 Legacy Drive | Plano, TX, 75024 | Work: 972.202.5850 | Mobile: 361.564.7223
rdurham@medassets.com<ma...@medassets.com>
Visit us at www.medassets.com<http://www.medassets.com>
Follow us on LinkedIn<http://www.linkedin.com/company/medassets>, YouTube<https://www.youtube.com/user/MedAssetsInc>, Twitter<https://twitter.com/MedAssets>, and Facebook<https://www.facebook.com/MedAssets>

*****Attention*****
This electronic transmission may contain confidential, sensitive, proprietary and/or privileged information belonging to the sender. This information, including any attached files, is intended only for the persons or entities to which it is addressed. Authorized recipients of this information are prohibited from disclosing the information to any unauthorized party and are required to properly dispose of the information upon fulfillment of its need/use, unless otherwise required by law. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by any person or entity other than the intended recipient is prohibited. If you have received this electronic transmission in error, please notify the sender and properly dispose of the information immediately.