[jira] [Created] (SOLR-6707) Recovery/election for invalid core results in rapid-fire re-attempts until /overseer/queue is clogged

["James Hardwick (JIRA)" <ji...@apache.org>]

James Hardwick created SOLR-6707:
------------------------------------

             Summary: Recovery/election for invalid core results in rapid-fire re-attempts until /overseer/queue is clogged
                 Key: SOLR-6707
                 URL: https://issues.apache.org/jira/browse/SOLR-6707
             Project: Solr
          Issue Type: Bug
    Affects Versions: 4.10
            Reporter: James Hardwick


We experienced an issue the other day that brought a production solr server down, and this is what we found after investigating:

- Running solr instance with two separate cores, one of which is perpetually down because it's configs are not yet completely updated for Solr-cloud. This was thought to be harmless since it's not currently in use. 
- Solr experienced an "internal server error" I believe due in part to a fairly new feature we are using, which seemingly caused all cores to go down. 
- Solr immediately went into recovery, and subsequent leader election for each shard of each core. 
- Our primary core recovered immediately. Our additional core which was never active in the first place, attempted to recover but of course couldn't due to the improper configs. 
- Solr then began rapid-fire reattempting recovery of said node, trying maybe 20-30 times per second.
- This in turn bombarded zookeepers /overseer/queue into oblivion
- At some point /overseer/queue becomes so backed up that normal cluster coordination can no longer play out, and Solr topples over. 

I know this is a bit of an unusual circumstance due to us keeping the dead core around, and our quick solution has been to remove said core. However I can see other potential scenarios that might cause the same issue to arise. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org