java.io.FileNotFoundException?

[Doug Steigerwald <ds...@mcclatchyinteractive.com>]

We just started hitting a FileNotFoundException for no real apparent reason for both our regular
index and our spellchecker index, and only a few minute after we restarted Solr.  I did some 
searching and didn't find much that helped.

We started to do some load testing, and after about 10 minutes we started getting these errors.

We hit the spellchecker every request through a SpellcheckComponent that we created (ie, code ripped 
out of SpellCheckRequestHandler for now).  It runs essentially the same code as the spellcheck 
request handler when we specify a parameter (spellcheck=true).

We have 34 cores.  All but two cores are fully optimized (haven't been updated in 2 months).  Only 
two cores are actively updated.  We started Solr around 11:45am, not much happened until 12:27 when 
we started load testing (just a few queries, maybe 100 updates).

find /home/dsteiger/local/solr/cores/*/data/index|wc -l  => 414
find /home/dsteiger/local/solr/cores/*/data/spell|wc -l  => 6 (only the two 'active' cores use the 
spell checker).  So, not many files are open.

Anyone have any idea what might cause the two below errors to happen?  When I restarted Solr around 
11:45am it was to test a new patch that set the mergeFactor in the lucene spellchecker to 2 instead 
of 300 because we kept running into 'too many files open' errors when rebuilding more than one spell 
index at a time.  The spell indexes were rebuilt manually using the mergeFactor of 300, solr 
restarted, and any subsequent rebuild of the spell index would use a mergeFactor of 2.

After we hit this error, I rebuilt the spell indexes with the new code replicated them to the slave, 
restarted Solr, and all has been well.  We ran the load testing for more than an hour and the issue 
hasn't returned.

Could the old spell indexes that were created using the high mergeFactor cause an issue like this 
somehow?  Could the opening and closing of searchers so fast cause this?  I don't have the slightest 
idea.  All of our search queries hit the slave, and the master just handles updates.  The master had 
no issues through all of this.

Caused by: java.io.IOException: cannot read directory
org.apache.lucene.store.FSDirectory@/home/dsteiger/local/solr/cores/qaa/data/spell: list() returned null
	at org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInfos.java:115)
	at org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:506)
	at org.apache.lucene.search.spell.SpellChecker.setSpellIndex(SpellChecker.java:102)
	at org.apache.lucene.search.spell.SpellChecker.<init>(SpellChecker.java:89)


And this happened I believe when running the snapinstaller (done through cron)...

Caused by: java.io.FileNotFoundException: no segments* file found in
org.apache.lucene.store.FSDirectory@/home/dsteiger/local/solr/cores/qab/data/index: files: null
         at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:587)
         at org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:63)
         at org.apache.lucene.index.IndexReader.open(IndexReader.java:209)
         at org.apache.lucene.index.IndexReader.open(IndexReader.java:173)
         at org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:93)
         at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:706)

We're running r614955.

Thanks.
Doug