Index Corruption (possibly during commit)

[Simon Wistow <si...@thegestalt.org>]

We have an indexing script which has been running for a couple of weeks 
now without problems. It indexes documents and then periodically commit 
(which is a tad redundant I suppose) both via the HTTP interface.

All documents are indexed to a master and a slave rsyncs them off using 
the standard 1.3.0 replication.

Recently the indexing script got into problems when the commit was 
taking longer than the request timeout. I killed the script, did a 
commit by hand (using 
bin/commit) and then started to index again and it still wouldn't 
commit. We then tried to go to the stats page and got the error

org.apache.lucene.index.CorruptIndexException:
doc counts differ for segment _mib: fieldsReader shows 1 but segmentInfo 
shows 718 at 
org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:960) at 
org.apache.solr.core.SolrCore.<init>(SolrCore.java:470) at 

This is a stock 1.3.0 running off tomcat 6.0.20 with 

java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode)

Linux solr.local 2.6.18-128.1.10.el5 #1 SMP Thu May 7 10:35:59 EDT 
2009 x86_64 x86_64 x86_64 GNU/Linux

Plenty of RAM and disk space (usage is 31% - 353G used from 534G)

CheckIndex says

Opening index @ index/

Segments file=segments_c8z numSegments=28 version=FORMAT_HAS_PROX 
[Lucene 2.4]

Checking only these segments: _mib:
  22 of 28: name=_mib docCount=718
    compound=false
    hasProx=true
    numFiles=9
    size (MB)=0.029
    has deletions [delFileName=_mib_1.del]
    test: open reader.........FAILED
    WARNING: fixIndex() would remove reference to this segment; full 
exception:
org.apache.lucene.index.CorruptIndexException: doc counts differ for 
segment _mib: fieldsReader shows 1 but segmentInfo shows 718
    at 
org.apache.lucene.index.SegmentReader$CoreReaders.openDocStores(SegmentReader.java:282)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:640)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:591)
    at 
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:491)
    at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:903)

WARNING: 1 broken segments (containing 718 documents) detected
WARNING: would write new segments file, and 718 documents would be lost, 
if -fix were specified


 
Any ideas? We can restore from back ups and back fill but really we'd 
love to know what caused this so we can avoid a repetition.

Simon