You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Li Li <fa...@gmail.com> on 2011/01/14 12:43:12 UTC
CorruptIndexException when indexing
hi all,
we have confronted this problem 3 times when testing
The exception stack is
Exception in thread "Lucene Merge Thread #2"
org.apache.lucene.index.MergePolicy$MergeException:
org.apache.lucene.index.CorruptIndexException: docs out of order (7286
<= 7286 )
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:355)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:319)
Caused by: org.apache.lucene.index.CorruptIndexException: docs out of
order (7286 <= 7286 )
at org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:75)
at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:880)
at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:818)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:756)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:187)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5354)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4937)
Or
Exception in thread "Lucene Merge Thread #0"
org.apache.lucene.index.MergePolicy$MergeException:
java.lang.ArrayIndexOutOfBoundsException: 330
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:355)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:319)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 330
at org.apache.lucene.util.BitVector.get(BitVector.java:102)
at org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:238)
at org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:168)
at org.apache.lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:98)
at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:870)
at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:818)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:756)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:187)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5354)
We did some minor modification based on lucene 2.9.1 and solr
1.4.0. we modified frq file to store 4 bytes for the positions of the
term occured
in these document(Accessing full postions in prx is time consuming
that can't meed our needs). I can't tell it's our bug or lucene's own
bug.
I searched the mail list and found the mail "problem during index
merge" posted in 2010-10-21. It's similar to our case.
It seems the docList in frq file is wrongly stored. When Merging,
when it's decoded, the wrong docID many larger than maxDocs(BitVector
deletedDocs)
which cause the second exception. Or docID delta is less than 0(it
reads wrongly) which cause the first exception
we are still continueing testing to turn off our modification and
open infoStream in solr-config.xml
We found a strange phenomenon. when we test, it sometimes hited
exceptions but in our production environment, it never hit any.
the hardware and software environments are the same. We checked
carefully and find the only difference is this line in solr-config.xml
<ramBufferSizeMB>32</ramBufferSizeMB> in testing environment
<ramBufferSizeMB>256</ramBufferSizeMB>in production environment
The indexed documents number for each machine is also roughly the
same. 10M+ documents.
I can't make sure the indice in production env are correct because
even there are some terms' docList are wrong, if the doc delta >0 and
don't have
some deleted documents, it will not hit the 2 exceptions.
The search results in production env and we don't find any strange results.
Will when the ramBufferSizeMB is too small results in index corruption?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: CorruptIndexException when indexing
Posted by Michael McCandless <lu...@mikemccandless.com>.
Phew! Thanks for bringing closure!
Yes, CheckIndex checks for index corruption. Best to run it with
assertions enabled...
Unfortunately random testing has limitations... I've seen similar
cases, where stuff passes the random tests but then only on
building/searching a "real" index (10M wikipedia docs) does something
go wrong...
Biasing the random generation towards the "extrema" might sometimes help.
Mike
On Sun, Jan 16, 2011 at 6:48 AM, Li Li <fa...@gmail.com> wrote:
> oh, sorry. It's our bug. we found it.
> CheckIndex is a tool to check the correctness of index?
> we check our modification by generating random documents and randomly test
> 3 main methods of SegmentTermDocs: next, read and skipTo. But the data is
> not "random" enough that some extreme positions are not tested so our
> modification
> make the index corrupted.
>
> 2011/1/15 Michael McCandless <lu...@mikemccandless.com>:
>> Different ramBufferSizeMB during indexing should never cause corruption!
>>
>> Can you try setting the ram buffer to 256 MB in your test env and see
>> if that makes the corruption go away?
>>
>> This could also be a hardware issue in your test env. If you run
>> CheckIndex on the corrupt index does it always fail in the same way?
>>
>> Mike
>>
>> On Fri, Jan 14, 2011 at 6:43 AM, Li Li <fa...@gmail.com> wrote:
>>> hi all,
>>> we have confronted this problem 3 times when testing
>>> The exception stack is
>>> Exception in thread "Lucene Merge Thread #2"
>>> org.apache.lucene.index.MergePolicy$MergeException:
>>> org.apache.lucene.index.CorruptIndexException: docs out of order (7286
>>> <= 7286 )
>>> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:355)
>>> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:319)
>>> Caused by: org.apache.lucene.index.CorruptIndexException: docs out of
>>> order (7286 <= 7286 )
>>> at org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:75)
>>> at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:880)
>>> at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:818)
>>> at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:756)
>>> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:187)
>>> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5354)
>>> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4937)
>>>
>>> Or
>>> Exception in thread "Lucene Merge Thread #0"
>>> org.apache.lucene.index.MergePolicy$MergeException:
>>> java.lang.ArrayIndexOutOfBoundsException: 330
>>> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:355)
>>> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:319)
>>> Caused by: java.lang.ArrayIndexOutOfBoundsException: 330
>>> at org.apache.lucene.util.BitVector.get(BitVector.java:102)
>>> at org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:238)
>>> at org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:168)
>>> at org.apache.lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:98)
>>> at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:870)
>>> at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:818)
>>> at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:756)
>>> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:187)
>>> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5354)
>>>
>>>
>>> We did some minor modification based on lucene 2.9.1 and solr
>>> 1.4.0. we modified frq file to store 4 bytes for the positions of the
>>> term occured
>>> in these document(Accessing full postions in prx is time consuming
>>> that can't meed our needs). I can't tell it's our bug or lucene's own
>>> bug.
>>> I searched the mail list and found the mail "problem during index
>>> merge" posted in 2010-10-21. It's similar to our case.
>>> It seems the docList in frq file is wrongly stored. When Merging,
>>> when it's decoded, the wrong docID many larger than maxDocs(BitVector
>>> deletedDocs)
>>> which cause the second exception. Or docID delta is less than 0(it
>>> reads wrongly) which cause the first exception
>>> we are still continueing testing to turn off our modification and
>>> open infoStream in solr-config.xml
>>>
>>> We found a strange phenomenon. when we test, it sometimes hited
>>> exceptions but in our production environment, it never hit any.
>>> the hardware and software environments are the same. We checked
>>> carefully and find the only difference is this line in solr-config.xml
>>> <ramBufferSizeMB>32</ramBufferSizeMB> in testing environment
>>> <ramBufferSizeMB>256</ramBufferSizeMB>in production environment
>>> The indexed documents number for each machine is also roughly the
>>> same. 10M+ documents.
>>> I can't make sure the indice in production env are correct because
>>> even there are some terms' docList are wrong, if the doc delta >0 and
>>> don't have
>>> some deleted documents, it will not hit the 2 exceptions.
>>> The search results in production env and we don't find any strange results.
>>>
>>> Will when the ramBufferSizeMB is too small results in index corruption?
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: CorruptIndexException when indexing
Posted by Li Li <fa...@gmail.com>.
oh, sorry. It's our bug. we found it.
CheckIndex is a tool to check the correctness of index?
we check our modification by generating random documents and randomly test
3 main methods of SegmentTermDocs: next, read and skipTo. But the data is
not "random" enough that some extreme positions are not tested so our
modification
make the index corrupted.
2011/1/15 Michael McCandless <lu...@mikemccandless.com>:
> Different ramBufferSizeMB during indexing should never cause corruption!
>
> Can you try setting the ram buffer to 256 MB in your test env and see
> if that makes the corruption go away?
>
> This could also be a hardware issue in your test env. If you run
> CheckIndex on the corrupt index does it always fail in the same way?
>
> Mike
>
> On Fri, Jan 14, 2011 at 6:43 AM, Li Li <fa...@gmail.com> wrote:
>> hi all,
>> we have confronted this problem 3 times when testing
>> The exception stack is
>> Exception in thread "Lucene Merge Thread #2"
>> org.apache.lucene.index.MergePolicy$MergeException:
>> org.apache.lucene.index.CorruptIndexException: docs out of order (7286
>> <= 7286 )
>> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:355)
>> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:319)
>> Caused by: org.apache.lucene.index.CorruptIndexException: docs out of
>> order (7286 <= 7286 )
>> at org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:75)
>> at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:880)
>> at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:818)
>> at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:756)
>> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:187)
>> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5354)
>> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4937)
>>
>> Or
>> Exception in thread "Lucene Merge Thread #0"
>> org.apache.lucene.index.MergePolicy$MergeException:
>> java.lang.ArrayIndexOutOfBoundsException: 330
>> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:355)
>> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:319)
>> Caused by: java.lang.ArrayIndexOutOfBoundsException: 330
>> at org.apache.lucene.util.BitVector.get(BitVector.java:102)
>> at org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:238)
>> at org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:168)
>> at org.apache.lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:98)
>> at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:870)
>> at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:818)
>> at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:756)
>> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:187)
>> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5354)
>>
>>
>> We did some minor modification based on lucene 2.9.1 and solr
>> 1.4.0. we modified frq file to store 4 bytes for the positions of the
>> term occured
>> in these document(Accessing full postions in prx is time consuming
>> that can't meed our needs). I can't tell it's our bug or lucene's own
>> bug.
>> I searched the mail list and found the mail "problem during index
>> merge" posted in 2010-10-21. It's similar to our case.
>> It seems the docList in frq file is wrongly stored. When Merging,
>> when it's decoded, the wrong docID many larger than maxDocs(BitVector
>> deletedDocs)
>> which cause the second exception. Or docID delta is less than 0(it
>> reads wrongly) which cause the first exception
>> we are still continueing testing to turn off our modification and
>> open infoStream in solr-config.xml
>>
>> We found a strange phenomenon. when we test, it sometimes hited
>> exceptions but in our production environment, it never hit any.
>> the hardware and software environments are the same. We checked
>> carefully and find the only difference is this line in solr-config.xml
>> <ramBufferSizeMB>32</ramBufferSizeMB> in testing environment
>> <ramBufferSizeMB>256</ramBufferSizeMB>in production environment
>> The indexed documents number for each machine is also roughly the
>> same. 10M+ documents.
>> I can't make sure the indice in production env are correct because
>> even there are some terms' docList are wrong, if the doc delta >0 and
>> don't have
>> some deleted documents, it will not hit the 2 exceptions.
>> The search results in production env and we don't find any strange results.
>>
>> Will when the ramBufferSizeMB is too small results in index corruption?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: CorruptIndexException when indexing
Posted by Michael McCandless <lu...@mikemccandless.com>.
Different ramBufferSizeMB during indexing should never cause corruption!
Can you try setting the ram buffer to 256 MB in your test env and see
if that makes the corruption go away?
This could also be a hardware issue in your test env. If you run
CheckIndex on the corrupt index does it always fail in the same way?
Mike
On Fri, Jan 14, 2011 at 6:43 AM, Li Li <fa...@gmail.com> wrote:
> hi all,
> we have confronted this problem 3 times when testing
> The exception stack is
> Exception in thread "Lucene Merge Thread #2"
> org.apache.lucene.index.MergePolicy$MergeException:
> org.apache.lucene.index.CorruptIndexException: docs out of order (7286
> <= 7286 )
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:355)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:319)
> Caused by: org.apache.lucene.index.CorruptIndexException: docs out of
> order (7286 <= 7286 )
> at org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:75)
> at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:880)
> at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:818)
> at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:756)
> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:187)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5354)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4937)
>
> Or
> Exception in thread "Lucene Merge Thread #0"
> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.ArrayIndexOutOfBoundsException: 330
> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:355)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:319)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 330
> at org.apache.lucene.util.BitVector.get(BitVector.java:102)
> at org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:238)
> at org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:168)
> at org.apache.lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:98)
> at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:870)
> at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:818)
> at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:756)
> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:187)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5354)
>
>
> We did some minor modification based on lucene 2.9.1 and solr
> 1.4.0. we modified frq file to store 4 bytes for the positions of the
> term occured
> in these document(Accessing full postions in prx is time consuming
> that can't meed our needs). I can't tell it's our bug or lucene's own
> bug.
> I searched the mail list and found the mail "problem during index
> merge" posted in 2010-10-21. It's similar to our case.
> It seems the docList in frq file is wrongly stored. When Merging,
> when it's decoded, the wrong docID many larger than maxDocs(BitVector
> deletedDocs)
> which cause the second exception. Or docID delta is less than 0(it
> reads wrongly) which cause the first exception
> we are still continueing testing to turn off our modification and
> open infoStream in solr-config.xml
>
> We found a strange phenomenon. when we test, it sometimes hited
> exceptions but in our production environment, it never hit any.
> the hardware and software environments are the same. We checked
> carefully and find the only difference is this line in solr-config.xml
> <ramBufferSizeMB>32</ramBufferSizeMB> in testing environment
> <ramBufferSizeMB>256</ramBufferSizeMB>in production environment
> The indexed documents number for each machine is also roughly the
> same. 10M+ documents.
> I can't make sure the indice in production env are correct because
> even there are some terms' docList are wrong, if the doc delta >0 and
> don't have
> some deleted documents, it will not hit the 2 exceptions.
> The search results in production env and we don't find any strange results.
>
> Will when the ramBufferSizeMB is too small results in index corruption?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org