You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Tyler V <ty...@gmail.com> on 2008/02/29 23:26:01 UTC
Corrupted Indexes Under Lucene 2.3 (and 2.3.1)
After upgrading to Lucene 2.3 (and subsequently 2.3.1), our
application has experienced sporadic index corruptions on our larger
(and more frequently updated) indexes. These indexes experienced no
corruptions under any prior version of Lucene (which we have been
using for several years).
The pattern of failure seems to be consistent. First, we receive an
exception like the following:
java.lang.IndexOutOfBoundsException: Index: 4788, Size: 4762
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at org.apache.lucene.index.DocumentsWriter$ThreadState.init(DocumentsWriter.java:749)
at org.apache.lucene.index.DocumentsWriter.getThreadState(DocumentsWriter.java:2391)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:2434)
at org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:2422)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1445)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1424)
at com.myapp.indexing.IndexerRunner.run(IndexerRunner.java:134)
at java.lang.Thread.run(Thread.java:619)
When we experience this error, we run a writer.flush() then a writer.close().
Then, we get this exception when trying to re-open the index:
org.apache.lucene.index.CorruptIndexException: doc counts differ for
segment _c2z13: fieldsReader shows 2 but segmentInfo shows 3
at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:313)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:197)
at org.apache.lucene.index.MultiSegmentReader.<init>(MultiSegmentReader.java:55)
at org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:75)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:636)
at org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:63)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:209)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:192)
at com.myapp.indexing.IndexerRunner.run(IndexerRunner.java:107)
at java.lang.Thread.run(Thread.java:619)
Running the check index application included with 2.3 enables us to
remove the bad documents from the index, but this workaround is less
than desirable. It would be greatly appreciated if anyone could shed
some light on our issue.
Regards,
Tyler
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Corrupted Indexes Under Lucene 2.3 (and 2.3.1)
Posted by Tyler V <ty...@gmail.com>.
Thanks for the reply Yonik.
Our workflow is as follows:
We build a very large document and put the document on a queue to be
added to our "complete" index. This queue is serviced by a separate
thread, which actually adds the document to the "complete" index.
Once the document has been placed on the queue, we pause for other
processing, then remove a bunch of fields from the document and place
this document on a separate queue to be serviced by another thread,
which will add the document to a "summary" index.
The concurrency problem here is clear, as it is possible for a
document's fields to be modified while the document is being added to
the "complete" index. From what you and Mike have mentioned, I
believe this simultaneous modification is causing the first exception.
The assumption that was being made in my algorithm is that the
"complete" index queue would be serviced before the document's fields
are removed for the "summary" index. This assumption is not correct.
The solution that I am testing now is to "clone" the original document
before any fields are removed. Thus the original document will not be
modified and the concurrency issues will be avoided.
Given that this problem only occurred around once a week, it will take
me a while before I can report success. But from reading the posts
from Mike and yourself, it seems that this is indeed the cause of the
corruption.
Thanks again for you insights, I will report back with my results.
Tyler
On Fri, Feb 29, 2008 at 6:01 PM, Yonik Seeley <yo...@apache.org> wrote:
>
> On Fri, Feb 29, 2008 at 7:05 PM, Tyler V <ty...@gmail.com> wrote:
> > Mike -- Thanks so much for the prompt reply.
> >
> > You are right, we are accessing these documents with multiple threads
> > (and have always been). However, I am wondering if the increased
> > indexing speed in 2.3 has revealed a hidden concurrency issue.
>
> You are modifying the documents from multiple threads?
>
> My fault... I removed the synchronization on Document (changed from
> Vector to ArrayList). It was never guaranteed to be thread-safe for
> modification, and almost never makes sense without external
> synchronization anyway.
>
> If you really need to modify a single document from multiple threads,
> please synchronize.
> That explains the first exception, but no the second. I assume you
> aren't still changing the document while it's being indexed?
> It appears as if the original exception causes corruption.
>
> -Yonik
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Corrupted Indexes Under Lucene 2.3 (and 2.3.1)
Posted by Michael McCandless <lu...@mikemccandless.com>.
Note that there are actually two concurrency issues to guard against
here:
* Document itself cannot be changed (fields added or removed) from
multiple threads without external synchronization.
* Document cannot be changed from one thread while another thread is
calling writer.addDocument. It's this case that's causing your
1st exception.
Please report back on whether adding correct external synchronization
resolves this!
I can explain why the original exception leads to index corruption:
IndexWriter incorrectly increments the buffered document count even
when there is an exception and the document was not in fact added to
the segment. I will fix this so an exception when init'ing a document
doesn't then lead to index corruption...
Thanks for reporting this Tyler!
Mike
Yonik Seeley wrote:
> On Fri, Feb 29, 2008 at 7:05 PM, Tyler V <ty...@gmail.com> wrote:
>> Mike -- Thanks so much for the prompt reply.
>>
>> You are right, we are accessing these documents with multiple
>> threads
>> (and have always been). However, I am wondering if the increased
>> indexing speed in 2.3 has revealed a hidden concurrency issue.
>
> You are modifying the documents from multiple threads?
>
> My fault... I removed the synchronization on Document (changed from
> Vector to ArrayList). It was never guaranteed to be thread-safe for
> modification, and almost never makes sense without external
> synchronization anyway.
>
> If you really need to modify a single document from multiple threads,
> please synchronize.
> That explains the first exception, but no the second. I assume you
> aren't still changing the document while it's being indexed?
> It appears as if the original exception causes corruption.
>
> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Corrupted Indexes Under Lucene 2.3 (and 2.3.1)
Posted by Yonik Seeley <yo...@apache.org>.
On Fri, Feb 29, 2008 at 7:05 PM, Tyler V <ty...@gmail.com> wrote:
> Mike -- Thanks so much for the prompt reply.
>
> You are right, we are accessing these documents with multiple threads
> (and have always been). However, I am wondering if the increased
> indexing speed in 2.3 has revealed a hidden concurrency issue.
You are modifying the documents from multiple threads?
My fault... I removed the synchronization on Document (changed from
Vector to ArrayList). It was never guaranteed to be thread-safe for
modification, and almost never makes sense without external
synchronization anyway.
If you really need to modify a single document from multiple threads,
please synchronize.
That explains the first exception, but no the second. I assume you
aren't still changing the document while it's being indexed?
It appears as if the original exception causes corruption.
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Corrupted Indexes Under Lucene 2.3 (and 2.3.1)
Posted by Tyler V <ty...@gmail.com>.
Mike -- Thanks so much for the prompt reply.
You are right, we are accessing these documents with multiple threads
(and have always been). However, I am wondering if the increased
indexing speed in 2.3 has revealed a hidden concurrency issue.
I am going to add in some additional concurrency checks, if I have
questions I will post another question to the list.
Thanks again for the reply.
Tyler
On Fri, Feb 29, 2008 at 2:46 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
>
> Not good! (I'm sorry).
>
> That first exception is worrisome. It's the root cause here.
>
> Can you describe your documents? That exception, if I'm reading it
> right, seems to imply that you have documents with 4762 fields. Is
> that right?
>
> Are you using multiple threads? Is it possible that you cal
> addDocument with a given document, but another thread removing fields
> from that document (or, removing elements from the List returned by
> Document.getFields())? That's the only way I can explain that first
> exception.
>
> Mike
>
>
>
> Tyler V wrote:
>
> > After upgrading to Lucene 2.3 (and subsequently 2.3.1), our
> > application has experienced sporadic index corruptions on our larger
> > (and more frequently updated) indexes. These indexes experienced no
> > corruptions under any prior version of Lucene (which we have been
> > using for several years).
> >
> > The pattern of failure seems to be consistent. First, we receive an
> > exception like the following:
> >
> > java.lang.IndexOutOfBoundsException: Index: 4788, Size: 4762
> > at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> > at java.util.ArrayList.get(ArrayList.java:322)
> > at org.apache.lucene.index.DocumentsWriter$ThreadState.init
> > (DocumentsWriter.java:749)
> > at org.apache.lucene.index.DocumentsWriter.getThreadState
> > (DocumentsWriter.java:2391)
> > at org.apache.lucene.index.DocumentsWriter.updateDocument
> > (DocumentsWriter.java:2434)
> > at org.apache.lucene.index.DocumentsWriter.addDocument
> > (DocumentsWriter.java:2422)
> > at org.apache.lucene.index.IndexWriter.addDocument
> > (IndexWriter.java:1445)
> > at org.apache.lucene.index.IndexWriter.addDocument
> > (IndexWriter.java:1424)
> > at com.myapp.indexing.IndexerRunner.run(IndexerRunner.java:
> > 134)
> > at java.lang.Thread.run(Thread.java:619)
> >
> > When we experience this error, we run a writer.flush() then a
> > writer.close().
> >
> > Then, we get this exception when trying to re-open the index:
> >
> > org.apache.lucene.index.CorruptIndexException: doc counts differ for
> > segment _c2z13: fieldsReader shows 2 but segmentInfo shows 3
> > at org.apache.lucene.index.SegmentReader.initialize
> > (SegmentReader.java:313)
> > at org.apache.lucene.index.SegmentReader.get
> > (SegmentReader.java:262)
> > at org.apache.lucene.index.SegmentReader.get
> > (SegmentReader.java:197)
> > at org.apache.lucene.index.MultiSegmentReader.<init>
> > (MultiSegmentReader.java:55)
> > at org.apache.lucene.index.DirectoryIndexReader$1.doBody
> > (DirectoryIndexReader.java:75)
> > at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run
> > (SegmentInfos.java:636)
> > at org.apache.lucene.index.DirectoryIndexReader.open
> > (DirectoryIndexReader.java:63)
> > at org.apache.lucene.index.IndexReader.open
> > (IndexReader.java:209)
> > at org.apache.lucene.index.IndexReader.open
> > (IndexReader.java:192)
> > at com.myapp.indexing.IndexerRunner.run(IndexerRunner.java:
> > 107)
> > at java.lang.Thread.run(Thread.java:619)
> >
> > Running the check index application included with 2.3 enables us to
> > remove the bad documents from the index, but this workaround is less
> > than desirable. It would be greatly appreciated if anyone could shed
> > some light on our issue.
> >
> > Regards,
> >
> > Tyler
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Corrupted Indexes Under Lucene 2.3 (and 2.3.1)
Posted by Michael McCandless <lu...@mikemccandless.com>.
Not good! (I'm sorry).
That first exception is worrisome. It's the root cause here.
Can you describe your documents? That exception, if I'm reading it
right, seems to imply that you have documents with 4762 fields. Is
that right?
Are you using multiple threads? Is it possible that you cal
addDocument with a given document, but another thread removing fields
from that document (or, removing elements from the List returned by
Document.getFields())? That's the only way I can explain that first
exception.
Mike
Tyler V wrote:
> After upgrading to Lucene 2.3 (and subsequently 2.3.1), our
> application has experienced sporadic index corruptions on our larger
> (and more frequently updated) indexes. These indexes experienced no
> corruptions under any prior version of Lucene (which we have been
> using for several years).
>
> The pattern of failure seems to be consistent. First, we receive an
> exception like the following:
>
> java.lang.IndexOutOfBoundsException: Index: 4788, Size: 4762
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.get(ArrayList.java:322)
> at org.apache.lucene.index.DocumentsWriter$ThreadState.init
> (DocumentsWriter.java:749)
> at org.apache.lucene.index.DocumentsWriter.getThreadState
> (DocumentsWriter.java:2391)
> at org.apache.lucene.index.DocumentsWriter.updateDocument
> (DocumentsWriter.java:2434)
> at org.apache.lucene.index.DocumentsWriter.addDocument
> (DocumentsWriter.java:2422)
> at org.apache.lucene.index.IndexWriter.addDocument
> (IndexWriter.java:1445)
> at org.apache.lucene.index.IndexWriter.addDocument
> (IndexWriter.java:1424)
> at com.myapp.indexing.IndexerRunner.run(IndexerRunner.java:
> 134)
> at java.lang.Thread.run(Thread.java:619)
>
> When we experience this error, we run a writer.flush() then a
> writer.close().
>
> Then, we get this exception when trying to re-open the index:
>
> org.apache.lucene.index.CorruptIndexException: doc counts differ for
> segment _c2z13: fieldsReader shows 2 but segmentInfo shows 3
> at org.apache.lucene.index.SegmentReader.initialize
> (SegmentReader.java:313)
> at org.apache.lucene.index.SegmentReader.get
> (SegmentReader.java:262)
> at org.apache.lucene.index.SegmentReader.get
> (SegmentReader.java:197)
> at org.apache.lucene.index.MultiSegmentReader.<init>
> (MultiSegmentReader.java:55)
> at org.apache.lucene.index.DirectoryIndexReader$1.doBody
> (DirectoryIndexReader.java:75)
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run
> (SegmentInfos.java:636)
> at org.apache.lucene.index.DirectoryIndexReader.open
> (DirectoryIndexReader.java:63)
> at org.apache.lucene.index.IndexReader.open
> (IndexReader.java:209)
> at org.apache.lucene.index.IndexReader.open
> (IndexReader.java:192)
> at com.myapp.indexing.IndexerRunner.run(IndexerRunner.java:
> 107)
> at java.lang.Thread.run(Thread.java:619)
>
> Running the check index application included with 2.3 enables us to
> remove the bad documents from the index, but this workaround is less
> than desirable. It would be greatly appreciated if anyone could shed
> some light on our issue.
>
> Regards,
>
> Tyler
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org