You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Ian Lea <ia...@gmail.com> on 2008/03/18 12:38:01 UTC
CorruptIndexException with some versions of java
Hi
When bulk loading into a new index I'm seeing this exception
Exception in thread "Thread-1"
org.apache.lucene.index.MergePolicy$MergeException:
org.apache.lucene.index.CorruptIndexException: doc counts differ for
segment _4l: fieldsReader shows 67861 but segmentInfo shows 67862
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:271)
Caused by: org.apache.lucene.index.CorruptIndexException: doc counts
differ for segment _4l: fieldsReader shows 67861 but segmentInfo shows
67862
at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:313)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3093)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240)
when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux, with
lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from yesterday.
With java version 1.6.0_03-b05 things work fine.
The exception happens a few hundred thousand documents into the load.
A different program updating a different index with different data on
a different server gave a similar error on version 1.6.0_05-b13 and
lucene 2.3.0.
Any ideas? Is this maybe a known issue or am I missing something obvious?
--
Ian.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: CorruptIndexException with some versions of java
Posted by Michael McCandless <lu...@mikemccandless.com>.
Spookily, I believe this is in fact some sort of JVM hotspot compiler
bug. Adding -Xbatch (forces up-front compilation) prevents (works
around) the problem.
I added a counter, to IndexOutput, of total bytes written, and then
in SegmentMerger.mergeFields, I added an assert that the #docs
written through FieldsWriter matched the number of bytes written to
the fdx file. Insanely, that assert would fail when the issue was
hit, which is just not possible from the code. It's as if when the
BG compilation finished and the native method got swapped into the
JVM, it somehow "missed" a call to IndexOutput.writeLong.
I committed some of these asserts into SegmentMerger on both 2.3 &
trunk.
I'll respond on the java-user thread to bring closure here.
Mike
Yonik Seeley wrote:
> On Wed, Mar 19, 2008 at 1:18 PM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>> I'm CC'ing java-dev. To sum up: sometimes when we merge fields, the
>> fdx file ends up exactly one document too short. In adding numerous
>> asserts around this code in SegmentMerger.java, insanely, somehow
>> the
>> call to indexStream.writeLong fails to actually "happen" when we
>> call
>> FieldsWriter.addDocument, even though from looking at the code I see
>> no way to explain that. As hard as it is to believe, it really is
>> looking like a strange JVM bug at this point...
>
> JDK 1.6_04 was a major upgrade of hotspot.
> Is there a way to dump the compiled machine code that hotspot
> eventually generates for that method?
>
> -Yonik
>
>
>> Has anyone else seen any odd behavior on update 4 or 5 of JDK 1.6?
>> The issue does not happen on previous updates of JDK 1.6.
>>
>> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: CorruptIndexException with some versions of java
Posted by Yonik Seeley <yo...@apache.org>.
On Wed, Mar 19, 2008 at 1:18 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> I'm CC'ing java-dev. To sum up: sometimes when we merge fields, the
> fdx file ends up exactly one document too short. In adding numerous
> asserts around this code in SegmentMerger.java, insanely, somehow the
> call to indexStream.writeLong fails to actually "happen" when we call
> FieldsWriter.addDocument, even though from looking at the code I see
> no way to explain that. As hard as it is to believe, it really is
> looking like a strange JVM bug at this point...
JDK 1.6_04 was a major upgrade of hotspot.
Is there a way to dump the compiled machine code that hotspot
eventually generates for that method?
-Yonik
> Has anyone else seen any odd behavior on update 4 or 5 of JDK 1.6?
> The issue does not happen on previous updates of JDK 1.6.
>
> Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: CorruptIndexException with some versions of java
Posted by Michael McCandless <lu...@mikemccandless.com>.
Hi Ian,
Can you try this patch? Grasping at straws at this point, trying to
isolate where the JVM fails us...
I'm CC'ing java-dev. To sum up: sometimes when we merge fields, the
fdx file ends up exactly one document too short. In adding numerous
asserts around this code in SegmentMerger.java, insanely, somehow the
call to indexStream.writeLong fails to actually "happen" when we call
FieldsWriter.addDocument, even though from looking at the code I see
no way to explain that. As hard as it is to believe, it really is
looking like a strange JVM bug at this point...
Has anyone else seen any odd behavior on update 4 or 5 of JDK 1.6?
The issue does not happen on previous updates of JDK 1.6.
Mike
Re: CorruptIndexException with some versions of java
Posted by Michael McCandless <lu...@mikemccandless.com>.
Ian can you attach your version of SegmentMerger.java? Somehow my
lines are off from yours.
Mike
Ian Lea wrote:
> Mike
>
>
> Latest patch produces similar exception:
>
> Exception in thread "Lucene Merge Thread #0"
> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.AssertionError: after mergeFields: fdx size mismatch: 65184
> docs vs 521464 length in bytes of _c9.fdx
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(
> ConcurrentMergeScheduler.java:320)
> at org.apache.lucene.index.ConcurrentMergeScheduler
> $MergeThread.run(ConcurrentMergeScheduler.java:297)
> Caused by: java.lang.AssertionError: after mergeFields: fdx size
> mismatch: 65184 docs vs 521464 length in bytes of _c9.fdx
> at org.apache.lucene.index.SegmentMerger.mergeFields
> (SegmentMerger.java:347)
> at org.apache.lucene.index.SegmentMerger.merge
> (SegmentMerger.java:133)
> at org.apache.lucene.index.IndexWriter.mergeMiddle
> (IndexWriter.java:3852)
> at org.apache.lucene.index.IndexWriter.merge
> (IndexWriter.java:3504)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge
> (ConcurrentMergeScheduler.java:211)
> at org.apache.lucene.index.ConcurrentMergeScheduler
> $MergeThread.run(ConcurrentMergeScheduler.java:266)
>
> Latest infostream attached.
>
>
> --
> Ian.
>
>
> On Tue, Mar 18, 2008 at 6:05 PM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>>
>> Hi Ian,
>>
>> Sheesh that's odd. The SegmentMerger produced an .fdx file that is
>> one document too short.
>>
>> Can you run with this patch now, again applied to head of 2.3
>> branch? I just added another assert inside the loop that does the
>> field merging.
>>
>> I will scrutinize this code...
>>
>> Mike
>>
>>
>>
>>
>> Ian Lea wrote:
>>> Mike
>>>
>>>
>>> Patch applied and test re-run and picked up an assertion error this
>>> time:
>>>
>>> Exception in thread "Lucene Merge Thread #0"
>>> org.apache.lucene.index.MergePolicy$MergeException:
>>> java.lang.AssertionError: after mergeFields: fdx size mismatch:
>>> 72357
>>> docs vs 578848 length in bytes of _3o.fdx
>>> at
>>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeExceptio
>>> n(
>>> ConcurrentMergeScheduler.java:320)
>>> at org.apache.lucene.index.ConcurrentMergeScheduler
>>> $MergeThread.run(ConcurrentMergeScheduler.java:297)
>>> Caused by: java.lang.AssertionError: after mergeFields: fdx size
>>> mismatch: 72357 docs vs 578848 length in bytes of _3o.fdx
>>> at org.apache.lucene.index.SegmentMerger.mergeFields
>>> (SegmentMerger.java:342)
>>> at org.apache.lucene.index.SegmentMerger.merge
>>> (SegmentMerger.java:133)
>>> at org.apache.lucene.index.IndexWriter.mergeMiddle
>>> (IndexWriter.java:3852)
>>> at org.apache.lucene.index.IndexWriter.merge
>>> (IndexWriter.java:3504)
>>> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge
>>> (ConcurrentMergeScheduler.java:211)
>>> at org.apache.lucene.index.ConcurrentMergeScheduler
>>> $MergeThread.run(ConcurrentMergeScheduler.java:266)
>>>
>>> The infostream output is attached. Since this email is to you
>>> and the
>>> list it should make it to you.
>>>
>>>
>>>
>>> Yonik: I haven't been able to make TestStressIndexing2 fail.
>>>
>>>
>>> --
>>> Ian.
>>>
>>>
>>> On Tue, Mar 18, 2008 at 4:19 PM, Michael McCandless
>>> <lu...@mikemccandless.com> wrote:
>>>>
>>>> Ian,
>>>>
>>>> Could you apply the attached patch applied to the head of the 2.3
>>>> branch?
>>>>
>>>> It only adds more asserts, to try to pinpoint where exactly this
>>>> corruption starts.
>>>>
>>>> Then, re-run the test with asserts enabled and infoStream
>>>> turned on
>>>> and post back. Thanks.
>>>>
>>>> Mike
>>>>
>>>>
>>>>
>>>>
>>>> Ian Lea wrote:
>>>>
>>>>> It's failed on servers running SuSE 10.0 and 8.2 (ancient!)
>>>>>
>>>>> $ uname -a shows
>>>>> Linux phoebe 2.6.13-15-smp #1 SMP Tue Sep 13 14:56:15 UTC 2005
>>>>> x86_64
>>>>> x86_64 x86_64 GNU/Linux
>>>>>
>>>>> and
>>>>>
>>>>> Linux phobos 2.4.20-64GB-SMP #1 SMP Mon Mar 17 17:56:03 UTC 2003
>>>>> i686
>>>>> unknown unknown GNU/Linux
>>>>>
>>>>> The first one has a 2.8Ghz Intel CPU, don't know about the second.
>>>>>
>>>>>
>>>>> I'll try and run the stress test.
>>>>>
>>>>>
>>>>> --
>>>>> Ian.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Mar 18, 2008 at 2:17 PM, Yonik Seeley <yo...@apache.org>
>>>>> wrote:
>>>>>>
>>>>>> On Tue, Mar 18, 2008 at 7:38 AM, Ian Lea <ia...@gmail.com>
>>>>>> wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>>
>>>>>>> When bulk loading into a new index I'm seeing this exception
>>>>>>>
>>>>>>> Exception in thread "Thread-1"
>>>>>>> org.apache.lucene.index.MergePolicy$MergeException:
>>>>>>> org.apache.lucene.index.CorruptIndexException: doc counts
>>>>>>> differ
>>>>>>> for
>>>>>>> segment _4l: fieldsReader shows 67861 but segmentInfo shows
>>>>>>> 67862
>>>>>>> at org.apache.lucene.index.ConcurrentMergeScheduler
>>>>>>> $MergeThread.run(ConcurrentMergeScheduler.java:271)
>>>>>>> Caused by: org.apache.lucene.index.CorruptIndexException: doc
>>>>>>> counts
>>>>>>> differ for segment _4l: fieldsReader shows 67861 but
>>>>>>> segmentInfo
>>>>>>> shows
>>>>>>> 67862
>>>>>>> at org.apache.lucene.index.SegmentReader.initialize
>>>>>>> (SegmentReader.java:313)
>>>>>>> at org.apache.lucene.index.SegmentReader.get
>>>>>>> (SegmentReader.java:262)
>>>>>>> at org.apache.lucene.index.SegmentReader.get
>>>>>>> (SegmentReader.java:221)
>>>>>>> at org.apache.lucene.index.IndexWriter.mergeMiddle
>>>>>>> (IndexWriter.java:3093)
>>>>>>> at org.apache.lucene.index.IndexWriter.merge
>>>>>>> (IndexWriter.java:2834)
>>>>>>> at org.apache.lucene.index.ConcurrentMergeScheduler
>>>>>>> $MergeThread.run(ConcurrentMergeScheduler.java:240)
>>>>>>>
>>>>>>> when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux,
>>>>>>> with
>>>>>>> lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from
>>>>>>> yesterday.
>>>>>>>
>>>>>>> With java version 1.6.0_03-b05 things work fine.
>>>>>>>
>>>>>>> The exception happens a few hundred thousand documents into the
>>>>>>> load.
>>>>>>>
>>>>>>> A different program updating a different index with different
>>>>>>> data on
>>>>>>> a different server gave a similar error on version 1.6.0_05-
>>>>>>> b13 and
>>>>>>> lucene 2.3.0.
>>>>>>>
>>>>>>> Any ideas? Is this maybe a known issue or am I missing
>>>>>>> something obvious?
>>>>>>
>>>>>> My guess is perhaps a thread safety bug, more likely in Lucene
>>>>>> indexing code (less likely in the JVM or specific libc).
>>>>>>
>>>>>> What Linux version are you using?
>>>>>> What hardware are you running on (specifically, the CPU)?
>>>>>>
>>>>>> If possible, it would be great if you could check out Lucene
>>>>>> trunk,
>>>>>> crank up the iterations by modifying the TestStressIndexing2 and
>>>>>> maybe
>>>>>> fiddle with some of the other parameters in
>>>>>> TestStressIndexing2.testMultiConfig(), and see if you can get
>>>>>> it to
>>>>>> fail.
>>>>>>
>>>>>>
>>>>>> -Yonik
>>>>>>
>>>>>>
>>>>>> -----------------------------------------------------------------
>>>>>> --
>>>>>> --
>>>>>>
>>>>>>
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-
>>>>>> help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> --
>>>>> -
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>
>>>>
>>>>
>>>> <infostream.zip>
>>
>>
>>
>> <infostream.zip>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: CorruptIndexException with some versions of java
Posted by Michael McCandless <lu...@mikemccandless.com>.
Just to bring closure here: this in fact looks like some sort of JVM
hotspot compiler issue, as best we can tell.
Running java with -Xbatch (forces up front compilation) prevents
(works around) the issue.
I've committed some additional assertions to the particular Lucene
code (merging of stored fields) that was tripping this issue, which
will be in 2.3.2 and 2.4.
Mike
Ian Lea wrote:
> Mike
>
>
> Latest patch produces similar exception:
>
> Exception in thread "Lucene Merge Thread #0"
> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.AssertionError: after mergeFields: fdx size mismatch: 65184
> docs vs 521464 length in bytes of _c9.fdx
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(
> ConcurrentMergeScheduler.java:320)
> at org.apache.lucene.index.ConcurrentMergeScheduler
> $MergeThread.run(ConcurrentMergeScheduler.java:297)
> Caused by: java.lang.AssertionError: after mergeFields: fdx size
> mismatch: 65184 docs vs 521464 length in bytes of _c9.fdx
> at org.apache.lucene.index.SegmentMerger.mergeFields
> (SegmentMerger.java:347)
> at org.apache.lucene.index.SegmentMerger.merge
> (SegmentMerger.java:133)
> at org.apache.lucene.index.IndexWriter.mergeMiddle
> (IndexWriter.java:3852)
> at org.apache.lucene.index.IndexWriter.merge
> (IndexWriter.java:3504)
> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge
> (ConcurrentMergeScheduler.java:211)
> at org.apache.lucene.index.ConcurrentMergeScheduler
> $MergeThread.run(ConcurrentMergeScheduler.java:266)
>
> Latest infostream attached.
>
>
> --
> Ian.
>
>
> On Tue, Mar 18, 2008 at 6:05 PM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>>
>> Hi Ian,
>>
>> Sheesh that's odd. The SegmentMerger produced an .fdx file that is
>> one document too short.
>>
>> Can you run with this patch now, again applied to head of 2.3
>> branch? I just added another assert inside the loop that does the
>> field merging.
>>
>> I will scrutinize this code...
>>
>> Mike
>>
>>
>>
>>
>> Ian Lea wrote:
>>> Mike
>>>
>>>
>>> Patch applied and test re-run and picked up an assertion error this
>>> time:
>>>
>>> Exception in thread "Lucene Merge Thread #0"
>>> org.apache.lucene.index.MergePolicy$MergeException:
>>> java.lang.AssertionError: after mergeFields: fdx size mismatch:
>>> 72357
>>> docs vs 578848 length in bytes of _3o.fdx
>>> at
>>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeExceptio
>>> n(
>>> ConcurrentMergeScheduler.java:320)
>>> at org.apache.lucene.index.ConcurrentMergeScheduler
>>> $MergeThread.run(ConcurrentMergeScheduler.java:297)
>>> Caused by: java.lang.AssertionError: after mergeFields: fdx size
>>> mismatch: 72357 docs vs 578848 length in bytes of _3o.fdx
>>> at org.apache.lucene.index.SegmentMerger.mergeFields
>>> (SegmentMerger.java:342)
>>> at org.apache.lucene.index.SegmentMerger.merge
>>> (SegmentMerger.java:133)
>>> at org.apache.lucene.index.IndexWriter.mergeMiddle
>>> (IndexWriter.java:3852)
>>> at org.apache.lucene.index.IndexWriter.merge
>>> (IndexWriter.java:3504)
>>> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge
>>> (ConcurrentMergeScheduler.java:211)
>>> at org.apache.lucene.index.ConcurrentMergeScheduler
>>> $MergeThread.run(ConcurrentMergeScheduler.java:266)
>>>
>>> The infostream output is attached. Since this email is to you
>>> and the
>>> list it should make it to you.
>>>
>>>
>>>
>>> Yonik: I haven't been able to make TestStressIndexing2 fail.
>>>
>>>
>>> --
>>> Ian.
>>>
>>>
>>> On Tue, Mar 18, 2008 at 4:19 PM, Michael McCandless
>>> <lu...@mikemccandless.com> wrote:
>>>>
>>>> Ian,
>>>>
>>>> Could you apply the attached patch applied to the head of the 2.3
>>>> branch?
>>>>
>>>> It only adds more asserts, to try to pinpoint where exactly this
>>>> corruption starts.
>>>>
>>>> Then, re-run the test with asserts enabled and infoStream
>>>> turned on
>>>> and post back. Thanks.
>>>>
>>>> Mike
>>>>
>>>>
>>>>
>>>>
>>>> Ian Lea wrote:
>>>>
>>>>> It's failed on servers running SuSE 10.0 and 8.2 (ancient!)
>>>>>
>>>>> $ uname -a shows
>>>>> Linux phoebe 2.6.13-15-smp #1 SMP Tue Sep 13 14:56:15 UTC 2005
>>>>> x86_64
>>>>> x86_64 x86_64 GNU/Linux
>>>>>
>>>>> and
>>>>>
>>>>> Linux phobos 2.4.20-64GB-SMP #1 SMP Mon Mar 17 17:56:03 UTC 2003
>>>>> i686
>>>>> unknown unknown GNU/Linux
>>>>>
>>>>> The first one has a 2.8Ghz Intel CPU, don't know about the second.
>>>>>
>>>>>
>>>>> I'll try and run the stress test.
>>>>>
>>>>>
>>>>> --
>>>>> Ian.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Mar 18, 2008 at 2:17 PM, Yonik Seeley <yo...@apache.org>
>>>>> wrote:
>>>>>>
>>>>>> On Tue, Mar 18, 2008 at 7:38 AM, Ian Lea <ia...@gmail.com>
>>>>>> wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>>
>>>>>>> When bulk loading into a new index I'm seeing this exception
>>>>>>>
>>>>>>> Exception in thread "Thread-1"
>>>>>>> org.apache.lucene.index.MergePolicy$MergeException:
>>>>>>> org.apache.lucene.index.CorruptIndexException: doc counts
>>>>>>> differ
>>>>>>> for
>>>>>>> segment _4l: fieldsReader shows 67861 but segmentInfo shows
>>>>>>> 67862
>>>>>>> at org.apache.lucene.index.ConcurrentMergeScheduler
>>>>>>> $MergeThread.run(ConcurrentMergeScheduler.java:271)
>>>>>>> Caused by: org.apache.lucene.index.CorruptIndexException: doc
>>>>>>> counts
>>>>>>> differ for segment _4l: fieldsReader shows 67861 but
>>>>>>> segmentInfo
>>>>>>> shows
>>>>>>> 67862
>>>>>>> at org.apache.lucene.index.SegmentReader.initialize
>>>>>>> (SegmentReader.java:313)
>>>>>>> at org.apache.lucene.index.SegmentReader.get
>>>>>>> (SegmentReader.java:262)
>>>>>>> at org.apache.lucene.index.SegmentReader.get
>>>>>>> (SegmentReader.java:221)
>>>>>>> at org.apache.lucene.index.IndexWriter.mergeMiddle
>>>>>>> (IndexWriter.java:3093)
>>>>>>> at org.apache.lucene.index.IndexWriter.merge
>>>>>>> (IndexWriter.java:2834)
>>>>>>> at org.apache.lucene.index.ConcurrentMergeScheduler
>>>>>>> $MergeThread.run(ConcurrentMergeScheduler.java:240)
>>>>>>>
>>>>>>> when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux,
>>>>>>> with
>>>>>>> lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from
>>>>>>> yesterday.
>>>>>>>
>>>>>>> With java version 1.6.0_03-b05 things work fine.
>>>>>>>
>>>>>>> The exception happens a few hundred thousand documents into the
>>>>>>> load.
>>>>>>>
>>>>>>> A different program updating a different index with different
>>>>>>> data on
>>>>>>> a different server gave a similar error on version 1.6.0_05-
>>>>>>> b13 and
>>>>>>> lucene 2.3.0.
>>>>>>>
>>>>>>> Any ideas? Is this maybe a known issue or am I missing
>>>>>>> something obvious?
>>>>>>
>>>>>> My guess is perhaps a thread safety bug, more likely in Lucene
>>>>>> indexing code (less likely in the JVM or specific libc).
>>>>>>
>>>>>> What Linux version are you using?
>>>>>> What hardware are you running on (specifically, the CPU)?
>>>>>>
>>>>>> If possible, it would be great if you could check out Lucene
>>>>>> trunk,
>>>>>> crank up the iterations by modifying the TestStressIndexing2 and
>>>>>> maybe
>>>>>> fiddle with some of the other parameters in
>>>>>> TestStressIndexing2.testMultiConfig(), and see if you can get
>>>>>> it to
>>>>>> fail.
>>>>>>
>>>>>>
>>>>>> -Yonik
>>>>>>
>>>>>>
>>>>>> -----------------------------------------------------------------
>>>>>> --
>>>>>> --
>>>>>>
>>>>>>
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-
>>>>>> help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> --
>>>>> -
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>
>>>>
>>>>
>>>> <infostream.zip>
>>
>>
>>
>> <infostream.zip>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: CorruptIndexException with some versions of java
Posted by Ian Lea <ia...@gmail.com>.
Mike
Latest patch produces similar exception:
Exception in thread "Lucene Merge Thread #0"
org.apache.lucene.index.MergePolicy$MergeException:
java.lang.AssertionError: after mergeFields: fdx size mismatch: 65184
docs vs 521464 length in bytes of _c9.fdx
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:320)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:297)
Caused by: java.lang.AssertionError: after mergeFields: fdx size
mismatch: 65184 docs vs 521464 length in bytes of _c9.fdx
at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:347)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:133)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3852)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3504)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:211)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:266)
Latest infostream attached.
--
Ian.
On Tue, Mar 18, 2008 at 6:05 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
>
> Hi Ian,
>
> Sheesh that's odd. The SegmentMerger produced an .fdx file that is
> one document too short.
>
> Can you run with this patch now, again applied to head of 2.3
> branch? I just added another assert inside the loop that does the
> field merging.
>
> I will scrutinize this code...
>
> Mike
>
>
>
>
> Ian Lea wrote:
> > Mike
> >
> >
> > Patch applied and test re-run and picked up an assertion error this
> > time:
> >
> > Exception in thread "Lucene Merge Thread #0"
> > org.apache.lucene.index.MergePolicy$MergeException:
> > java.lang.AssertionError: after mergeFields: fdx size mismatch: 72357
> > docs vs 578848 length in bytes of _3o.fdx
> > at
> > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(
> > ConcurrentMergeScheduler.java:320)
> > at org.apache.lucene.index.ConcurrentMergeScheduler
> > $MergeThread.run(ConcurrentMergeScheduler.java:297)
> > Caused by: java.lang.AssertionError: after mergeFields: fdx size
> > mismatch: 72357 docs vs 578848 length in bytes of _3o.fdx
> > at org.apache.lucene.index.SegmentMerger.mergeFields
> > (SegmentMerger.java:342)
> > at org.apache.lucene.index.SegmentMerger.merge
> > (SegmentMerger.java:133)
> > at org.apache.lucene.index.IndexWriter.mergeMiddle
> > (IndexWriter.java:3852)
> > at org.apache.lucene.index.IndexWriter.merge
> > (IndexWriter.java:3504)
> > at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge
> > (ConcurrentMergeScheduler.java:211)
> > at org.apache.lucene.index.ConcurrentMergeScheduler
> > $MergeThread.run(ConcurrentMergeScheduler.java:266)
> >
> > The infostream output is attached. Since this email is to you and the
> > list it should make it to you.
> >
> >
> >
> > Yonik: I haven't been able to make TestStressIndexing2 fail.
> >
> >
> > --
> > Ian.
> >
> >
> > On Tue, Mar 18, 2008 at 4:19 PM, Michael McCandless
> > <lu...@mikemccandless.com> wrote:
> >>
> >> Ian,
> >>
> >> Could you apply the attached patch applied to the head of the 2.3
> >> branch?
> >>
> >> It only adds more asserts, to try to pinpoint where exactly this
> >> corruption starts.
> >>
> >> Then, re-run the test with asserts enabled and infoStream turned on
> >> and post back. Thanks.
> >>
> >> Mike
> >>
> >>
> >>
> >>
> >> Ian Lea wrote:
> >>
> >>> It's failed on servers running SuSE 10.0 and 8.2 (ancient!)
> >>>
> >>> $ uname -a shows
> >>> Linux phoebe 2.6.13-15-smp #1 SMP Tue Sep 13 14:56:15 UTC 2005
> >>> x86_64
> >>> x86_64 x86_64 GNU/Linux
> >>>
> >>> and
> >>>
> >>> Linux phobos 2.4.20-64GB-SMP #1 SMP Mon Mar 17 17:56:03 UTC 2003
> >>> i686
> >>> unknown unknown GNU/Linux
> >>>
> >>> The first one has a 2.8Ghz Intel CPU, don't know about the second.
> >>>
> >>>
> >>> I'll try and run the stress test.
> >>>
> >>>
> >>> --
> >>> Ian.
> >>>
> >>>
> >>>
> >>> On Tue, Mar 18, 2008 at 2:17 PM, Yonik Seeley <yo...@apache.org>
> >>> wrote:
> >>>>
> >>>> On Tue, Mar 18, 2008 at 7:38 AM, Ian Lea <ia...@gmail.com> wrote:
> >>>>> Hi
> >>>>>
> >>>>>
> >>>>> When bulk loading into a new index I'm seeing this exception
> >>>>>
> >>>>> Exception in thread "Thread-1"
> >>>>> org.apache.lucene.index.MergePolicy$MergeException:
> >>>>> org.apache.lucene.index.CorruptIndexException: doc counts differ
> >>>>> for
> >>>>> segment _4l: fieldsReader shows 67861 but segmentInfo shows 67862
> >>>>> at org.apache.lucene.index.ConcurrentMergeScheduler
> >>>>> $MergeThread.run(ConcurrentMergeScheduler.java:271)
> >>>>> Caused by: org.apache.lucene.index.CorruptIndexException: doc
> >>>>> counts
> >>>>> differ for segment _4l: fieldsReader shows 67861 but segmentInfo
> >>>>> shows
> >>>>> 67862
> >>>>> at org.apache.lucene.index.SegmentReader.initialize
> >>>>> (SegmentReader.java:313)
> >>>>> at org.apache.lucene.index.SegmentReader.get
> >>>>> (SegmentReader.java:262)
> >>>>> at org.apache.lucene.index.SegmentReader.get
> >>>>> (SegmentReader.java:221)
> >>>>> at org.apache.lucene.index.IndexWriter.mergeMiddle
> >>>>> (IndexWriter.java:3093)
> >>>>> at org.apache.lucene.index.IndexWriter.merge
> >>>>> (IndexWriter.java:2834)
> >>>>> at org.apache.lucene.index.ConcurrentMergeScheduler
> >>>>> $MergeThread.run(ConcurrentMergeScheduler.java:240)
> >>>>>
> >>>>> when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux, with
> >>>>> lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from yesterday.
> >>>>>
> >>>>> With java version 1.6.0_03-b05 things work fine.
> >>>>>
> >>>>> The exception happens a few hundred thousand documents into the
> >>>>> load.
> >>>>>
> >>>>> A different program updating a different index with different
> >>>>> data on
> >>>>> a different server gave a similar error on version 1.6.0_05-
> >>>>> b13 and
> >>>>> lucene 2.3.0.
> >>>>>
> >>>>> Any ideas? Is this maybe a known issue or am I missing
> >>>>> something obvious?
> >>>>
> >>>> My guess is perhaps a thread safety bug, more likely in Lucene
> >>>> indexing code (less likely in the JVM or specific libc).
> >>>>
> >>>> What Linux version are you using?
> >>>> What hardware are you running on (specifically, the CPU)?
> >>>>
> >>>> If possible, it would be great if you could check out Lucene
> >>>> trunk,
> >>>> crank up the iterations by modifying the TestStressIndexing2 and
> >>>> maybe
> >>>> fiddle with some of the other parameters in
> >>>> TestStressIndexing2.testMultiConfig(), and see if you can get
> >>>> it to
> >>>> fail.
> >>>>
> >>>>
> >>>> -Yonik
> >>>>
> >>>>
> >>>> -------------------------------------------------------------------
> >>>> --
> >>>>
> >>>>
> >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>>
> >>>>
> >>>
> >>> --------------------------------------------------------------------
> >>> -
> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >>
> >>
> >>
> >> <infostream.zip>
>
>
>
Re: CorruptIndexException with some versions of java
Posted by Michael McCandless <lu...@mikemccandless.com>.
Hi Ian,
Sheesh that's odd. The SegmentMerger produced an .fdx file that is
one document too short.
Can you run with this patch now, again applied to head of 2.3
branch? I just added another assert inside the loop that does the
field merging.
I will scrutinize this code...
Mike
Re: CorruptIndexException with some versions of java
Posted by Ian Lea <ia...@gmail.com>.
Mike
Patch applied and test re-run and picked up an assertion error this time:
Exception in thread "Lucene Merge Thread #0"
org.apache.lucene.index.MergePolicy$MergeException:
java.lang.AssertionError: after mergeFields: fdx size mismatch: 72357
docs vs 578848 length in bytes of _3o.fdx
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:320)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:297)
Caused by: java.lang.AssertionError: after mergeFields: fdx size
mismatch: 72357 docs vs 578848 length in bytes of _3o.fdx
at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:342)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:133)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3852)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3504)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:211)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:266)
The infostream output is attached. Since this email is to you and the
list it should make it to you.
Yonik: I haven't been able to make TestStressIndexing2 fail.
--
Ian.
On Tue, Mar 18, 2008 at 4:19 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
>
> Ian,
>
> Could you apply the attached patch applied to the head of the 2.3
> branch?
>
> It only adds more asserts, to try to pinpoint where exactly this
> corruption starts.
>
> Then, re-run the test with asserts enabled and infoStream turned on
> and post back. Thanks.
>
> Mike
>
>
>
>
> Ian Lea wrote:
>
> > It's failed on servers running SuSE 10.0 and 8.2 (ancient!)
> >
> > $ uname -a shows
> > Linux phoebe 2.6.13-15-smp #1 SMP Tue Sep 13 14:56:15 UTC 2005 x86_64
> > x86_64 x86_64 GNU/Linux
> >
> > and
> >
> > Linux phobos 2.4.20-64GB-SMP #1 SMP Mon Mar 17 17:56:03 UTC 2003 i686
> > unknown unknown GNU/Linux
> >
> > The first one has a 2.8Ghz Intel CPU, don't know about the second.
> >
> >
> > I'll try and run the stress test.
> >
> >
> > --
> > Ian.
> >
> >
> >
> > On Tue, Mar 18, 2008 at 2:17 PM, Yonik Seeley <yo...@apache.org>
> > wrote:
> >>
> >> On Tue, Mar 18, 2008 at 7:38 AM, Ian Lea <ia...@gmail.com> wrote:
> >>> Hi
> >>>
> >>>
> >>> When bulk loading into a new index I'm seeing this exception
> >>>
> >>> Exception in thread "Thread-1"
> >>> org.apache.lucene.index.MergePolicy$MergeException:
> >>> org.apache.lucene.index.CorruptIndexException: doc counts differ
> >>> for
> >>> segment _4l: fieldsReader shows 67861 but segmentInfo shows 67862
> >>> at org.apache.lucene.index.ConcurrentMergeScheduler
> >>> $MergeThread.run(ConcurrentMergeScheduler.java:271)
> >>> Caused by: org.apache.lucene.index.CorruptIndexException: doc
> >>> counts
> >>> differ for segment _4l: fieldsReader shows 67861 but segmentInfo
> >>> shows
> >>> 67862
> >>> at org.apache.lucene.index.SegmentReader.initialize
> >>> (SegmentReader.java:313)
> >>> at org.apache.lucene.index.SegmentReader.get
> >>> (SegmentReader.java:262)
> >>> at org.apache.lucene.index.SegmentReader.get
> >>> (SegmentReader.java:221)
> >>> at org.apache.lucene.index.IndexWriter.mergeMiddle
> >>> (IndexWriter.java:3093)
> >>> at org.apache.lucene.index.IndexWriter.merge
> >>> (IndexWriter.java:2834)
> >>> at org.apache.lucene.index.ConcurrentMergeScheduler
> >>> $MergeThread.run(ConcurrentMergeScheduler.java:240)
> >>>
> >>> when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux, with
> >>> lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from yesterday.
> >>>
> >>> With java version 1.6.0_03-b05 things work fine.
> >>>
> >>> The exception happens a few hundred thousand documents into the
> >>> load.
> >>>
> >>> A different program updating a different index with different
> >>> data on
> >>> a different server gave a similar error on version 1.6.0_05-b13 and
> >>> lucene 2.3.0.
> >>>
> >>> Any ideas? Is this maybe a known issue or am I missing
> >>> something obvious?
> >>
> >> My guess is perhaps a thread safety bug, more likely in Lucene
> >> indexing code (less likely in the JVM or specific libc).
> >>
> >> What Linux version are you using?
> >> What hardware are you running on (specifically, the CPU)?
> >>
> >> If possible, it would be great if you could check out Lucene trunk,
> >> crank up the iterations by modifying the TestStressIndexing2 and
> >> maybe
> >> fiddle with some of the other parameters in
> >> TestStressIndexing2.testMultiConfig(), and see if you can get it to
> >> fail.
> >>
> >>
> >> -Yonik
> >>
> >>
> >> ---------------------------------------------------------------------
> >>
> >>
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
>
Re: CorruptIndexException with some versions of java
Posted by Michael McCandless <lu...@mikemccandless.com>.
Ian,
Could you apply the attached patch applied to the head of the 2.3
branch?
It only adds more asserts, to try to pinpoint where exactly this
corruption starts.
Then, re-run the test with asserts enabled and infoStream turned on
and post back. Thanks.
Mike
Re: CorruptIndexException with some versions of java
Posted by Ian Lea <ia...@gmail.com>.
It's failed on servers running SuSE 10.0 and 8.2 (ancient!)
$ uname -a shows
Linux phoebe 2.6.13-15-smp #1 SMP Tue Sep 13 14:56:15 UTC 2005 x86_64
x86_64 x86_64 GNU/Linux
and
Linux phobos 2.4.20-64GB-SMP #1 SMP Mon Mar 17 17:56:03 UTC 2003 i686
unknown unknown GNU/Linux
The first one has a 2.8Ghz Intel CPU, don't know about the second.
I'll try and run the stress test.
--
Ian.
On Tue, Mar 18, 2008 at 2:17 PM, Yonik Seeley <yo...@apache.org> wrote:
>
> On Tue, Mar 18, 2008 at 7:38 AM, Ian Lea <ia...@gmail.com> wrote:
> > Hi
> >
> >
> > When bulk loading into a new index I'm seeing this exception
> >
> > Exception in thread "Thread-1"
> > org.apache.lucene.index.MergePolicy$MergeException:
> > org.apache.lucene.index.CorruptIndexException: doc counts differ for
> > segment _4l: fieldsReader shows 67861 but segmentInfo shows 67862
> > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:271)
> > Caused by: org.apache.lucene.index.CorruptIndexException: doc counts
> > differ for segment _4l: fieldsReader shows 67861 but segmentInfo shows
> > 67862
> > at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:313)
> > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
> > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3093)
> > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
> > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240)
> >
> > when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux, with
> > lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from yesterday.
> >
> > With java version 1.6.0_03-b05 things work fine.
> >
> > The exception happens a few hundred thousand documents into the load.
> >
> > A different program updating a different index with different data on
> > a different server gave a similar error on version 1.6.0_05-b13 and
> > lucene 2.3.0.
> >
> > Any ideas? Is this maybe a known issue or am I missing something obvious?
>
> My guess is perhaps a thread safety bug, more likely in Lucene
> indexing code (less likely in the JVM or specific libc).
>
> What Linux version are you using?
> What hardware are you running on (specifically, the CPU)?
>
> If possible, it would be great if you could check out Lucene trunk,
> crank up the iterations by modifying the TestStressIndexing2 and maybe
> fiddle with some of the other parameters in
> TestStressIndexing2.testMultiConfig(), and see if you can get it to
> fail.
>
>
> -Yonik
>
> ---------------------------------------------------------------------
>
>
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: CorruptIndexException with some versions of java
Posted by Yonik Seeley <yo...@apache.org>.
On Tue, Mar 18, 2008 at 7:38 AM, Ian Lea <ia...@gmail.com> wrote:
> Hi
>
>
> When bulk loading into a new index I'm seeing this exception
>
> Exception in thread "Thread-1"
> org.apache.lucene.index.MergePolicy$MergeException:
> org.apache.lucene.index.CorruptIndexException: doc counts differ for
> segment _4l: fieldsReader shows 67861 but segmentInfo shows 67862
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:271)
> Caused by: org.apache.lucene.index.CorruptIndexException: doc counts
> differ for segment _4l: fieldsReader shows 67861 but segmentInfo shows
> 67862
> at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:313)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3093)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240)
>
> when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux, with
> lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from yesterday.
>
> With java version 1.6.0_03-b05 things work fine.
>
> The exception happens a few hundred thousand documents into the load.
>
> A different program updating a different index with different data on
> a different server gave a similar error on version 1.6.0_05-b13 and
> lucene 2.3.0.
>
> Any ideas? Is this maybe a known issue or am I missing something obvious?
My guess is perhaps a thread safety bug, more likely in Lucene
indexing code (less likely in the JVM or specific libc).
What Linux version are you using?
What hardware are you running on (specifically, the CPU)?
If possible, it would be great if you could check out Lucene trunk,
crank up the iterations by modifying the TestStressIndexing2 and maybe
fiddle with some of the other parameters in
TestStressIndexing2.testMultiConfig(), and see if you can get it to
fail.
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: CorruptIndexException with some versions of java
Posted by Michael McCandless <lu...@mikemccandless.com>.
I don't see an attachment here -- maybe the mailing list software
stripped it off. If so can you send directly to me? Thanks.
Mike
Ian Lea wrote:
> Documents are biblio records. All have title, author etc. stored,
> some have a few extra fields as well. Typically around 25 fields per
> doc. The index is created with compound format, everything else as
> default.
>
> I've rerun the job until failure. Different numbers this time, but
> basically the same exception. In the failing pass it was trying to
> load 100K docs.
>
> Exception in thread "Thread-1"
> org.apache.lucene.index.MergePolicy$MergeException:
> org.apache.lucene.index.CorruptIndexException: doc counts differ for
> segment _cb: fieldsReader shows 71663 but segmentInfo shows 71664
> at org.apache.lucene.index.ConcurrentMergeScheduler
> $MergeThread.run(ConcurrentMergeScheduler.java:271)
> Caused by: org.apache.lucene.index.CorruptIndexException: doc counts
> differ for segment _cb: fieldsReader shows 71663 but segmentInfo shows
> 71664
> at org.apache.lucene.index.SegmentReader.initialize
> (SegmentReader.java:313)
> at org.apache.lucene.index.SegmentReader.get
> (SegmentReader.java:262)
> at org.apache.lucene.index.SegmentReader.get
> (SegmentReader.java:221)
> at org.apache.lucene.index.IndexWriter.mergeMiddle
> (IndexWriter.java:3099)
> at org.apache.lucene.index.IndexWriter.merge
> (IndexWriter.java:2834)
> at org.apache.lucene.index.ConcurrentMergeScheduler
> $MergeThread.run(ConcurrentMergeScheduler.java:240)
>
> As mentioned, the index is loaded in chunks - the infostream from the
> failing pass is attached. All infostreams from all 19 odd runs leading
> up to the failure available as well if that would help.
>
>
> Running with -ea doesn't seem to have made any difference.
>
>
> --
> Ian.
>
>
> On Tue, Mar 18, 2008 at 12:09 PM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>>
>> Can you call IndexWriter.setInfoStream(...) and get the error to
>> happen and post back the resulting output? And, turn on assertions
>> (java -ea) since that may catch the issue sooner.
>>
>> Can you describe you are setting up IndexWriter (autoCommit,
>> compound, etc.), and what your documents are like? Do your
>> documents
>> have a fixed schema (same fields every time), or it varies such that
>> some documents have no stored fields and some do?
>>
>> Mike
>>
>>
>>
>> Ian Lea wrote:
>>
>>> Hi
>>>
>>>
>>> When bulk loading into a new index I'm seeing this exception
>>>
>>> Exception in thread "Thread-1"
>>> org.apache.lucene.index.MergePolicy$MergeException:
>>> org.apache.lucene.index.CorruptIndexException: doc counts differ for
>>> segment _4l: fieldsReader shows 67861 but segmentInfo shows 67862
>>> at org.apache.lucene.index.ConcurrentMergeScheduler
>>> $MergeThread.run
>>> (ConcurrentMergeScheduler.java:271)
>>> Caused by: org.apache.lucene.index.CorruptIndexException: doc counts
>>> differ for segment _4l: fieldsReader shows 67861 but segmentInfo
>>> shows
>>> 67862
>>> at org.apache.lucene.index.SegmentReader.initialize
>>> (SegmentReader.java:313)
>>> at org.apache.lucene.index.SegmentReader.get
>>> (SegmentReader.java:262)
>>> at org.apache.lucene.index.SegmentReader.get
>>> (SegmentReader.java:221)
>>> at org.apache.lucene.index.IndexWriter.mergeMiddle
>>> (IndexWriter.java:3093)
>>> at org.apache.lucene.index.IndexWriter.merge
>>> (IndexWriter.java:2834)
>>> at org.apache.lucene.index.ConcurrentMergeScheduler
>>> $MergeThread.run
>>> (ConcurrentMergeScheduler.java:240)
>>>
>>> when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux, with
>>> lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from yesterday.
>>>
>>> With java version 1.6.0_03-b05 things work fine.
>>>
>>> The exception happens a few hundred thousand documents into the
>>> load.
>>>
>>> A different program updating a different index with different
>>> data on
>>> a different server gave a similar error on version 1.6.0_05-b13 and
>>> lucene 2.3.0.
>>>
>>>
>>> Any ideas? Is this maybe a known issue or am I missing something
>>> obvious?
>>>
>>>
>>>
>>> --
>>> Ian.
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: CorruptIndexException with some versions of java
Posted by Ian Lea <ia...@gmail.com>.
Documents are biblio records. All have title, author etc. stored,
some have a few extra fields as well. Typically around 25 fields per
doc. The index is created with compound format, everything else as
default.
I've rerun the job until failure. Different numbers this time, but
basically the same exception. In the failing pass it was trying to
load 100K docs.
Exception in thread "Thread-1"
org.apache.lucene.index.MergePolicy$MergeException:
org.apache.lucene.index.CorruptIndexException: doc counts differ for
segment _cb: fieldsReader shows 71663 but segmentInfo shows 71664
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:271)
Caused by: org.apache.lucene.index.CorruptIndexException: doc counts
differ for segment _cb: fieldsReader shows 71663 but segmentInfo shows
71664
at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:313)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240)
As mentioned, the index is loaded in chunks - the infostream from the
failing pass is attached. All infostreams from all 19 odd runs leading
up to the failure available as well if that would help.
Running with -ea doesn't seem to have made any difference.
--
Ian.
On Tue, Mar 18, 2008 at 12:09 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
>
> Can you call IndexWriter.setInfoStream(...) and get the error to
> happen and post back the resulting output? And, turn on assertions
> (java -ea) since that may catch the issue sooner.
>
> Can you describe you are setting up IndexWriter (autoCommit,
> compound, etc.), and what your documents are like? Do your documents
> have a fixed schema (same fields every time), or it varies such that
> some documents have no stored fields and some do?
>
> Mike
>
>
>
> Ian Lea wrote:
>
> > Hi
> >
> >
> > When bulk loading into a new index I'm seeing this exception
> >
> > Exception in thread "Thread-1"
> > org.apache.lucene.index.MergePolicy$MergeException:
> > org.apache.lucene.index.CorruptIndexException: doc counts differ for
> > segment _4l: fieldsReader shows 67861 but segmentInfo shows 67862
> > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run
> > (ConcurrentMergeScheduler.java:271)
> > Caused by: org.apache.lucene.index.CorruptIndexException: doc counts
> > differ for segment _4l: fieldsReader shows 67861 but segmentInfo shows
> > 67862
> > at org.apache.lucene.index.SegmentReader.initialize
> > (SegmentReader.java:313)
> > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
> > at org.apache.lucene.index.IndexWriter.mergeMiddle
> > (IndexWriter.java:3093)
> > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
> > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run
> > (ConcurrentMergeScheduler.java:240)
> >
> > when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux, with
> > lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from yesterday.
> >
> > With java version 1.6.0_03-b05 things work fine.
> >
> > The exception happens a few hundred thousand documents into the load.
> >
> > A different program updating a different index with different data on
> > a different server gave a similar error on version 1.6.0_05-b13 and
> > lucene 2.3.0.
> >
> >
> > Any ideas? Is this maybe a known issue or am I missing something
> > obvious?
> >
> >
> >
> > --
> > Ian.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: CorruptIndexException with some versions of java
Posted by Michael McCandless <lu...@mikemccandless.com>.
Can you call IndexWriter.setInfoStream(...) and get the error to
happen and post back the resulting output? And, turn on assertions
(java -ea) since that may catch the issue sooner.
Can you describe you are setting up IndexWriter (autoCommit,
compound, etc.), and what your documents are like? Do your documents
have a fixed schema (same fields every time), or it varies such that
some documents have no stored fields and some do?
Mike
Ian Lea wrote:
> Hi
>
>
> When bulk loading into a new index I'm seeing this exception
>
> Exception in thread "Thread-1"
> org.apache.lucene.index.MergePolicy$MergeException:
> org.apache.lucene.index.CorruptIndexException: doc counts differ for
> segment _4l: fieldsReader shows 67861 but segmentInfo shows 67862
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run
> (ConcurrentMergeScheduler.java:271)
> Caused by: org.apache.lucene.index.CorruptIndexException: doc counts
> differ for segment _4l: fieldsReader shows 67861 but segmentInfo shows
> 67862
> at org.apache.lucene.index.SegmentReader.initialize
> (SegmentReader.java:313)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
> at org.apache.lucene.index.IndexWriter.mergeMiddle
> (IndexWriter.java:3093)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run
> (ConcurrentMergeScheduler.java:240)
>
> when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux, with
> lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from yesterday.
>
> With java version 1.6.0_03-b05 things work fine.
>
> The exception happens a few hundred thousand documents into the load.
>
> A different program updating a different index with different data on
> a different server gave a similar error on version 1.6.0_05-b13 and
> lucene 2.3.0.
>
>
> Any ideas? Is this maybe a known issue or am I missing something
> obvious?
>
>
>
> --
> Ian.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: CorruptIndexException with some versions of java
Posted by Ian Lea <ia...@gmail.com>.
The data is loaded in chunks of up to 100K docs in separate runs of
the program if that helps answer the first question. All buffers have
default values, docs are small but not tiny, JVM is running with
default settings.
Answers to previous questions, and infostream, will follow once the
job has failed again.
--
Ian.
On Tue, Mar 18, 2008 at 1:24 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
>
> One question: do you know whether 67,861 docs "feels like" a newly
> flushed segment, or, the result of a merge?
>
> Ie, roughly how many docs are you buffering in IndexWriter before it
> flushes? Are they very small documents and your RAM buffer is large?
>
> Mike
>
>
>
> Ian Lea wrote:
>
> > Hi
> >
> >
> > When bulk loading into a new index I'm seeing this exception
> >
> > Exception in thread "Thread-1"
> > org.apache.lucene.index.MergePolicy$MergeException:
> > org.apache.lucene.index.CorruptIndexException: doc counts differ for
> > segment _4l: fieldsReader shows 67861 but segmentInfo shows 67862
> > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run
> > (ConcurrentMergeScheduler.java:271)
> > Caused by: org.apache.lucene.index.CorruptIndexException: doc counts
> > differ for segment _4l: fieldsReader shows 67861 but segmentInfo shows
> > 67862
> > at org.apache.lucene.index.SegmentReader.initialize
> > (SegmentReader.java:313)
> > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
> > at org.apache.lucene.index.IndexWriter.mergeMiddle
> > (IndexWriter.java:3093)
> > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
> > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run
> > (ConcurrentMergeScheduler.java:240)
> >
> > when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux, with
> > lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from yesterday.
> >
> > With java version 1.6.0_03-b05 things work fine.
> >
> > The exception happens a few hundred thousand documents into the load.
> >
> > A different program updating a different index with different data on
> > a different server gave a similar error on version 1.6.0_05-b13 and
> > lucene 2.3.0.
> >
> >
> > Any ideas? Is this maybe a known issue or am I missing something
> > obvious?
> >
> >
> >
> > --
> > Ian.
> >
>
>
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: CorruptIndexException with some versions of java
Posted by Michael McCandless <lu...@mikemccandless.com>.
One question: do you know whether 67,861 docs "feels like" a newly
flushed segment, or, the result of a merge?
Ie, roughly how many docs are you buffering in IndexWriter before it
flushes? Are they very small documents and your RAM buffer is large?
Mike
Ian Lea wrote:
> Hi
>
>
> When bulk loading into a new index I'm seeing this exception
>
> Exception in thread "Thread-1"
> org.apache.lucene.index.MergePolicy$MergeException:
> org.apache.lucene.index.CorruptIndexException: doc counts differ for
> segment _4l: fieldsReader shows 67861 but segmentInfo shows 67862
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run
> (ConcurrentMergeScheduler.java:271)
> Caused by: org.apache.lucene.index.CorruptIndexException: doc counts
> differ for segment _4l: fieldsReader shows 67861 but segmentInfo shows
> 67862
> at org.apache.lucene.index.SegmentReader.initialize
> (SegmentReader.java:313)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
> at org.apache.lucene.index.IndexWriter.mergeMiddle
> (IndexWriter.java:3093)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run
> (ConcurrentMergeScheduler.java:240)
>
> when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux, with
> lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from yesterday.
>
> With java version 1.6.0_03-b05 things work fine.
>
> The exception happens a few hundred thousand documents into the load.
>
> A different program updating a different index with different data on
> a different server gave a similar error on version 1.6.0_05-b13 and
> lucene 2.3.0.
>
>
> Any ideas? Is this maybe a known issue or am I missing something
> obvious?
>
>
>
> --
> Ian.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org