You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Torben Riis <tr...@multi-support.com> on 2019/05/01 13:26:11 UTC
ArrayIndexOutOfBoundsException during System.arraycopy in BKDWriter
Hi,
I'm a bit stuck here and needs a clue or two in order to continue our investigations. Hope that someone can help. :)
Periodically, around once a month, we get the below ArrayIndexOutOfBoundsException on our system. We use multiple indexes and the error can originate from any of them, but the error always occurs in line 1217 in BKDWriter (during a System.arraycopy).
We found a couple of issues on the net regarding JIT optimization problem related to J9, but they all looks like they have been resolved and cannot be reproduced anymore. But nevertheless, we have just added the -Xint flag (disable JIT compiler) in order to see whether this has any impact. Unfortunately we do not have the result of this yet, but I'll of course post it when it is known.
Are there any of you clever guys out there, that has some good ideas further investigations? Or have seen such issue before?
We are using Lucene 6.6.0 and runs on IBM J9 on the IBM I platform.
Java version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 8.0.5.25 - pap6480sr5fp25-20181030_01(SR5 FP25))
IBM J9 VM (build 2.9, JRE 1.8.0 OS/400 ppc64-64-Bit Compressed References 20181029_400846 (JIT enabled, AOT enabled)
OpenJ9 - c5c78da
OMR - 3d5ac33
IBM - 8c1bdc2)
JCL - 20181022_01 based on Oracle jdk8u191-b26
NOTICE: If no version information is found above, this could indicate a corrupted Java installation!
Java detected was: /QOpenSys/QIBM/ProdData/JavaVM/jdk80/64bit/bin/java
-Dmultiarchive.basepath=/home/NEXTOWN/Multi-Support/Next -Xms128m -Xmx2048m
Stacktrace
Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:07.970 CEST [Lucene Merge Thread #0] org.apache.lucene.index.MergePolicy$MergeException: java.lang.ArrayIndexOutOfBoundsException
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
Caused by: 2019-05-01T06:10:07.971 CEST [Lucene Merge Thread #0] java.lang.ArrayIndexOutOfBoundsException
at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217)
at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197)
at org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078)
at org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245)
at org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82)
at org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648)
at org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560)
at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212)
at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:08.075 CEST [Lucene Merge Thread #0] org.apache.lucene.index.MergePolicy$MergeException: java.lang.ArrayIndexOutOfBoundsException
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
Caused by: 2019-05-01T06:10:08.076 CEST [Lucene Merge Thread #0] java.lang.ArrayIndexOutOfBoundsException
at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217)
at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197)
at org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078)
at org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245)
at org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82)
at org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648)
at org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560)
at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212)
at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler
BKDWriter.java
[cid:image003.png@01D5002C.7B1A9B80]
Kind regards
Torben Riis
-------------------------
Torben Riis
Lead Architect
[cid:image001.png@01D50032.347008E0]
Multi Support * Making good business run better
+45 96 600 600 * www.multi-support.com
SV: ArrayIndexOutOfBoundsException during System.arraycopy in
BKDWriter
Posted by Torben Riis <tr...@multi-support.com>.
Hi,
Sorry for the late answer (vacation).
Indeed, we very soon realized that the -Xint argument was a quite naive choice. We have changed the arguments to specific excludes for every class / inner classes in the bkd package. Moreover, we have also made a copy of the whole installation in production, and added the opposite arguments (enforce jit optimizations for the specific classes). First we will try with optLevel=hot, but if this doesn’t show anything we will afterwards try with veryHot and scorching. I’ll post the results of our findings on the upcoming JIRA issue.
Unfortunately it’s not possible to run OpenJDK on the IBM i platform, but it is worth mentioning that our product is a standard product, which typically run on the Windows or Linux platform using AdoptOpenJDK. Currently we have a couple of hundred installations running out there on these platforms, and without any problems. But on the IBM I platform with J9 we sometimes see this exception.
On production we have disabled jit optimization for the following classes
-Xjit:verbose,exclude={org/apache/lucene/util/bkd/BKDWriter.*|org/apache/lucene/util/bkd/BKDReader.*|org/apache/lucene/util/bkd/BKDReader$IndexTree.*|org/apache/lucene/util/bkd/BKDReader$IntersectState.*|org/apache/lucene/util/bkd/BKDReader$LegacyIndexTree.*|org/apache/lucene/util/bkd/BKDReader$PackedIndexTree.*|org/apache/lucene/util/bkd/BKDWriter.*|org/apache/lucene/util/bkd/BKDWriter$OneDimensionBKDWriter.*|org/apache/lucene/util/bkd/BKDWriter$MergeReader.*|org/apache/lucene/util/bkd/BKDWriter$BKDMergeQueue.*|org/apache/lucene/util/bkd/BKDWriter$PathSlice.*|org/apache/lucene/util/bkd/DocIdsWriter.*|org/apache/lucene/util/bkd/HeapPointReader.*|org/apache/lucene/util/bkd/HeapPointWriter.*|org/apache/lucene/util/bkd/MutablePointsReaderUtils.*|org/apache/lucene/util/bkd/OfflinePointReader.*|org/apache/lucene/util/bkd/OfflinePointWriter.*|org/apache/lucene/util/bkd/PointReader.*|org/apache/lucene/util/bkd/PointWriter.*},vlog=/home/nextown/multi-support/next/logs/jit
On the copy of production we have enforced jit optimization for the following classes
-Xjit:{org/apache/lucene/util/bkd/BKDWriter.*|org/apache/lucene/util/bkd/BKDReader.*|org/apache/lucene/util/bkd/BKDReader$IndexTree.*|org/apache/lucene/util/bkd/BKDReader$IntersectState.*|org/apache/lucene/util/bkd/BKDReader$LegacyIndexTree.*|org/apache/lucene/util/bkd/BKDReader$PackedIndexTree.*|org/apache/lucene/util/bkd/BKDWriter.*|org/apache/lucene/util/bkd/BKDWriter$OneDimensionBKDWriter.*|org/apache/lucene/util/bkd/BKDWriter$MergeReader.*|org/apache/lucene/util/bkd/BKDWriter$BKDMergeQueue.*|org/apache/lucene/util/bkd/BKDWriter$PathSlice.*|org/apache/lucene/util/bkd/DocIdsWriter.*|org/apache/lucene/util/bkd/HeapPointReader.*|org/apache/lucene/util/bkd/HeapPointWriter.*|org/apache/lucene/util/bkd/MutablePointsReaderUtils.*|org/apache/lucene/util/bkd/OfflinePointReader.*|org/apache/lucene/util/bkd/OfflinePointWriter.*|org/apache/lucene/util/bkd/PointReader.*|org/apache/lucene/util/bkd/PointWriter.*}(count=0,optlevel=hot)
I’ll create a JIRA task right away.
Med venlig hilsen
Torben Riis
—————————————————————————
Torben Riis
Lead Architect
[cid:image001.png@01D504C9.2891A970]
Multi Support · Making good business run better
+45 96 600 600 · www.multi-support.com
Fra: Michael McCandless <lu...@mikemccandless.com>
Sendt: 3. maj 2019 10:05
Til: Lucene Users <ja...@lucene.apache.org>; Torben Riis <tr...@multi-support.com>
Emne: Re: ArrayIndexOutOfBoundsException during System.arraycopy in BKDWriter
Note that the -Xint flag will make your code run tremendously more slowly! Likely to the point of not really being usable. But it'd be interesting to see if that side-steps the bug.
Is it possible to test with OpenJDK as well?
The BKDWriter code is quite complex, so it is also possible there is a Lucene bug at work.
Can you open an issue in Lucene's jira and we can iterate there?
Thanks,
Mike McCandless
http://blog.mikemccandless.com
On Wed, May 1, 2019 at 9:34 AM Torben Riis <tr...@multi-support.com>> wrote:
Hi,
I’m a bit stuck here and needs a clue or two in order to continue our investigations. Hope that someone can help. :)
Periodically, around once a month, we get the below ArrayIndexOutOfBoundsException on our system. We use multiple indexes and the error can originate from any of them, but the error always occurs in line 1217 in BKDWriter (during a System.arraycopy).
We found a couple of issues on the net regarding JIT optimization problem related to J9, but they all looks like they have been resolved and cannot be reproduced anymore. But nevertheless, we have just added the -Xint flag (disable JIT compiler) in order to see whether this has any impact. Unfortunately we do not have the result of this yet, but I’ll of course post it when it is known.
Are there any of you clever guys out there, that has some good ideas further investigations? Or have seen such issue before?
We are using Lucene 6.6.0 and runs on IBM J9 on the IBM I platform.
Java version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 8.0.5.25 - pap6480sr5fp25-20181030_01(SR5 FP25))
IBM J9 VM (build 2.9, JRE 1.8.0 OS/400 ppc64-64-Bit Compressed References 20181029_400846 (JIT enabled, AOT enabled)
OpenJ9 - c5c78da
OMR - 3d5ac33
IBM - 8c1bdc2)
JCL - 20181022_01 based on Oracle jdk8u191-b26
NOTICE: If no version information is found above, this could indicate a corrupted Java installation!
Java detected was: /QOpenSys/QIBM/ProdData/JavaVM/jdk80/64bit/bin/java
-Dmultiarchive.basepath=/home/NEXTOWN/Multi-Support/Next -Xms128m -Xmx2048m
Stacktrace
Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:07.970 CEST [Lucene Merge Thread #0] org.apache.lucene.index.MergePolicy$MergeException: java.lang.ArrayIndexOutOfBoundsException
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
Caused by: 2019-05-01T06:10:07.971 CEST [Lucene Merge Thread #0] java.lang.ArrayIndexOutOfBoundsException
at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217)
at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197)
at org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078)
at org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245)
at org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82)
at org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648)
at org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560)
at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212)
at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:08.075 CEST [Lucene Merge Thread #0] org.apache.lucene.index.MergePolicy$MergeException: java.lang.ArrayIndexOutOfBoundsException
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
Caused by: 2019-05-01T06:10:08.076 CEST [Lucene Merge Thread #0] java.lang.ArrayIndexOutOfBoundsException
at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217)
at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197)
at org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078)
at org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245)
at org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82)
at org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648)
at org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560)
at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212)
at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler
BKDWriter.java
Kind regards
Torben Riis
—————————————————————————
Torben Riis
Lead Architect
Multi Support · Making good business run better
+45 96 600 600 · www.multi-support.com<http://www.multi-support.com>
Re: ArrayIndexOutOfBoundsException during System.arraycopy in BKDWriter
Posted by Michael McCandless <lu...@mikemccandless.com>.
Note that the -Xint flag will make your code run tremendously more
slowly! Likely to the point of not really being usable. But it'd be
interesting to see if that side-steps the bug.
Is it possible to test with OpenJDK as well?
The BKDWriter code is quite complex, so it is also possible there is a
Lucene bug at work.
Can you open an issue in Lucene's jira and we can iterate there?
Thanks,
Mike McCandless
http://blog.mikemccandless.com
On Wed, May 1, 2019 at 9:34 AM Torben Riis <tr...@multi-support.com> wrote:
> Hi,
>
>
>
> I’m a bit stuck here and needs a clue or two in order to continue our
> investigations. Hope that someone can help. :)
>
>
>
> Periodically, around once a month, we get the below
> ArrayIndexOutOfBoundsException on our system. We use multiple indexes and
> the error can originate from any of them, but the error always occurs in
> line 1217 in BKDWriter (during a System.arraycopy).
>
>
>
> We found a couple of issues on the net regarding JIT optimization problem
> related to J9, but they all looks like they have been resolved and cannot
> be reproduced anymore. But nevertheless, we have just added the -Xint flag
> (disable JIT compiler) in order to see whether this has any impact.
> Unfortunately we do not have the result of this yet, but I’ll of course
> post it when it is known.
>
>
>
> Are there any of you clever guys out there, that has some good ideas
> further investigations? Or have seen such issue before?
>
>
>
> We are using Lucene 6.6.0 and runs on IBM J9 on the IBM I platform.
>
>
>
>
>
> *Java version*
>
> java version "1.8.0_191"
>
> Java(TM) SE Runtime Environment (build 8.0.5.25 -
> pap6480sr5fp25-20181030_01(SR5 FP25))
>
> IBM J9 VM (build 2.9, JRE 1.8.0 OS/400 ppc64-64-Bit Compressed References
> 20181029_400846 (JIT enabled, AOT enabled)
>
> OpenJ9 - c5c78da
>
> OMR - 3d5ac33
>
> IBM - 8c1bdc2)
>
> JCL - 20181022_01 based on Oracle jdk8u191-b26
>
> NOTICE: If no version information is found above, this could indicate a
> corrupted Java installation!
>
> Java detected was: /QOpenSys/QIBM/ProdData/JavaVM/jdk80/64bit/bin/java
>
> -Dmultiarchive.basepath=/home/NEXTOWN/Multi-Support/Next -Xms128m -Xmx2048m
>
>
>
>
>
> *Stacktrace*
>
> Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:07.970 CEST
> [Lucene Merge Thread #0]
> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.ArrayIndexOutOfBoundsException
>
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
>
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
>
> Caused by: 2019-05-01T06:10:07.971 CEST [Lucene Merge Thread #0]
> java.lang.ArrayIndexOutOfBoundsException
>
> at
> org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217)
>
> at
> org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197)
>
> at
> org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078)
>
> at
> org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245)
>
> at
> org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82)
>
> at
> org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648)
>
> at
> org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560)
>
> at
> org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212)
>
> at
> org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173)
>
> at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122)
>
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
>
> at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
>
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
>
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
>
> Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:08.075 CEST
> [Lucene Merge Thread #0]
> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.ArrayIndexOutOfBoundsException
>
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
>
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
>
> Caused by: 2019-05-01T06:10:08.076 CEST [Lucene Merge Thread #0]
> java.lang.ArrayIndexOutOfBoundsException
>
> at
> org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217)
>
> at
> org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197)
>
> at
> org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078)
>
> at
> org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245)
>
> at
> org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82)
>
> at
> org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648)
>
> at
> org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560)
>
> at
> org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212)
>
> at
> org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173)
>
> at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122)
>
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
>
> at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
>
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
>
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler
>
>
>
>
>
> *BKDWriter.java*
>
>
>
>
>
>
> Kind regards
>
> Torben Riis
>
> —————————————————————————
> Torben Riis
> Lead Architect
>
>
> Multi Support · Making good business run better
> +45 96 600 600 · www.multi-support.com
>
>
>
>
>
>
>