You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (JIRA)" <ji...@apache.org> on 2013/04/22 19:27:15 UTC
[jira] [Comment Edited] (LUCENE-4948) Stink bug in
PostingsHighlighter
[ https://issues.apache.org/jira/browse/LUCENE-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13638191#comment-13638191 ]
Uwe Schindler edited comment on LUCENE-4948 at 4/22/13 5:26 PM:
----------------------------------------------------------------
This is easy to explain: J9 does not know compressed oops. On Oracle 64 bit JVMs the pointers are compressed, so a reference is 32 bits, which is identical to the size of ints. So they grow identical. In IBM J9 (64 bit, it only fails with 64 bits!) a reference is 8 bytes. So the reference array grows different than the int[] array.
This bug also affect useres with Oracle JDK, if they have > 32 Gig of heap...
Whenever we use ArrayUtil.grow() we have to take care that the sizes may differ. Please don't change the growing pattern, the code should (when it requires that several arrays are sized in parallel) calculate the new size before and use a suitable type to do that.
A good test would be: We make the growing pattern in ArrayUtil allow "test mode" maybe and make the factor random? Just an idea. Maybe we can do this before running tests using bytecode modification.
was (Author: thetaphi):
This is easy to explain: J9 does not know compressed oops. On Oracle 64 bit JVMs the pointers are compressed, so a reference is 32 bits, which is identical to the size of ints. So they grow identical. In IBM J9 (64 bit, it only fails with 64 bits!) a reference is 8 bytes. So the reference array grows different than the int[] array.
This bug also affect useres with Oracle JDK, if the have > 32 Gig of heap...
Whenever we use ArrayUtil.grow() we have to take care that the sizes may differ. Please don't change the growing pattern, the code should (when it requires that several arrays are sized in paralel, calculate the new size before and use a suitable type to do that).
A good test would be: We make the growing pattern in ArrayUtil allow "test mode" maybe and make the factor random? Just an idea. Maybe we can do this before running tests using bytecode modification.
> Stink bug in PostingsHighlighter
> --------------------------------
>
> Key: LUCENE-4948
> URL: https://issues.apache.org/jira/browse/LUCENE-4948
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/highlighter
> Reporter: Michael McCandless
> Fix For: 5.0, 4.4
>
>
> This test fail reproduces on IBM J9:
> {noformat}
> NOTE: reproduce with: ant test -Dtestcase=TestPostingsHighlighter -Dtests.method=testCambridgeMA -Dtests.seed=2A9A93DAC39E0938 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_HN -Dtests.timezone=America/Yellowknife -Dtests.file.encoding=UTF-8
> {noformat}
> {noformat}
> Stack Trace:
> java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 37
> at __randomizedtesting.SeedInfo.seed([2A9A93DAC39E0938:AB8FF071AD305139]:0)
> at org.apache.lucene.search.postingshighlight.Passage.addMatch(Passage.java:53)
> at org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightDoc(PostingsHighlighter.java:547)
> at org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightField(PostingsHighlighter.java:425)
> at org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightFields(PostingsHighlighter.java:364)
> at org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlightFields(PostingsHighlighter.java:268)
> at org.apache.lucene.search.postingshighlight.PostingsHighlighter.highlight(PostingsHighlighter.java:198)
> at org.apache.lucene.search.postingshighlight.TestPostingsHighlighter.testCambridgeMA(TestPostingsHighlighter.java:373)
> {noformat}
> I think it's because J9 grows arrays in a different progression than other JVMs ... we should fix PostingsHighlighter to forcefully grow the arrays to the same length instead of this:
> {noformat}
> if (numMatches == matchStarts.length) {
> matchStarts = ArrayUtil.grow(matchStarts, numMatches+1);
> matchEnds = ArrayUtil.grow(matchEnds, numMatches+1);
> BytesRef newMatchTerms[] = new BytesRef[ArrayUtil.oversize(numMatches+1, RamUsageEstimator.NUM_BYTES_OBJECT_REF)];
> System.arraycopy(matchTerms, 0, newMatchTerms, 0, numMatches);
> matchTerms = newMatchTerms;
> }
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org