You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "ChrisHegarty (via GitHub)" <gi...@apache.org> on 2024/08/15 13:39:07 UTC
[I] TestHnswFloatVectorGraph.testRandomReadWriteAndMerge fails with java.lang.IndexOutOfBoundsException [lucene]
ChrisHegarty opened a new issue, #13659:
URL: https://github.com/apache/lucene/issues/13659
### Description
```
TestHnswFloatVectorGraph > testRandomReadWriteAndMerge FAILED
java.lang.IndexOutOfBoundsException: Index 2147483647 out of bounds for length 18
at __randomizedtesting.SeedInfo.seed([11F871BAF8E02ADC:343906638A98C7FC]:0)
at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:100)
at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:106)
at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:302)
at java.base/java.util.Objects.checkIndex(Objects.java:385)
at java.base/java.util.ArrayList.get(ArrayList.java:427)
at org.apache.lucene.util.hnsw.RandomAccessVectorValues$1.vectorValue(RandomAccessVectorValues.java:135)
at org.apache.lucene.codecs.hnsw.DefaultFlatVectorScorer$FloatScoringSupplier$1.score(DefaultFlatVectorScorer.java:142)
at org.apache.lucene.util.hnsw.HnswGraphSearcher.searchLevel(HnswGraphSearcher.java:206)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.connectComponents(HnswGraphBuilder.java:462)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.connectComponents(HnswGraphBuilder.java:423)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.finish(HnswGraphBuilder.java:416)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.getCompletedGraph(HnswGraphBuilder.java:182)
at org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter$FieldWriter.getGraph(Lucene99HnswVectorsWriter.java:618)
at org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.writeField(Lucene99HnswVectorsWriter.java:187)
at org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.flush(Lucene99HnswVectorsWriter.java:149)
at org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.flush(PerFieldKnnVectorsFormat.java:115)
at org.apache.lucene.index.VectorValuesConsumer.flush(VectorValuesConsumer.java:76)
at org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:305)
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:456)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
at org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649)
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3674)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:4122)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4084)
at org.apache.lucene.util.hnsw.HnswGraphTestCase.testRandomReadWriteAndMerge(HnswGraphTestCase.java:189)
at org.apache.lucene.util.hnsw.TestHnswFloatVectorGraph.testRandomReadWriteAndMerge(TestHnswFloatVectorGraph.java:40)
...
```
### Gradle command to reproduce
./gradlew :lucene:core:test --tests "org.apache.lucene.util.hnsw.TestHnswFloatVectorGraph.testRandomReadWriteAndMerge" -Ptests.jvms=6 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1" -Ptests.seed=11F871BAF8E02ADC -Ptests.locale=hy-AM -Ptests.timezone=America/Vancouver -Ptests.gui=false -Ptests.file.encoding=UTF-8 -Ptests.vectorsize=128 -Ptests.forceintegervectors=true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] TestHnswFloatVectorGraph.testRandomReadWriteAndMerge fails with java.lang.IndexOutOfBoundsException [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent closed issue #13659: TestHnswFloatVectorGraph.testRandomReadWriteAndMerge fails with java.lang.IndexOutOfBoundsException
URL: https://github.com/apache/lucene/issues/13659
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] TestHnswFloatVectorGraph.testRandomReadWriteAndMerge fails with java.lang.IndexOutOfBoundsException [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on issue #13659:
URL: https://github.com/apache/lucene/issues/13659#issuecomment-2296727766
> I mean it would be nice if we didn't generate these degenerate Component in the first place? But this will work
I had the same thought, but then backed away quickly! ;-) . I think that @benwtrent's fix is the safest thing to do here. There are several usages of `Component` that are a little less clear to me, but we minimally need to create such `new Component(NO_MORE_DOCS, x)`, where `x` is greater than equal to `0`, for such things as determining `componentSizes`.
Maybe the javadoc in `Component` could be updated to hint a little more at the potential usages. But that can be done separately, if at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] TestHnswFloatVectorGraph.testRandomReadWriteAndMerge fails with java.lang.IndexOutOfBoundsException [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on issue #13659:
URL: https://github.com/apache/lucene/issues/13659#issuecomment-2291853791
From what I can tell:
- `components` will return a component where the entry_point is `NO_MORE_DOCS`
- This occurs because the `notFullyConnected.nextSetBit(0);` returns `NO_MORE_DOCS`
- When checking for connectedness, we look at `size` not entry point
```
// connect other components to the largest one
Component c0 = components.stream().max(Comparator.comparingInt(Component::size)).get();
```
I am not sure what the behavior should be here, but it seems to me that a component whose entry point is `NO_MORE_DOCS` shouldn't get added to the components list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] TestHnswFloatVectorGraph.testRandomReadWriteAndMerge fails with java.lang.IndexOutOfBoundsException [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on issue #13659:
URL: https://github.com/apache/lucene/issues/13659#issuecomment-2291899075
@msokolov ^ what say you? It seems that a `NO_MORE_DOCS` indicates that the `notFullyConnected` just didn't ever get selected. Then later, when iterating the components
```
RandomVectorScorer scorer = scorerSupplier.scorer(c.start());
```
And `start()` is `NO_MORE_DOCS`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] TestHnswFloatVectorGraph.testRandomReadWriteAndMerge fails with java.lang.IndexOutOfBoundsException [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on issue #13659:
URL: https://github.com/apache/lucene/issues/13659#issuecomment-2291778869
Yeah, gitbisect indicate `217828736c4 - gh-12627: HnswGraphBuilder connects disconnected HNSW graph components (#13566) (7 days ago) <Michael Sokolov>`
I am not 100% sure whats up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] TestHnswFloatVectorGraph.testRandomReadWriteAndMerge fails with java.lang.IndexOutOfBoundsException [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on issue #13659:
URL: https://github.com/apache/lucene/issues/13659#issuecomment-2292058048
Oh, the gift that keeps on giving! Your solution seems reasonable. I mean it would be nice if we didn't generate these degenerate Component in the first place? But this will work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org