[GitHub] [lucene] rmuir commented on pull request #11867: Add monster test that indexes 1M vectors

[GitBox <gi...@apache.org>]

rmuir commented on PR #11867:
URL: https://github.com/apache/lucene/pull/11867#issuecomment-1286335299

   With current test i hit the exception on the 9.4 tag: BUILD FAILED in 2h 24m 45s:
   2GB heap. Never saw any significant time (e.g. 0.1%) in GC or other jvm threads when inspecting the running test:
   The initial indexing takes about an hour and then the forcemerge takes an eternity (over an hour), but it works:
   ```
   org.apache.lucene.document.TestManyKnnVectors > testLargeSegment FAILED
       java.lang.IllegalStateException: Vector data length 3072000000 not matching size=1000000 * dim=768 * byteSize=4 = -1222967296
           at __randomizedtesting.SeedInfo.seed([CF186B7BCEFCCF79:EBD7012A6CACC57]:0)
           at org.apache.lucene.codecs.lucene94.Lucene94HnswVectorsReader.validateFieldEntry(Lucene94HnswVectorsReader.java:185)
           at org.apache.lucene.codecs.lucene94.Lucene94HnswVectorsReader.readFields(Lucene94HnswVectorsReader.java:156)
           at org.apache.lucene.codecs.lucene94.Lucene94HnswVectorsReader.readMetadata(Lucene94HnswVectorsReader.java:103)
           at org.apache.lucene.codecs.lucene94.Lucene94HnswVectorsReader.<init>(Lucene94HnswVectorsReader.java:64)
           at org.apache.lucene.codecs.lucene94.Lucene94HnswVectorsFormat.fieldsReader(Lucene94HnswVectorsFormat.java:157)
           at org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsReader.<init>(PerFieldKnnVectorsFormat.java:219)
           at org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat.fieldsReader(PerFieldKnnVectorsFormat.java:81)
           at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:157)
           at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:91)
           at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:179)
           at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:221)
           at org.apache.lucene.index.IndexWriter.lambda$getReader$0(IndexWriter.java:536)
           at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:138)
           at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:598)
           at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:112)
           at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:91)
           at org.apache.lucene.document.TestManyKnnVectors.testLargeSegment(TestManyKnnVectors.java:94)
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org