You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2022/11/08 06:44:46 UTC
[GitHub] [lucene] rmuir opened a new pull request, #11906: Add monster test for many knn docs
rmuir opened a new pull request, #11906:
URL: https://github.com/apache/lucene/pull/11906
Goal is to reproduce #11905
Basically I adapted the test from #11867 as a start, tweaked the indexing and merging to try to get it running reasonably, since it needs to make single segment of 16M docs for this case.
I'm leaving it running overnight, hopefully it solves the problem or is at least close, as I haven't run it to completion yet. Will look more tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir closed pull request #11906: Add monster test for many knn docs
Posted by GitBox <gi...@apache.org>.
rmuir closed pull request #11906: Add monster test for many knn docs
URL: https://github.com/apache/lucene/pull/11906
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #11906: Add monster test for many knn docs
Posted by GitBox <gi...@apache.org>.
rmuir commented on PR #11906:
URL: https://github.com/apache/lucene/pull/11906#issuecomment-1307150684
i bumped the ram and restarted the test. but it is really broken that i can flush out all the docs with a 512MB heap, but need many many gigabytes to merge them together. and its only 16 million docs with a single field which is a vector of length 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #11906: Add monster test for many knn docs
Posted by GitBox <gi...@apache.org>.
rmuir commented on PR #11906:
URL: https://github.com/apache/lucene/pull/11906#issuecomment-1310221605
this test is folded into #11905
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jdconrad commented on pull request #11906: Add monster test for many knn docs
Posted by GitBox <gi...@apache.org>.
jdconrad commented on PR #11906:
URL: https://github.com/apache/lucene/pull/11906#issuecomment-1308006136
Just as confirmation I'm seeing `FixedBitSet.clear` taking up a lot of time as well when running this test.
```
"Lucene Merge Thread #0" #18 daemon prio=5 os_prio=0 cpu=347309.85ms elapsed=348.16s tid=0x00007f8ab81121e0 nid=41015 runnable [0x00007f8adbafb000]
java.lang.Thread.State: RUNNABLE
at org.apache.lucene.util.FixedBitSet.clear(FixedBitSet.java:556)
at org.apache.lucene.util.hnsw.HnswGraphSearcher.prepareScratchState(HnswGraphSearcher.java:287)
at org.apache.lucene.util.hnsw.HnswGraphSearcher.searchLevel(HnswGraphSearcher.java:211)
at org.apache.lucene.util.hnsw.HnswGraphSearcher.searchLevel(HnswGraphSearcher.java:196)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.addGraphNode(HnswGraphBuilder.java:186)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.addGraphNode(HnswGraphBuilder.java:194)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.addVectors(HnswGraphBuilder.java:151)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.build(HnswGraphBuilder.java:143)
at org.apache.lucene.codecs.lucene94.Lucene94HnswVectorsWriter.mergeOneField(Lucene94HnswVectorsWriter.java:424)
at org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.mergeOneField(PerFieldKnnVectorsFormat.java:118)
at org.apache.lucene.codecs.KnnVectorsWriter.merge(KnnVectorsWriter.java:80)
at org.apache.lucene.index.SegmentMerger.mergeVectorValues(SegmentMerger.java:255)
at org.apache.lucene.index.SegmentMerger$$Lambda$185/0x0000000800dc58a8.merge(Unknown Source)
at org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:298)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:149)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5139)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4679)
at org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6428)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:639)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:700)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #11906: Add monster test for many knn docs
Posted by GitBox <gi...@apache.org>.
rmuir commented on PR #11906:
URL: https://github.com/apache/lucene/pull/11906#issuecomment-1307432615
I looked into why the test is taking eternity to run, the super slow merge at the end is spending all its time clearing bitsets! Looks like the wrong datastructure...
```
java.lang.Thread.State: RUNNABLE
at java.util.Arrays.fill([java.base@17.0.4.1/Arrays.java:3169](http://java.base@17.0.4.1/Arrays.java:3169))
at org.apache.lucene.util.FixedBitSet.clear(FixedBitSet.java:556)
at org.apache.lucene.util.hnsw.HnswGraphSearcher.prepareScratchState(HnswGraphSearcher.java:287)
at org.apache.lucene.util.hnsw.HnswGraphSearcher.searchLevel(HnswGraphSearcher.java:211)
at org.apache.lucene.util.hnsw.HnswGraphSearcher.searchLevel(HnswGraphSearcher.java:196)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.addGraphNode(HnswGraphBuilder.java:181)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.addGraphNode(HnswGraphBuilder.java:194)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.addVectors(HnswGraphBuilder.java:151)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.build(HnswGraphBuilder.java:143)
at org.apache.lucene.codecs.lucene94.Lucene94HnswVectorsWriter.mergeOneField(Lucene94HnswVectorsWriter.java:424)
at org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.mergeOneField(PerFieldKnnVectorsFormat.java:118)
at org.apache.lucene.codecs.KnnVectorsWriter.merge(KnnVectorsWriter.java:80)
at org.apache.lucene.index.SegmentMerger.mergeVectorValues(SegmentMerger.java:255)
at org.apache.lucene.index.SegmentMerger$$Lambda$191/0x0000000800293730.merge(Unknown Source)
at org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:298)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:149)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #11906: Add monster test for many knn docs
Posted by GitBox <gi...@apache.org>.
rmuir commented on PR #11906:
URL: https://github.com/apache/lucene/pull/11906#issuecomment-1308119525
current test still doesn't fail. checkIndex just calls nextDoc() on low-level vectors but we may need to invoke skipping to find the issue. That's my theory at least.
one thing missing is that, other than checking for vector's lengths and other super-basic stuff, we don't have any kind of "logical checksums" to ensure all the decoded data is really correct. For things like postings we have these (the index statistics: docFreq, termFreq, totalTermFreq, sumDocFreq, etc). That's going to limit how well we can find issues with tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org