You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "mccullocht (via GitHub)" <gi...@apache.org> on 2025/09/29 18:34:43 UTC
[PR] Implement off-heap scoring for OSQ 4, 7, and 8 bit representations [lucene]
mccullocht opened a new pull request, #15257:
URL: https://github.com/apache/lucene/pull/15257
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement off-heap scoring for OSQ 4, 7, and 8 bit representations [lucene]
Posted by "mccullocht (via GitHub)" <gi...@apache.org>.
mccullocht commented on PR #15257:
URL: https://github.com/apache/lucene/pull/15257#issuecomment-3357554058
I did move just the vector dot product off-heap and I'm not planning to do anything clever with the corrections. I'm not sure that would pay off anyway -- you'd have to transpose from row view to column view to parallelize that work, and it would be 128-bit on x86 which may not go well.
I was assuming that accessing the corrective terms was messing with performance but larger jfr stacks point at a more mysterious culprit. This PR spends more time in lane reduction (???) and 128-bit loads of data from the memory segment (probably memory latency). For the latter case its weird that I don't see anything on the baseline when I know that copying to the heap should be inducing a similar hit.
```
baseline:
36.66% 12745 org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#dotProductBody512() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#dotProductBody() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#uint8DotProduct() [Inlined code]
at org.apache.lucene.util.VectorUtil#uint8DotProduct() [Inlined code]
at org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer#quantizedScore() [Inlined code]
at org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer$1#score() [JIT compiled code]
at org.apache.lucene.util.hnsw.RandomVectorScorer#bulkScore() [Inlined code]
at org.apache.lucene.util.hnsw.HnswGraphSearcher#searchLevel() [JIT compiled code]
at org.apache.lucene.util.hnsw.AbstractHnswGraphSearcher#search() [Inlined code]
at org.apache.lucene.util.hnsw.HnswGraphSearcher#search() [Inlined code]
25.90% 9005 org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#dotProductBody512() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#dotProductBody() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#uint8DotProduct() [Inlined code]
at org.apache.lucene.util.VectorUtil#uint8DotProduct() [Inlined code]
at org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer#quantizedScore() [Inlined code]
at org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer$1#score() [JIT compiled code]
at org.apache.lucene.util.hnsw.RandomVectorScorer#bulkScore() [Inlined code]
at org.apache.lucene.util.hnsw.HnswGraphSearcher#searchLevel() [JIT compiled code]
at org.apache.lucene.util.hnsw.AbstractHnswGraphSearcher#search() [JIT compiled code]
at org.apache.lucene.util.hnsw.HnswGraphSearcher#search() [JIT compiled code]
8.64% 3005 jdk.incubator.vector.IntVector#reduceLanesTemplate() [Inlined code]
at jdk.incubator.vector.Int512Vector#reduceLanes() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#dotProductBody512() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#dotProductBody() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#uint8DotProduct() [Inlined code]
at org.apache.lucene.util.VectorUtil#uint8DotProduct() [Inlined code]
at org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer#quantizedScore() [Inlined code]
at org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer$1#score() [JIT compiled code]
at org.apache.lucene.util.hnsw.RandomVectorScorer#bulkScore() [Inlined code]
at org.apache.lucene.util.hnsw.HnswGraphSearcher#searchLevel() [JIT compiled code]
candidate:
33.93% 11848 jdk.incubator.vector.IntVector#reduceLanesTemplate() [Inlined code]
at jdk.incubator.vector.Int512Vector#reduceLanes() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#dotProductBody512() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#dotProductBody() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#uint8DotProduct() [Inlined code]
at org.apache.lucene.internal.vectorization.Lucene104MemorySegmentScalarQuantizedVectorScorer$RandomVectorScorerImpl#score() [JIT compiled code]
at org.apache.lucene.util.hnsw.RandomVectorScorer#bulkScore() [Inlined code]
at org.apache.lucene.util.hnsw.HnswGraphSearcher#searchLevel() [JIT compiled code]
at org.apache.lucene.util.hnsw.AbstractHnswGraphSearcher#search() [Inlined code]
at org.apache.lucene.util.hnsw.HnswGraphSearcher#search() [Inlined code]
23.97% 8369 jdk.incubator.vector.IntVector#reduceLanesTemplate() [Inlined code]
at jdk.incubator.vector.Int512Vector#reduceLanes() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#dotProductBody512() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#dotProductBody() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#uint8DotProduct() [Inlined code]
at org.apache.lucene.internal.vectorization.Lucene104MemorySegmentScalarQuantizedVectorScorer$RandomVectorScorerImpl#score() [JIT compiled code]
at org.apache.lucene.util.hnsw.RandomVectorScorer#bulkScore() [Inlined code]
at org.apache.lucene.util.hnsw.HnswGraphSearcher#searchLevel() [JIT compiled code]
at org.apache.lucene.util.hnsw.AbstractHnswGraphSearcher#search() [JIT compiled code]
at org.apache.lucene.util.hnsw.HnswGraphSearcher#search() [JIT compiled code]
13.33% 4655 jdk.internal.misc.ScopedMemoryAccess#loadFromMemorySegmentScopedInternal() [Inlined code]
at jdk.internal.misc.ScopedMemoryAccess#loadFromMemorySegment() [Inlined code]
at jdk.incubator.vector.ByteVector#fromMemorySegment0Template() [Inlined code]
at jdk.incubator.vector.Byte128Vector#fromMemorySegment0() [Inlined code]
at jdk.incubator.vector.ByteVector#fromMemorySegment() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport$MemorySegmentLoader#load() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#dotProductBody512() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#dotProductBody() [Inlined code]
at org.apache.lucene.internal.vectorization.PanamaVectorUtilSupport#uint8DotProduct() [Inlined code]
at org.apache.lucene.internal.vectorization.Lucene104MemorySegmentScalarQuantizedVectorScorer$RandomVectorScorerImpl#score() [JIT compiled code]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement off-heap scoring for OSQ 4, 7, and 8 bit representations [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #15257:
URL: https://github.com/apache/lucene/pull/15257#issuecomment-3358056205
@mccullocht you might find this interesting: https://github.com/apache/lucene/pull/15272
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement off-heap scoring for OSQ 4, 7, and 8 bit representations [lucene]
Posted by "mccullocht (via GitHub)" <gi...@apache.org>.
mccullocht commented on PR #15257:
URL: https://github.com/apache/lucene/pull/15257#issuecomment-3358133079
I plan to try the simplest thing first and just copy the dot product code for byte[] x MemorySegment to see if that yields an improvement, then go from there. I hoped that the JVM would monomorphize these calls but I guess not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement off-heap scoring for OSQ 4, 7, and 8 bit representations [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #15257:
URL: https://github.com/apache/lucene/pull/15257#issuecomment-3356512508
@mccullocht maybe we only do the byte part of the comparisons off-heap? Then apply the corrections all on heap. I would assume applying corrections is pretty cheap, but even then if we did it in bulk, maybe on-heap bulk correction application is pretty fast?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement off-heap scoring for OSQ 4, 7, and 8 bit representations [lucene]
Posted by "mccullocht (via GitHub)" <gi...@apache.org>.
mccullocht commented on PR #15257:
URL: https://github.com/apache/lucene/pull/15257#issuecomment-3362432115
Ok, repeated the experiment with a specific byte[] x MemorySegment implementation. In the luceneutil benchmarks I'm not suffering from the same inlining/pollution issues. The profiles remain the same where suddenly `reduceLanes()` becomes very expensive. I have not tried this on other hardware (e.g. a Mac) yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org