You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "ChrisHegarty (via GitHub)" <gi...@apache.org> on 2025/08/13 15:44:56 UTC
[PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]
ChrisHegarty opened a new pull request, #15062:
URL: https://github.com/apache/lucene/pull/15062
This commit reworks the off-heap float32 bulk scoring to reduce garbage creation. The JVM fails to optimise things as we'd like, so we can just avoid some of the patterns in the first place!
1. Avoid slicing memory segments: our index input can span over up to 16GB by default, so we can just score a single segment (rather than slicing per scoring operation.
2. The hierarchy to allow scoring against an on-heap and off-heap query does not optimise well, so just flatten this and do the most obvious and straightforward thing - passing and switch on a null array or invalid address offset.
Before
```
Benchmark (pollute) (size) Mode Cnt Score Error Units
VectorScorerFloat32Benchmark.dotProductOptBulkScore false 1024 avgt 15 3.411 ± 0.262 ms/op
VectorScorerFloat32Benchmark.dotProductOptBulkScore:gc.alloc.rate false 1024 avgt 15 3.935 ± 0.012 MB/sec
VectorScorerFloat32Benchmark.dotProductOptBulkScore:gc.alloc.rate.norm false 1024 avgt 15 14120.816 ± 1070.540 B/op
VectorScorerFloat32Benchmark.dotProductOptBulkScore:gc.count false 1024 avgt 15 ≈ 0 counts
```
After
```
Benchmark (size) Mode Cnt Score Error Units
VectorScorerFloat32Benchmark.dotProductOptBulkScore 1024 avgt 15 3.557 ± 0.254 ms/op
VectorScorerFloat32Benchmark.dotProductOptBulkScore:gc.alloc.rate 1024 avgt 15 240.211 ± 17.073 MB/sec
VectorScorerFloat32Benchmark.dotProductOptBulkScore:gc.alloc.rate.norm 1024 avgt 15 894804.925 ± 1074.917 B/op
VectorScorerFloat32Benchmark.dotProductOptBulkScore:gc.count 1024 avgt 15 4.000 counts
VectorScorerFloat32Benchmark.dotProductOptBulkScore:gc.time 1024 avgt 15 24.000 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty merged PR #15062:
URL: https://github.com/apache/lucene/pull/15062
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on PR #15062:
URL: https://github.com/apache/lucene/pull/15062#issuecomment-3184460782
Screenshot from the lucene nightlies best show the problem!
<img width="1467" height="719" alt="Screenshot 2025-08-13 at 16 45 58" src="https://github.com/user-attachments/assets/cca12c51-2e6c-4d0b-bbf9-34bd3370f888" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on PR #15062:
URL: https://github.com/apache/lucene/pull/15062#issuecomment-3185388460
dataset: 1M cohere 768d
linux-x64 (m6i.2xlarge, x64, Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz, AVX 512)
```
baseline
recall latency(ms) netCPU avgCpuCount nDoc topK fanout maxConn beamWidth quantized index(s) index_docs/s force_merge(s) num_segments index_size(MB) vec_disk(MB) vec_RAM(MB) indexType
0.941 2.512 2.496 0.994 1000000 100 50 64 250 no 969.02 1031.97 319.90 1 3019.55 2929.688 2929.688 HNSW
candidate
recall latency(ms) netCPU avgCpuCount nDoc topK fanout maxConn beamWidth quantized index(s) index_docs/s force_merge(s) num_segments index_size(MB) vec_disk(MB) vec_RAM(MB) indexType
0.942 2.336 2.329 0.997 1000000 100 50 64 250 no 997.27 1002.73 257.97 1 3018.74 2929.688 2929.688 HNSW
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #15062:
URL: https://github.com/apache/lucene/pull/15062#issuecomment-3184473701
Could you run a lucene-util end to end test to see if this significantly changes indexing or search performance? I would expect any perf changes to be very minimal right? Its only focus is removing the temporary object creation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #15062:
URL: https://github.com/apache/lucene/pull/15062#issuecomment-3184462026
😱
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15062:
URL: https://github.com/apache/lucene/pull/15062#issuecomment-3184477083
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15062:
URL: https://github.com/apache/lucene/pull/15062#issuecomment-3184457239
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org