You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "ChrisHegarty (via GitHub)" <gi...@apache.org> on 2025/08/13 15:44:56 UTC

[PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]

ChrisHegarty opened a new pull request, #15062:
URL: https://github.com/apache/lucene/pull/15062

   This commit reworks the off-heap float32 bulk scoring to reduce garbage creation. The JVM fails to optimise things as we'd like, so we can just avoid some of the patterns in the first place!
   
   1. Avoid slicing memory segments: our index input can span over up to 16GB by default, so we can just score a single segment (rather than slicing per scoring operation.
   2. The hierarchy to allow scoring against an on-heap and off-heap query does not optimise well, so just flatten this and do the most obvious and straightforward thing - passing and switch on a null array or invalid address offset. 
   
   Before
   ```
   Benchmark                                                               (pollute)  (size)  Mode  Cnt      Score      Error   Units
   VectorScorerFloat32Benchmark.dotProductOptBulkScore                         false    1024  avgt   15      3.411 ±    0.262   ms/op
   VectorScorerFloat32Benchmark.dotProductOptBulkScore:gc.alloc.rate           false    1024  avgt   15      3.935 ±    0.012  MB/sec
   VectorScorerFloat32Benchmark.dotProductOptBulkScore:gc.alloc.rate.norm      false    1024  avgt   15  14120.816 ± 1070.540    B/op
   VectorScorerFloat32Benchmark.dotProductOptBulkScore:gc.count                false    1024  avgt   15        ≈ 0             counts
   ```
   
   After
   ```
   Benchmark                                                               (size)  Mode  Cnt       Score      Error   Units
   VectorScorerFloat32Benchmark.dotProductOptBulkScore                       1024  avgt   15       3.557 ±    0.254   ms/op
   VectorScorerFloat32Benchmark.dotProductOptBulkScore:gc.alloc.rate         1024  avgt   15     240.211 ±   17.073  MB/sec
   VectorScorerFloat32Benchmark.dotProductOptBulkScore:gc.alloc.rate.norm    1024  avgt   15  894804.925 ± 1074.917    B/op
   VectorScorerFloat32Benchmark.dotProductOptBulkScore:gc.count              1024  avgt   15       4.000             counts
   VectorScorerFloat32Benchmark.dotProductOptBulkScore:gc.time               1024  avgt   15      24.000                 ms
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]

Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty merged PR #15062:
URL: https://github.com/apache/lucene/pull/15062


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]

Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on PR #15062:
URL: https://github.com/apache/lucene/pull/15062#issuecomment-3184460782

   Screenshot from the lucene nightlies best show the problem!
   
   <img width="1467" height="719" alt="Screenshot 2025-08-13 at 16 45 58" src="https://github.com/user-attachments/assets/cca12c51-2e6c-4d0b-bbf9-34bd3370f888" />
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]

Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on PR #15062:
URL: https://github.com/apache/lucene/pull/15062#issuecomment-3185388460

   dataset: 1M cohere 768d
   
   linux-x64 (m6i.2xlarge, x64, Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz, AVX 512)
   ```
   baseline 
   recall  latency(ms)  netCPU  avgCpuCount     nDoc  topK  fanout  maxConn  beamWidth  quantized  index(s)  index_docs/s  force_merge(s)  num_segments  index_size(MB)  vec_disk(MB)  vec_RAM(MB)  indexType
   0.941        2.512   2.496        0.994  1000000   100      50       64        250         no    969.02       1031.97          319.90             1         3019.55      2929.688     2929.688       HNSW
   
   candidate 
   recall  latency(ms)  netCPU  avgCpuCount     nDoc  topK  fanout  maxConn  beamWidth  quantized  index(s)  index_docs/s  force_merge(s)  num_segments  index_size(MB)  vec_disk(MB)  vec_RAM(MB)  indexType
   0.942        2.336   2.329        0.997  1000000   100      50       64        250         no    997.27       1002.73          257.97             1         3018.74      2929.688     2929.688       HNSW
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]

Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #15062:
URL: https://github.com/apache/lucene/pull/15062#issuecomment-3184473701

   Could you run a lucene-util end to end test to see if this significantly changes indexing or search performance? I would expect any perf changes to be very minimal right? Its only focus is removing the temporary object creation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]

Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #15062:
URL: https://github.com/apache/lucene/pull/15062#issuecomment-3184462026

   😱 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15062:
URL: https://github.com/apache/lucene/pull/15062#issuecomment-3184477083

   This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Rework off-heap float32 bulk scoring to reduce garbage creation [lucene]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15062:
URL: https://github.com/apache/lucene/pull/15062#issuecomment-3184457239

   This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org