You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Pranshu-S (via GitHub)" <gi...@apache.org> on 2026/10/04 12:08:13 UTC
Re: [PR] Add dedup-aware HNSW format that builds the graph over distinct vectors and expands matches to all referencing documents [lucene]
Pranshu-S commented on PR #16731:
URL: https://github.com/apache/lucene/pull/16731#issuecomment-5979760229
Interestingly, from my local runs it seems that there is a tip-off point where this new Reader/Writer pays off: When the number of unqiue vectors within a field are less - that is same vector is being repeated in the same field. Logically it can happen - such as when different records reference the same underlying item embedding.
Here is what my results came out as:
Total Unique = 100
```
==========================================================================
COMPARISON (candidate vs baseline; delta/pct = candidate - baseline)
float32 vectors, filtered KNN search
metrics are mean±stddev over iterations (baseline n=5, candidate n=5)
==========================================================================
metric baseline(n=5) candidate(n=5) delta pct
----------------- ----------------- ----------------- --------- ------
recall 0.809±0.008 0.982±0.000 +0.173 +21.4%
latency(ms) 0.158±0.013 0.212±0.010 +0.055 +34.8%
netCPU 0.152±0.013 0.206±0.010 +0.055 +36.0%
avgCpuCount 0.962±0.003 0.971±0.003 +0.009 +0.9%
nDoc 10000 10000
searchType KNN KNN
topK 10 10
fanout 100 100
resultSimilarity N/A N/A
decay N/A N/A
resultCount 10.000 10.000
maxConn 32 32
beamWidth 200 200
quantized 7 bits 7 bits
indexEncoding float32 float32
visited 230.200±7.155 100.000±0.000 -130.200 -56.6%
index(s) 1.568±0.038 0.890±0.019 -0.678 -43.2%
index_docs/s 6379.994±161.224 11232.210±231.098 +4852.216 +76.1%
merge(s) 0.000±0.000 0.000±0.000 +0.000
force_merge(s) 2.180±0.055 1.060±0.045 -1.120 -51.4%
num_segments 1.000±0.000 1.000±0.000 +0.000 +0.0%
index_size(MB) 0.658±0.004 0.620±0.000 -0.038 -5.8%
filterStrategy index-time-filter index-time-filter
filterSelectivity 0.50 0.50
overSample 1.000 1.000
vec_disk(MB) 48.981±0.000 48.981±0.000 +0.000 +0.0%
vec_RAM(MB) 9.918±0.000 9.918±0.000 +0.000 +0.0%
bp-reorder false false
indexType HNSW HNSW
rerank no no
```
Total Unique = 1000
```
==========================================================================
COMPARISON (candidate vs baseline; delta/pct = candidate - baseline)
float32 vectors, filtered KNN search
metrics are mean±stddev over iterations (baseline n=5, candidate n=5)
==========================================================================
metric baseline(n=5) candidate(n=5) delta pct
----------------- ----------------- ----------------- --------- ------
recall 0.964±0.002 0.965±0.002 +0.001 +0.1%
latency(ms) 0.225±0.004 0.224±0.028 -0.001 -0.3%
netCPU 0.217±0.003 0.217±0.028 +0.000 +0.2%
avgCpuCount 0.966±0.006 0.970±0.007 +0.004 +0.4%
nDoc 10000 10000
searchType KNN KNN
topK 10 10
fanout 100 100
resultSimilarity N/A N/A
decay N/A N/A
resultCount 10.000 10.000
maxConn 32 32
beamWidth 200 200
quantized 7 bits 7 bits
indexEncoding float32 float32
visited 496.000±8.746 422.400±2.302 -73.600 -14.8%
index(s) 2.406±0.065 1.920±0.166 -0.486 -20.2%
index_docs/s 4159.980±109.441 5241.906±499.526 +1081.926 +26.0%
merge(s) 0.000±0.000 0.000±0.000 +0.000
force_merge(s) 3.450±0.071 1.666±0.047 -1.784 -51.7%
num_segments 1.000±0.000 1.000±0.000 +0.000 +0.0%
index_size(MB) 5.140±0.000 5.100±0.000 -0.040 -0.8%
filterStrategy index-time-filter index-time-filter
filterSelectivity 0.50 0.50
overSample 1.000 1.000
vec_disk(MB) 48.981±0.000 48.981±0.000 +0.000 +0.0%
vec_RAM(MB) 9.918±0.000 9.918±0.000 +0.000 +0.0%
bp-reorder false false
indexType HNSW HNSW
rerank no no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org