You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Pranshu-S (via GitHub)" <gi...@apache.org> on 2026/10/04 12:08:13 UTC

Re: [PR] Add dedup-aware HNSW format that builds the graph over distinct vectors and expands matches to all referencing documents [lucene]

Pranshu-S commented on PR #16731:
URL: https://github.com/apache/lucene/pull/16731#issuecomment-5979760229

   Interestingly, from my local runs it seems that there is a tip-off point where this new Reader/Writer pays off: When the number of unqiue vectors within a field are less - that is same vector is being repeated in the same field. Logically it can happen - such as when different records reference the same underlying item embedding.
   
   Here is what my results came out as:
   
   Total Unique = 100
   ```
   ==========================================================================
   COMPARISON  (candidate vs baseline;  delta/pct = candidate - baseline)
               float32 vectors, filtered KNN search
   metrics are mean±stddev over iterations (baseline n=5, candidate n=5)
   ==========================================================================
   metric             baseline(n=5)      candidate(n=5)     delta      pct
   -----------------  -----------------  -----------------  ---------  ------
   recall             0.809±0.008        0.982±0.000        +0.173     +21.4%
   latency(ms)        0.158±0.013        0.212±0.010        +0.055     +34.8%
   netCPU             0.152±0.013        0.206±0.010        +0.055     +36.0%
   avgCpuCount        0.962±0.003        0.971±0.003        +0.009     +0.9%
   nDoc               10000              10000
   searchType         KNN                KNN
   topK               10                 10
   fanout             100                100
   resultSimilarity   N/A                N/A
   decay              N/A                N/A
   resultCount        10.000             10.000
   maxConn            32                 32
   beamWidth          200                200
   quantized          7 bits             7 bits
   indexEncoding      float32            float32
   visited            230.200±7.155      100.000±0.000      -130.200   -56.6%
   index(s)           1.568±0.038        0.890±0.019        -0.678     -43.2%
   index_docs/s       6379.994±161.224   11232.210±231.098  +4852.216  +76.1%
   merge(s)           0.000±0.000        0.000±0.000        +0.000
   force_merge(s)     2.180±0.055        1.060±0.045        -1.120     -51.4%
   num_segments       1.000±0.000        1.000±0.000        +0.000     +0.0%
   index_size(MB)     0.658±0.004        0.620±0.000        -0.038     -5.8%
   filterStrategy     index-time-filter  index-time-filter
   filterSelectivity  0.50               0.50
   overSample         1.000              1.000
   vec_disk(MB)       48.981±0.000       48.981±0.000       +0.000     +0.0%
   vec_RAM(MB)        9.918±0.000        9.918±0.000        +0.000     +0.0%
   bp-reorder         false              false
   indexType          HNSW               HNSW
   rerank             no                 no
   ```
   Total Unique = 1000
   ```
   ==========================================================================
   COMPARISON  (candidate vs baseline;  delta/pct = candidate - baseline)
               float32 vectors, filtered KNN search
   metrics are mean±stddev over iterations (baseline n=5, candidate n=5)
   ==========================================================================
   metric             baseline(n=5)      candidate(n=5)     delta      pct
   -----------------  -----------------  -----------------  ---------  ------
   recall             0.964±0.002        0.965±0.002        +0.001     +0.1%
   latency(ms)        0.225±0.004        0.224±0.028        -0.001     -0.3%
   netCPU             0.217±0.003        0.217±0.028        +0.000     +0.2%
   avgCpuCount        0.966±0.006        0.970±0.007        +0.004     +0.4%
   nDoc               10000              10000
   searchType         KNN                KNN
   topK               10                 10
   fanout             100                100
   resultSimilarity   N/A                N/A
   decay              N/A                N/A
   resultCount        10.000             10.000
   maxConn            32                 32
   beamWidth          200                200
   quantized          7 bits             7 bits
   indexEncoding      float32            float32
   visited            496.000±8.746      422.400±2.302      -73.600    -14.8%
   index(s)           2.406±0.065        1.920±0.166        -0.486     -20.2%
   index_docs/s       4159.980±109.441   5241.906±499.526   +1081.926  +26.0%
   merge(s)           0.000±0.000        0.000±0.000        +0.000
   force_merge(s)     3.450±0.071        1.666±0.047        -1.784     -51.7%
   num_segments       1.000±0.000        1.000±0.000        +0.000     +0.0%
   index_size(MB)     5.140±0.000        5.100±0.000        -0.040     -0.8%
   filterStrategy     index-time-filter  index-time-filter
   filterSelectivity  0.50               0.50
   overSample         1.000              1.000
   vec_disk(MB)       48.981±0.000       48.981±0.000       +0.000     +0.0%
   vec_RAM(MB)        9.918±0.000        9.918±0.000        +0.000     +0.0%
   bp-reorder         false              false
   indexType          HNSW               HNSW
   rerank             no                 no


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org