[PR] [fix](index) Recompute destination token counts during compaction [doris-thirdparty]

["Authorlove (via GitHub)" <gi...@apache.org>]

Authorlove opened a new pull request, #420:
URL: https://github.com/apache/doris-thirdparty/pull/420

   ### What problem does this PR solve?
   
   Issue Number: None
   
   Related PR: Apache Doris integration PR (to be linked)
   
   Problem Summary:
   
   `IndexWriter::indexCompaction` filters deleted documents through the row-ID translation map but copies the sum of every source field's token count into the first destination index and writes zero to the remaining destinations. Consequently BM25 uses stale average document lengths after deletes and incorrect per-segment statistics after a split.
   
   A minimal Doris reproduction uses a single-bucket UNIQUE KEY table with an English inverted index and automatic compaction disabled: retain `alpha` and `alpha alpha` followed by 14 `omega` tokens, and insert a delete-sign row containing 240 `alpha` plus 16 `omega` tokens. Compare `SEARCH('body:alpha') ORDER BY score() DESC` after full compaction with a fresh table containing just the two visible rows. For these two visible documents of lengths 1 and 16 and one deleted document of length 256, the clean index contains 17 tokens. Native index compaction retains a token count of 273: on Doris 4.1.5 the visible documents score 0.3069872 / 0.33349273, while rebuilding the same documents scores 0.28530657 / 0.20084909 and reverses their ranking.
   
   Accumulate exact term frequencies from retained postings into each destination field while merging terms, then write those destination totals alongside the remapped norms. This also repairs stale source totals when their postings still retain frequencies. It neither changes BM25's formula nor changes visibility/statistics before compaction.
   
   An index with norms but without term frequencies cannot supply an exact count: byte norms are quantized. Reject native compaction for that case before opening output writers; the companion Doris change selects its existing raw-column index rebuild path. Indexes without norms continue to compact without requiring frequencies.
   
   ### Release note
   
   Correct BM25 token statistics after inverted-index compaction removes or redistributes documents.
   
   ### Check List (For Author)
   
   - Test
       - [x] Unit Test
       - [x] Manual test (pre-fix SQL reproduction on Doris 4.1.5)
       - [ ] Regression test
       - Native CLucene ASAN build, using the repository's CMake/`cl_test` entrypoints.
       - `cl_test -v -p IndexCompaction IndexCompress IndexCompressV3 testBlockMaxScoreV3`: 16 tests passed.
       - Rebuilt the original `08d45e541fef1e48dc4ab00d2d5fde9569b49ee3` implementation with the new tests: 2 new tests fail (expected destination token count 42, actual 315; missing-frequency guard absent); the 4 existing compaction tests pass.
       - Coverage includes deleted long documents, a non-exact byte-norm length (41 tokens), two source/destination indexes, multiple fields, empty field values, and indexes with/without norms or frequencies.
       - A full Doris master binary has not been built or exercised locally; the companion PR records its integration validation separately.
   - Behavior changed:
       - [x] Yes. Compacted norms carry exact destination token totals. Native compaction rejects norms-bearing indexes without frequencies; callers must rebuild them.
   - Does this need documentation?
       - [x] No. Correctness fix; no new setting or on-disk format.
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

Re: [PR] [fix](index) Recompute destination token counts during compaction [doris-thirdparty]

["airborne12 (via GitHub)" <gi...@apache.org>]

airborne12 merged PR #420:
URL: https://github.com/apache/doris-thirdparty/pull/420


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org