You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "guptashivam04 (via GitHub)" <gi...@apache.org> on 2026/10/22 03:03:52 UTC
[PR] branch-4.1: [fix](inverted index) Share BM25 statistics across parallel scanners of a tablet #69352 [doris]
guptashivam04 opened a new pull request, #69418:
URL: https://github.com/apache/doris/pull/69418
### What problem does this PR solve?
Issue Number: None
Related PR: #69352
Problem Summary:
Backport of #69352 to branch-4.1.
A parallel scan (`ParallelScannerBuilder`) splits one tablet into several scanners. Each `OlapScanner` then calls `CollectionStatistics::collect()` with only the `rs_splits` of its own split, so the BM25 doc count, avgdl and term doc frequencies come from a subset of the tablet's rowsets. Rows with identical text get different scores depending on which scanner reads them. Because the split depends on the tablet's row count and the core count, a row's score and its rank in `ORDER BY score()` change when unrelated rows are loaded, and pagination over a score-ordered search returns repeated and missing rows.
The reproduction in #69352 (8 rowsets with a different share of matching rows, forced split with `parallel_scan_min_rows_per_scanner=1024`) gives the 2,280 identical matching rows **15 distinct scores (1.34 to 5.81)** on 4.1.4, 4.1.5 and branch-4.1. With `enable_parallel_scan=false` they all get 3.647686.
Fix (same design as #69352, adapted to branch-4.1):
- `TabletCollectionStatistics` (in `storage/compaction/collection_statistics.h`) holds the `rs_splits` of a tablet's entire read source. The first scanner of the tablet to prepare collects the statistics once with `CollectionStatistics::collect()` over all of them, and every scanner of that tablet copies them into its own `CollectionStatistics`. Later scanners wait for the first collection and get its result or error.
- `CollectionStatistics::copy_collected_from()` copies the collected counters and resets the derived idf/avgdl caches, so no cache is shared between scanner threads.
- `ParallelScannerBuilder` creates one holder per tablet, only when the scan has a `score()` runtime. Both the by-rowid and by-segment build paths pass it to their scanners through `OlapScanner::Params`.
- The non-parallel path is unchanged. Each of its scanners already receives the tablet's full read source.
Differences from #69352: on branch-4.1, `CollectionStatistics` lives in `storage/compaction/`, has no `SearchPredicateCollector` constructor argument and no `collect_full_collection()`, so the holder keeps the read source's `rs_splits` and calls `collect()` on them. `OlapScanner::Params` is initialized positionally on this branch, so the new field is appended at the end.
Statistics remain per tablet. Making them consistent across tablets and BEs is out of scope.
### Release note
Fix BM25 `score()` returning different scores for identical rows of the same tablet when a parallel scan split the tablet into several scanners.
### Check List (For Author)
- Test
- [x] Regression test: `inverted_index_p0/test_bm25_score_parallel_scan_statistics` (same suite as #69352). Passes on branch-4.1 with this change.
- [x] Unit Test: `CollectionStatisticsDetailedTest.CopyCollectedFromScoresLikeSource`, `CollectionStatisticsTest.TabletCollectionStatisticsCollectsOnceAndCopiesToEveryCaller` (not run locally; left to CI)
- [x] Manual test: ran the reproduction from #69352 against branch-4.1 builds on macOS arm64.
| Build | `enable_parallel_scan=true` (split forced) | `enable_parallel_scan=false` |
|---|---|---|
| branch-4.1, without this change | 15 distinct scores (1.34 to 5.81) | 1 score (3.647686) |
| branch-4.1 `95846ddf` with this change | 1 score (3.647686), same over 5 repeated runs | 1 score (3.647686) |
- [ ] No need to test or manual test.
- Behavior changed:
- [x] Yes. Score queries that use a parallel scan now score every row of a tablet against statistics of the whole tablet. Scores can differ from earlier releases but are now independent of the scanner split.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](inverted index) Share BM25 statistics across parallel scanners of a tablet #69352 [doris]
Posted by "guptashivam04 (via GitHub)" <gi...@apache.org>.
guptashivam04 commented on PR #69418:
URL: https://github.com/apache/doris/pull/69418#issuecomment-6338674867
The `test_bm25_score_parallel_scan_statistics` failure came from fuzzy mode disabling `enable_common_expr_pushdown` on branch-4.1. I've pushed a fix that enables it in the test, like the other BM25 suites do. The other two P0 failures (`AbstractTestRegexpFunctions`, `test_time_series_compaction_policy`) aren't related. Could you re-run `run buildall`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](inverted index) Share BM25 statistics across parallel scanners of a tablet #69352 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69418:
URL: https://github.com/apache/doris/pull/69418#issuecomment-6352503284
# BE UT Coverage Report
Increment line coverage `33.82% (23/68)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/98be8b8d56506288da256f14c1379dc83ac6c345_98be8b8d56506288da256f14c1379dc83ac6c345/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/98be8b8d56506288da256f14c1379dc83ac6c345_98be8b8d56506288da256f14c1379dc83ac6c345/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 61.48% (26986/43893) |
| Line Coverage | 46.16% (281301/609447) |
| Region Coverage | 41.80% (222630/532658) |
| Branch Coverage | 43.33% (103449/238754) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](inverted index) Share BM25 statistics across parallel scanners of a tablet #69352 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69418:
URL: https://github.com/apache/doris/pull/69418#issuecomment-6353962435
# BE Regression && UT Coverage Report
Increment line coverage `97.06% (66/68)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69418_98be8b8d56506288da256f14c1379dc83ac6c345_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69418_98be8b8d56506288da256f14c1379dc83ac6c345_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 75.11% (32059/42683) |
| Line Coverage | 59.49% (359676/604565) |
| Region Coverage | 56.20% (299972/533715) |
| Branch Coverage | 57.09% (136138/238455) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](inverted index) Share BM25 statistics across parallel scanners of a tablet #69352 [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #69418:
URL: https://github.com/apache/doris/pull/69418
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](inverted index) Share BM25 statistics across parallel scanners of a tablet #69352 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69418:
URL: https://github.com/apache/doris/pull/69418#issuecomment-6339136970
# BE UT Coverage Report
Increment line coverage `33.82% (23/68)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/6a0b2e20338a87a89cb63df58e7ef828186e596f_6a0b2e20338a87a89cb63df58e7ef828186e596f/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/6a0b2e20338a87a89cb63df58e7ef828186e596f_6a0b2e20338a87a89cb63df58e7ef828186e596f/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 61.48% (26984/43893) |
| Line Coverage | 46.16% (281296/609447) |
| Region Coverage | 41.78% (222526/532658) |
| Branch Coverage | 43.33% (103455/238754) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](inverted index) Share BM25 statistics across parallel scanners of a tablet #69352 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69418:
URL: https://github.com/apache/doris/pull/69418#issuecomment-6357902225
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](inverted index) Share BM25 statistics across parallel scanners of a tablet #69352 [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #69418:
URL: https://github.com/apache/doris/pull/69418#issuecomment-6331841573
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](inverted index) Share BM25 statistics across parallel scanners of a tablet #69352 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69418:
URL: https://github.com/apache/doris/pull/69418#issuecomment-6331654558
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](inverted index) Share BM25 statistics across parallel scanners of a tablet #69352 [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #69418:
URL: https://github.com/apache/doris/pull/69418#issuecomment-6339186446
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](inverted index) Share BM25 statistics across parallel scanners of a tablet #69352 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69418:
URL: https://github.com/apache/doris/pull/69418#issuecomment-6357902718
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org