You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "airborne12 (via GitHub)" <gi...@apache.org> on 2026/02/04 10:24:45 UTC
[PR] [fix](inverted index) Fix empty string MATCH on keyword index returning wrong results [doris]
airborne12 opened a new pull request, #60500:
URL: https://github.com/apache/doris/pull/60500
## Proposed changes
Fix empty string MATCH on keyword index returning wrong results.
The multi-analyzer feature commit (2c950e140a5) incorrectly added an empty string check that prevented `MATCH ''` from finding rows with empty string values in keyword indexes.
For keyword index (no tokenization), empty string is a valid exact match value and should be matchable. The previous code incorrectly skipped empty strings with the comment "empty query should match nothing", which is wrong for keyword indexes.
## Problem
```sql
-- Table with keyword index (no parser)
CREATE TABLE test (id INT, col TEXT, INDEX idx(col) USING INVERTED);
INSERT INTO test VALUES (1, ''), (2, 'data');
-- Before fix: returns 0 (WRONG!)
-- After fix: returns 1 (CORRECT!)
SELECT count() FROM test WHERE col MATCH '';
```
## Changes
This fix removes the empty string check for keyword index paths in:
- `be/src/vec/functions/match.cpp` (slow path)
- `be/src/olap/rowset/segment_v2/inverted_index_reader.cpp` (index path)
- `be/src/olap/rowset/segment_v2/inverted_index/analyzer/analyzer.cpp`
Added regression test `test_empty_string_match.groovy` to cover:
- Empty string match on keyword index (both index and slow paths)
- Empty string match on tokenized index (should return 0)
- match_any and match_all with empty string
## Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test
- [ ] No need to test
- Behavior changed:
- [x] Yes. `MATCH ''` on keyword index now correctly matches rows with empty string values.
- Does this need documentation?
- [ ] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Fix empty string MATCH on keyword index returning wrong results [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60500:
URL: https://github.com/apache/doris/pull/60500#issuecomment-3850762595
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Fix empty string MATCH on keyword index returning wrong results [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #60500:
URL: https://github.com/apache/doris/pull/60500#issuecomment-3846580939
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Fix empty string MATCH on keyword index returning wrong results [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60500:
URL: https://github.com/apache/doris/pull/60500#issuecomment-3847570253
# BE Regression && UT Coverage Report
Increment line coverage `25.00% (1/4)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60500_45700bd7b1a461d6112bde23a69355fcb0e6c89c_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60500_45700bd7b1a461d6112bde23a69355fcb0e6c89c_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.26% (26461/36118) |
| Line Coverage | 56.33% (280502/497977) |
| Region Coverage | 53.95% (234577/434797) |
| Branch Coverage | 55.67% (101050/181509) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Fix empty string MATCH on keyword index returning wrong results [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60500:
URL: https://github.com/apache/doris/pull/60500#issuecomment-3847251234
# BE UT Coverage Report
Increment line coverage `25.00% (1/4)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/45700bd7b1a461d6112bde23a69355fcb0e6c89c_45700bd7b1a461d6112bde23a69355fcb0e6c89c/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/45700bd7b1a461d6112bde23a69355fcb0e6c89c_45700bd7b1a461d6112bde23a69355fcb0e6c89c/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.56% (19370/36854) |
| Line Coverage | 36.05% (179935/499172) |
| Region Coverage | 32.38% (139350/430381) |
| Branch Coverage | 33.41% (60398/180779) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Fix empty string MATCH on keyword index returning wrong results [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #60500:
URL: https://github.com/apache/doris/pull/60500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Fix empty string MATCH on keyword index returning wrong results [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60500:
URL: https://github.com/apache/doris/pull/60500#issuecomment-3850762714
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Fix empty string MATCH on keyword index returning wrong results [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60500:
URL: https://github.com/apache/doris/pull/60500#issuecomment-3850758986
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Fix empty string MATCH on keyword index returning wrong results [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #60500:
URL: https://github.com/apache/doris/pull/60500#issuecomment-3846586074
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Fix empty string MATCH on keyword index returning wrong results [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60500:
URL: https://github.com/apache/doris/pull/60500#issuecomment-3846926725
<details>
<summary>ClickBench: <b>Total hot run time: 28.49 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 45700bd7b1a461d6112bde23a69355fcb0e6c89c, data reload: false
query1 0.05 0.04 0.04
query2 0.09 0.04 0.04
query3 0.26 0.09 0.09
query4 1.60 0.11 0.11
query5 0.27 0.24 0.26
query6 1.17 0.68 0.67
query7 0.03 0.03 0.03
query8 0.04 0.04 0.04
query9 0.56 0.50 0.48
query10 0.55 0.55 0.54
query11 0.14 0.09 0.10
query12 0.14 0.10 0.12
query13 0.63 0.63 0.61
query14 1.05 1.08 1.07
query15 0.87 0.87 0.87
query16 0.42 0.45 0.42
query17 1.08 1.10 1.12
query18 0.23 0.21 0.21
query19 2.02 2.03 2.01
query20 0.02 0.01 0.01
query21 15.40 0.23 0.15
query22 5.32 0.05 0.05
query23 16.04 0.28 0.11
query24 1.56 0.59 0.34
query25 0.14 0.06 0.11
query26 0.14 0.14 0.13
query27 0.08 0.08 0.05
query28 5.24 1.15 0.97
query29 12.63 3.94 3.19
query30 0.29 0.13 0.12
query31 2.84 0.63 0.41
query32 3.24 0.60 0.49
query33 3.22 3.26 3.24
query34 16.15 5.43 4.73
query35 4.85 4.80 4.81
query36 0.65 0.50 0.48
query37 0.12 0.08 0.07
query38 0.08 0.04 0.04
query39 0.04 0.03 0.04
query40 0.19 0.16 0.16
query41 0.09 0.03 0.03
query42 0.05 0.03 0.03
query43 0.05 0.04 0.04
Total cold run time: 99.63 s
Total hot run time: 28.49 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Fix empty string MATCH on keyword index returning wrong results [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60500:
URL: https://github.com/apache/doris/pull/60500#issuecomment-3850759093
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Fix empty string MATCH on keyword index returning wrong results [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60500:
URL: https://github.com/apache/doris/pull/60500#issuecomment-3846853607
<details>
<summary>TPC-H: <b>Total hot run time: 32021 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 45700bd7b1a461d6112bde23a69355fcb0e6c89c, data reload: false
------ Round 1 ----------------------------------
q1 17627 5197 5053 5053
q2 2017 311 189 189
q3 10245 1309 764 764
q4 10223 894 309 309
q5 7547 2154 1945 1945
q6 204 181 148 148
q7 886 728 610 610
q8 9269 1376 1102 1102
q9 5168 4773 4904 4773
q10 6880 1960 1557 1557
q11 519 298 275 275
q12 331 378 225 225
q13 17771 4069 3233 3233
q14 232 265 212 212
q15 884 820 829 820
q16 667 671 611 611
q17 642 774 510 510
q18 6771 6557 7394 6557
q19 1277 1023 640 640
q20 415 400 247 247
q21 2856 2228 1955 1955
q22 388 335 286 286
Total cold run time: 102819 ms
Total hot run time: 32021 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5534 5466 5536 5466
q2 286 360 260 260
q3 2466 2848 2487 2487
q4 1470 1871 1534 1534
q5 4632 4612 4760 4612
q6 227 179 139 139
q7 1999 1952 1868 1868
q8 2541 2339 2380 2339
q9 7676 7654 7647 7647
q10 2818 2930 2594 2594
q11 543 475 449 449
q12 684 748 588 588
q13 3723 4022 3230 3230
q14 283 312 267 267
q15 837 798 781 781
q16 634 689 642 642
q17 1086 1248 1258 1248
q18 7495 7247 7308 7247
q19 837 853 844 844
q20 1962 2061 1924 1924
q21 4546 4133 4023 4023
q22 558 562 521 521
Total cold run time: 52837 ms
Total hot run time: 50710 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org