You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "feiniaofeiafei (via GitHub)" <gi...@apache.org> on 2026/10/14 08:56:05 UTC
[PR] [fix](tablet-prune) Prefer equality bounds when pruning hash tablets [doris]
feiniaofeiafei opened a new pull request, #69068:
URL: https://github.com/apache/doris/pull/69068
### What problem does this PR solve?
Problem Summary: When an equality bound and an IN predicate coexist on a hash distribution column, the pruner enumerates the IN values even though the equality already restricts the column to one value. Prefer inclusive, equal bounds before IN enumeration so redundant IN predicates cannot widen the candidate tablet set. Keep scan predicates unchanged, including when the equality contradicts the IN list.
### Release note
Improve tablet pruning when equality bounds and IN predicates coexist on hash distribution columns.
### Check List (For Author)
- Test: Unit Test — ./run-fe-ut.sh --run org.apache.doris.planner.HashDistributionPrunerTest (3 passed); FE Checkstyle passed.
- Behavior changed: Yes — use equality bounds before enumerating IN values; query filtering semantics are unchanged.
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tablet-prune) Prefer equality bounds when pruning hash tablets [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69068:
URL: https://github.com/apache/doris/pull/69068#issuecomment-6167909113
<details>
<summary>TPC-H: <b>Total hot run time: 29136 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 66adeebc7002b02708e2bc80b1e5cb3356a00b2f, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17573 5604 5493 5493
q2 2137 324 263 263
q3 10198 1423 856 856
q4 4686 483 343 343
q5 7423 790 541 541
q6 193 185 157 157
q7 759 812 605 605
q8 9246 1384 1108 1108
q9 5565 4256 4244 4244
q10 6831 1292 1023 1023
q11 443 259 243 243
q12 637 400 288 288
q13 18071 3593 2995 2995
q14 271 271 243 243
q15 q16 745 724 668 668
q17 1210 746 567 567
q18 7164 6258 6162 6162
q19 1247 960 596 596
q20 398 341 222 222
q21 5462 2248 2432 2248
q22 363 311 271 271
Total cold run time: 100622 ms
Total hot run time: 29136 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6123 6126 6164 6126
q2 708 558 556 556
q3 5244 5418 4983 4983
q4 1982 2116 1532 1532
q5 5275 5392 6053 5392
q6 256 202 154 154
q7 2373 2179 1930 1930
q8 3168 2953 2775 2775
q9 8138 8127 8164 8127
q10 3813 3679 3363 3363
q11 625 431 397 397
q12 678 734 507 507
q13 3611 3907 3308 3308
q14 317 336 297 297
q15 q16 715 720 672 672
q17 8386 7493 7416 7416
q18 13148 12365 13066 12365
q19 941 806 788 788
q20 2206 2163 1933 1933
q21 5765 4677 4779 4677
q22 482 442 393 393
Total cold run time: 73954 ms
Total hot run time: 67691 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tablet-prune) Prefer equality bounds when pruning hash tablets [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69068:
URL: https://github.com/apache/doris/pull/69068#issuecomment-6169668054
<details>
<summary>TPC-DS: <b>Total hot run time: 152427 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 66adeebc7002b02708e2bc80b1e5cb3356a00b2f, data reload: false
query5 4322 578 471 471
query6 428 210 189 189
query7 4799 438 223 223
query8 316 175 154 154
query9 8767 3944 3951 3944
query10 479 315 256 256
query11 5933 3806 3516 3516
query12 146 90 85 85
query13 1230 432 333 333
query14 6118 3988 3745 3745
query14_1 3533 3476 3494 3476
query15 207 202 182 182
query16 952 437 465 437
query17 865 661 539 539
query18 2428 433 314 314
query19 196 167 135 135
query20 85 86 81 81
query21 227 143 111 111
query22 13059 12991 12832 12832
query23 13638 12930 12569 12569
query23_1 12657 12487 12570 12487
query24 7092 825 476 476
query24_1 478 476 471 471
query25 538 428 368 368
query26 1316 258 144 144
query27 2748 449 284 284
query28 4590 1962 1934 1934
query29 1608 591 451 451
query30 302 225 184 184
query31 1022 925 807 807
query32 146 104 89 89
query33 521 309 255 255
query34 904 860 502 502
query35 756 754 645 645
query36 833 816 703 703
query37 121 102 97 97
query38 1787 1768 1702 1702
query39 801 800 754 754
query39_1 738 752 747 747
query40 228 120 100 100
query41 74 68 67 67
query42 88 86 83 83
query43 365 378 334 334
query44 1292 695 707 695
query45 182 178 162 162
query46 856 975 617 617
query47 2981 2979 2935 2935
query48 293 304 225 225
query49 573 402 285 285
query50 675 281 209 209
query51 10656 10564 10726 10564
query52 78 76 70 70
query53 185 207 157 157
query54 250 192 173 173
query55 70 69 64 64
query56 236 220 219 219
query57 1608 1571 1537 1537
query58 290 256 257 256
query59 2102 2164 1996 1996
query60 292 243 226 226
query61 141 143 143 143
query62 393 341 295 295
query63 202 160 169 160
query64 2706 938 766 766
query65 3288 3232 3259 3232
query66 1813 424 297 297
query67 20053 20087 20227 20087
query68 3259 1005 613 613
query69 394 310 242 242
query70 849 770 755 755
query71 285 226 207 207
query72 2832 2590 2273 2273
query73 529 524 295 295
query74 4621 4460 4327 4327
query75 2372 2295 1943 1943
query76 2401 1006 593 593
query77 334 365 285 285
query78 9697 9771 9177 9177
query79 958 907 506 506
query80 729 422 359 359
query81 562 322 273 273
query82 578 134 111 111
query83 299 196 176 176
query84 327 124 107 107
query85 882 412 378 378
query86 353 219 232 219
query87 2029 1941 1819 1819
query88 3630 2715 2692 2692
query89 338 302 259 259
query90 1759 180 177 177
query91 160 151 127 127
query92 101 90 85 85
query93 1030 1002 575 575
query94 510 305 288 288
query95 572 329 312 312
query96 673 529 229 229
query97 2482 2417 2317 2317
query98 165 150 143 143
query99 784 759 651 651
Total cold run time: 234909 ms
Total hot run time: 152427 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tablet-prune) Prefer equality bounds when pruning hash tablets [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69068:
URL: https://github.com/apache/doris/pull/69068#issuecomment-6169772834
<details>
<summary>ClickBench: <b>Total hot run time: 24.66 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 66adeebc7002b02708e2bc80b1e5cb3356a00b2f, data reload: false
query1 0.00 0.00 0.00
query2 0.10 0.05 0.05
query3 0.26 0.12 0.11
query4 1.61 0.12 0.12
query5 0.24 0.25 0.25
query6 1.16 0.69 0.69
query7 0.04 0.00 0.00
query8 0.05 0.03 0.03
query9 0.41 0.36 0.34
query10 0.57 0.56 0.58
query11 0.21 0.14 0.14
query12 0.18 0.15 0.14
query13 0.50 0.50 0.49
query14 1.03 1.01 1.00
query15 0.64 0.63 0.61
query16 0.35 0.34 0.35
query17 1.13 1.14 1.12
query18 0.23 0.21 0.20
query19 2.09 2.06 1.97
query20 0.02 0.01 0.02
query21 15.41 0.28 0.14
query22 4.43 0.05 0.05
query23 16.25 0.33 0.12
query24 3.27 0.59 0.35
query25 0.12 0.06 0.04
query26 1.05 0.26 0.16
query27 0.05 0.03 0.04
query28 1.59 0.59 0.40
query29 12.48 4.33 3.48
query30 0.29 0.14 0.15
query31 2.77 0.60 0.34
query32 3.23 0.62 0.52
query33 3.10 3.22 3.37
query34 15.74 4.05 3.38
query35 3.35 3.37 3.35
query36 0.54 0.48 0.42
query37 0.10 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.04 0.03
query40 0.17 0.16 0.14
query41 0.08 0.03 0.02
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 95 s
Total hot run time: 24.66 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tablet-prune) Prefer equality bounds when pruning hash tablets [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #69068:
URL: https://github.com/apache/doris/pull/69068
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tablet-prune) Prefer equality bounds when pruning hash tablets [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69068:
URL: https://github.com/apache/doris/pull/69068#issuecomment-6171177465
# FE UT Coverage Report
Increment line coverage ` 66.67% (4/6)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69068_66adeebc7002b02708e2bc80b1e5cb3356a00b2f/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69068_66adeebc7002b02708e2bc80b1e5cb3356a00b2f/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tablet-prune) Prefer equality bounds when pruning hash tablets [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69068:
URL: https://github.com/apache/doris/pull/69068#discussion_r4279158829
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HashDistributionPruner.java:
##########
@@ -102,18 +102,19 @@ public Collection<Long> prune(int columnId, PartitionKey hashKey, int complex) {
// return all subPartition
return getAllTabletIds();
}
+ // An equality bound is more selective than an IN predicate on the same column.
+ // Keep the remaining predicates for row filtering, including contradictory IN values.
+ if (filter.lowerBoundInclusive && filter.upperBoundInclusive
+ && filter.lowerBound != null && filter.upperBound != null
+ && 0 == filter.lowerBound.compareLiteral(filter.upperBound)) {
Review Comment:
[P2] Require exact distribution values before preferring bounds
`compareLiteral() == 0` does not establish an identical distribution value for strings: `'a'` and `'a\0b'` compare equal, but hash to buckets 3 and 17 with 32 buckets. For example, with the supported `disable_nereids_rules='REWRITE_FILTER_EXPRESSION,CONSTANT_PROPAGATION,INFER_PREDICATES'` setting, this reduced plan retains all three predicates:
```text
Filter(k >= 'a' AND k <= 'a\0b' AND k IN ('a', 'a\0b'))
OlapScan(VARCHAR k, HASH(k), 32 buckets)
```
`ExpressionColumnFilterConverter` supplies those distinct inclusive bounds plus the IN list. This branch now selects only the `'a'` tablet, omitting the matching `'a\0b'` row; previously IN enumeration selected both tablets. Retaining scan predicates cannot recover a skipped row. Require exact typed value identity before overriding IN enumeration and add a NUL-prefix range/IN regression. `LiteralExpr.equals()` also delegates to `compareLiteral()`, so it would not fix this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tablet-prune) Prefer equality bounds when pruning hash tablets [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69068:
URL: https://github.com/apache/doris/pull/69068#issuecomment-6169474950
<details>
<summary>TPC-H: <b>Total hot run time: 29273 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 66adeebc7002b02708e2bc80b1e5cb3356a00b2f, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17577 5638 5516 5516
q2 2101 303 251 251
q3 10244 1397 850 850
q4 4675 472 341 341
q5 7437 804 538 538
q6 185 181 153 153
q7 766 794 607 607
q8 9240 1329 1151 1151
q9 5524 4276 4276 4276
q10 6804 1296 1014 1014
q11 428 263 239 239
q12 631 398 282 282
q13 18094 3706 2985 2985
q14 274 263 248 248
q15 q16 746 725 677 677
q17 1201 737 560 560
q18 7127 6230 6194 6194
q19 1121 950 612 612
q20 407 344 232 232
q21 5512 2284 2366 2284
q22 363 309 263 263
Total cold run time: 100457 ms
Total hot run time: 29273 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6151 6134 6135 6134
q2 697 555 560 555
q3 5190 5409 4966 4966
q4 2012 2126 1528 1528
q5 5290 5430 6060 5430
q6 249 193 151 151
q7 2333 2123 1880 1880
q8 3263 2970 2706 2706
q9 8184 8143 8184 8143
q10 3845 3707 3305 3305
q11 612 432 410 410
q12 687 703 503 503
q13 3554 3928 3257 3257
q14 327 315 284 284
q15 q16 676 735 652 652
q17 8400 7599 7351 7351
q18 13191 12276 13070 12276
q19 910 794 791 791
q20 2170 2151 1950 1950
q21 5793 4690 4759 4690
q22 502 446 422 422
Total cold run time: 74036 ms
Total hot run time: 67384 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tablet-prune) Prefer equality bounds when pruning hash tablets [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69068:
URL: https://github.com/apache/doris/pull/69068#issuecomment-6164669151
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tablet-prune) Prefer equality bounds when pruning hash tablets [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69068:
URL: https://github.com/apache/doris/pull/69068#issuecomment-6171013962
# FE Regression Coverage Report
Increment line coverage ` 10.20% (5/49)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69068_66adeebc7002b02708e2bc80b1e5cb3356a00b2f_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69068_66adeebc7002b02708e2bc80b1e5cb3356a00b2f_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tablet-prune) Prefer equality bounds when pruning hash tablets [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #69068:
URL: https://github.com/apache/doris/pull/69068#issuecomment-6184635244
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tablet-prune) Prefer equality bounds when pruning hash tablets [doris]
Posted by "feiniaofeiafei (via GitHub)" <gi...@apache.org>.
feiniaofeiafei commented on PR #69068:
URL: https://github.com/apache/doris/pull/69068#issuecomment-6164671953
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org