You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Mryange (via GitHub)" <gi...@apache.org> on 2026/05/29 10:22:01 UTC
[PR] [fix](sort) keep heap TopN pruning for expression order keys [doris]
Mryange opened a new pull request, #63902:
URL: https://github.com/apache/doris/pull/63902
### What problem does this PR solve?
Problem Summary:
For TopN queries with computed order keys, such as `SELECT MURMUR_HASH3_32(number) AS n FROM numbers_20m ORDER BY n LIMIT 65535`, Doris created a TopN runtime predicate but could not push it down to storage because the target was an expression instead of a slot/key column. However, heap sort treated the existence of any runtime predicate as a reason to disable its local TopN pruning, so this path lost local pruning without gaining effective scan-side pruning.
This PR narrows that condition. Heap sort now disables local TopN pruning only when the runtime predicate targets are all slots. Expression order keys keep the local heap filter, which restores early row pruning in the sorter.
In local reproduction of `function_order_by_number_limit_big`, Doris improved from about 2.3s to 2.8s down to about 0.53s to 0.69s, and the new profile shows `TopNFilterRows: 19.555616M` instead of `0`.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](sort) keep heap TopN pruning for expression order keys [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #63902:
URL: https://github.com/apache/doris/pull/63902#issuecomment-4574155971
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](sort) keep heap TopN pruning for expression order keys [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63902:
URL: https://github.com/apache/doris/pull/63902#issuecomment-4574514458
<details>
<summary>TPC-H: <b>Total hot run time: 31470 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e46b6093e0a831ecb5f49400b3c98e86cc79ee3f, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17651 4039 4034 4034
q2 q3 10775 1369 824 824
q4 4699 473 356 356
q5 7825 2308 2110 2110
q6 380 177 140 140
q7 964 793 650 650
q8 9351 1777 1577 1577
q9 7045 4956 4977 4956
q10 6467 2255 1870 1870
q11 453 274 249 249
q12 703 432 314 314
q13 18195 3440 2819 2819
q14 268 264 234 234
q15 q16 824 776 715 715
q17 1006 943 986 943
q18 7019 5925 5603 5603
q19 1186 1302 1147 1147
q20 499 410 254 254
q21 5913 2641 2362 2362
q22 443 357 313 313
Total cold run time: 101666 ms
Total hot run time: 31470 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4541 4452 4378 4378
q2 q3 4554 4932 4360 4360
q4 2180 2219 1423 1423
q5 4506 4332 5079 4332
q6 258 199 152 152
q7 1997 1867 1611 1611
q8 2621 2223 2224 2223
q9 7971 7974 8101 7974
q10 4877 4757 4319 4319
q11 562 428 378 378
q12 759 771 550 550
q13 3378 3647 3008 3008
q14 302 311 282 282
q15 q16 704 725 667 667
q17 1370 1369 1346 1346
q18 7973 7396 6848 6848
q19 1112 1120 1134 1120
q20 2223 2230 1924 1924
q21 5365 4727 4559 4559
q22 512 466 427 427
Total cold run time: 57765 ms
Total hot run time: 51881 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](sort) keep heap TopN pruning for expression order keys [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange closed pull request #63902: [fix](sort) keep heap TopN pruning for expression order keys
URL: https://github.com/apache/doris/pull/63902
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](sort) keep heap TopN pruning for expression order keys [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63902:
URL: https://github.com/apache/doris/pull/63902#issuecomment-4573836100
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](sort) keep heap TopN pruning for expression order keys [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63902:
URL: https://github.com/apache/doris/pull/63902#issuecomment-4574633014
<details>
<summary>TPC-DS: <b>Total hot run time: 172273 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e46b6093e0a831ecb5f49400b3c98e86cc79ee3f, data reload: false
query5 4308 679 529 529
query6 349 223 198 198
query7 4295 616 294 294
query8 323 241 235 235
query9 8818 4156 4136 4136
query10 464 353 304 304
query11 5768 2543 2204 2204
query12 190 137 129 129
query13 1311 664 458 458
query14 6142 5551 5245 5245
query14_1 4594 4584 4511 4511
query15 232 213 188 188
query16 1009 443 476 443
query17 1180 767 621 621
query18 2454 499 379 379
query19 220 221 169 169
query20 141 139 133 133
query21 222 154 126 126
query22 13655 13551 13484 13484
query23 17430 16561 16190 16190
query23_1 16352 16397 16390 16390
query24 7453 1803 1325 1325
query24_1 1351 1351 1339 1339
query25 578 484 421 421
query26 1313 310 175 175
query27 2702 569 346 346
query28 4416 2063 2034 2034
query29 1004 622 510 510
query30 304 237 197 197
query31 1141 1086 962 962
query32 89 76 76 76
query33 569 361 303 303
query34 1184 1160 681 681
query35 784 806 707 707
query36 1433 1416 1295 1295
query37 158 113 96 96
query38 3245 3216 3076 3076
query39 923 941 926 926
query39_1 888 888 877 877
query40 236 146 128 128
query41 67 65 61 61
query42 112 112 114 112
query43 349 344 304 304
query44
query45 217 207 198 198
query46 1076 1200 759 759
query47 2404 2386 2287 2287
query48 399 415 292 292
query49 643 505 390 390
query50 993 354 258 258
query51 4337 4292 4273 4273
query52 114 112 126 112
query53 260 292 208 208
query54 323 282 261 261
query55 100 93 90 90
query56 304 313 310 310
query57 1451 1456 1314 1314
query58 309 278 267 267
query59 1649 1682 1459 1459
query60 330 329 316 316
query61 152 154 152 152
query62 706 646 600 600
query63 252 204 215 204
query64 2422 799 661 661
query65
query66 1711 500 361 361
query67 29742 29663 29521 29521
query68
query69 472 355 311 311
query70 1057 1033 995 995
query71 305 282 279 279
query72 2984 2724 2354 2354
query73 895 757 430 430
query74 5100 4955 4804 4804
query75 2701 2616 2272 2272
query76 2337 1165 872 872
query77 419 413 345 345
query78 12378 12397 11860 11860
query79 1503 1102 749 749
query80 642 548 454 454
query81 453 283 241 241
query82 1403 163 123 123
query83 360 278 257 257
query84 261 142 112 112
query85 901 541 463 463
query86 392 347 320 320
query87 3399 3404 3249 3249
query88 3721 2817 2803 2803
query89 453 395 345 345
query90 1920 190 188 188
query91 182 172 144 144
query92 82 80 77 77
query93 1514 1485 889 889
query94 551 352 329 329
query95 701 388 353 353
query96 1031 813 378 378
query97 2727 2762 2625 2625
query98 239 227 233 227
query99 1154 1165 1040 1040
Total cold run time: 254682 ms
Total hot run time: 172273 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](sort) keep heap TopN pruning for expression order keys [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63902:
URL: https://github.com/apache/doris/pull/63902#issuecomment-4576001392
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (13/13)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63902_e46b6093e0a831ecb5f49400b3c98e86cc79ee3f_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63902_e46b6093e0a831ecb5f49400b3c98e86cc79ee3f_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.81% (27361/38104) |
| Line Coverage | 55.54% (293793/528958) |
| Region Coverage | 52.34% (243593/465442) |
| Branch Coverage | 53.84% (105365/195711) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org