You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2025/04/25 09:34:37 UTC
[PR] [opt](nereids) set max row count for building runtime filter [doris]
englefly opened a new pull request, #50422:
URL: https://github.com/apache/doris/pull/50422
### What problem does this PR solve?
When generating runtime filters, restrict the row count on the build side. Excessive rows not only incur significant time overhead during filter generation but also degrade the filtering rate (selectivity) of the Bloom filter.
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) set max row count for building runtime filter [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50422:
URL: https://github.com/apache/doris/pull/50422#issuecomment-2829894114
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) set max row count for building runtime filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50422:
URL: https://github.com/apache/doris/pull/50422#issuecomment-2830002347
<details>
<summary>TPC-DS: <b>Total hot run time: 192644 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 39d0ba5a7d3ce4e994c26ec42d1eed9950d15d00, data reload: false
query1 1404 1105 1042 1042
query2 6146 1860 1907 1860
query3 11059 4557 4416 4416
query4 52773 25832 22993 22993
query5 5203 552 437 437
query6 339 195 193 193
query7 4951 486 283 283
query8 315 246 234 234
query9 5787 2561 2552 2552
query10 454 305 279 279
query11 15042 15080 14878 14878
query12 159 107 104 104
query13 1079 510 406 406
query14 10227 6293 6343 6293
query15 205 185 179 179
query16 7157 679 466 466
query17 1083 734 605 605
query18 1570 407 334 334
query19 189 189 162 162
query20 137 126 122 122
query21 213 129 117 117
query22 4422 4560 4554 4554
query23 34107 33585 33552 33552
query24 6756 2445 2395 2395
query25 459 482 420 420
query26 696 292 160 160
query27 2161 513 340 340
query28 3312 2168 2183 2168
query29 601 580 458 458
query30 272 219 201 201
query31 853 854 797 797
query32 77 63 65 63
query33 472 380 323 323
query34 751 871 524 524
query35 779 868 755 755
query36 935 1026 913 913
query37 120 107 87 87
query38 4178 4237 4260 4237
query39 1478 1421 1405 1405
query40 212 117 113 113
query41 54 53 53 53
query42 126 108 111 108
query43 511 512 471 471
query44 1302 825 836 825
query45 181 175 173 173
query46 865 1045 645 645
query47 1838 1852 1844 1844
query48 397 421 327 327
query49 723 513 408 408
query50 657 697 398 398
query51 4220 4253 4201 4201
query52 114 111 98 98
query53 230 253 182 182
query54 595 578 502 502
query55 80 89 79 79
query56 324 287 288 287
query57 1176 1203 1087 1087
query58 254 243 255 243
query59 2728 2843 2694 2694
query60 310 328 297 297
query61 132 130 128 128
query62 703 706 657 657
query63 221 179 180 179
query64 1732 1009 672 672
query65 4248 4233 4262 4233
query66 847 401 303 303
query67 15875 15501 15501 15501
query68 7156 894 525 525
query69 528 301 261 261
query70 1224 1141 1110 1110
query71 499 315 296 296
query72 5696 4775 4717 4717
query73 1394 618 354 354
query74 8950 9095 8953 8953
query75 3903 3218 2673 2673
query76 4305 1195 742 742
query77 680 365 314 314
query78 10005 10025 9275 9275
query79 4635 815 547 547
query80 633 509 435 435
query81 476 253 226 226
query82 513 126 93 93
query83 321 251 234 234
query84 293 153 84 84
query85 789 343 308 308
query86 334 290 265 265
query87 4389 4401 4305 4305
query88 3137 2229 2200 2200
query89 434 328 290 290
query90 1936 205 207 205
query91 140 141 111 111
query92 76 56 58 56
query93 2340 913 573 573
query94 668 413 318 318
query95 371 288 284 284
query96 485 576 271 271
query97 3233 3240 3106 3106
query98 230 211 199 199
query99 1426 1406 1299 1299
Total cold run time: 299775 ms
Total hot run time: 192644 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) set max row count for building runtime filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50422:
URL: https://github.com/apache/doris/pull/50422#issuecomment-2829969856
<details>
<summary>TPC-H: <b>Total hot run time: 33903 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 39d0ba5a7d3ce4e994c26ec42d1eed9950d15d00, data reload: false
------ Round 1 ----------------------------------
q1 25858 5030 5035 5030
q2 2071 269 183 183
q3 10398 1247 709 709
q4 10222 1003 526 526
q5 7523 2298 2394 2298
q6 178 164 131 131
q7 905 753 604 604
q8 9296 1384 1099 1099
q9 6651 5042 5047 5042
q10 6790 2331 1923 1923
q11 477 274 269 269
q12 343 353 214 214
q13 17757 3634 3078 3078
q14 241 220 207 207
q15 535 494 499 494
q16 441 441 401 401
q17 588 860 369 369
q18 7441 7158 7201 7158
q19 1212 931 555 555
q20 333 325 221 221
q21 4048 2633 2421 2421
q22 1060 1010 971 971
Total cold run time: 114368 ms
Total hot run time: 33903 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5055 5093 5069 5069
q2 240 331 234 234
q3 2128 2631 2281 2281
q4 1374 1780 1374 1374
q5 4389 4396 4464 4396
q6 219 166 126 126
q7 2030 1930 1754 1754
q8 2586 2560 2500 2500
q9 7278 7269 7094 7094
q10 2985 3186 2703 2703
q11 575 498 489 489
q12 666 742 599 599
q13 3468 3868 3270 3270
q14 289 317 279 279
q15 527 478 482 478
q16 464 511 484 484
q17 1147 1568 1371 1371
q18 7763 7622 7554 7554
q19 805 792 944 792
q20 1917 1916 1801 1801
q21 5327 4826 4795 4795
q22 1090 1061 1030 1030
Total cold run time: 52322 ms
Total hot run time: 50473 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) set max row count for building runtime filter [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly closed pull request #50422: [opt](nereids) set max row count for building runtime filter
URL: https://github.com/apache/doris/pull/50422
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) set max row count for building runtime filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50422:
URL: https://github.com/apache/doris/pull/50422#issuecomment-2830016824
<details>
<summary>ClickBench: <b>Total hot run time: 29.49 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 39d0ba5a7d3ce4e994c26ec42d1eed9950d15d00, data reload: false
query1 0.03 0.03 0.03
query2 0.13 0.10 0.11
query3 0.26 0.18 0.19
query4 1.59 0.20 0.20
query5 0.57 0.57 0.60
query6 1.18 0.71 0.71
query7 0.03 0.02 0.02
query8 0.04 0.03 0.04
query9 0.55 0.51 0.52
query10 0.57 0.58 0.57
query11 0.15 0.11 0.11
query12 0.14 0.11 0.12
query13 0.61 0.59 0.60
query14 1.16 1.21 1.21
query15 0.86 0.84 0.84
query16 0.39 0.37 0.38
query17 1.01 1.02 1.02
query18 0.21 0.20 0.19
query19 1.85 1.76 1.79
query20 0.02 0.01 0.01
query21 15.40 0.90 0.53
query22 0.75 1.26 0.64
query23 14.90 1.42 0.67
query24 7.24 1.36 0.74
query25 0.50 0.28 0.13
query26 0.59 0.16 0.15
query27 0.05 0.06 0.05
query28 9.86 0.91 0.44
query29 12.54 4.04 3.31
query30 0.25 0.09 0.07
query31 2.81 0.60 0.38
query32 3.23 0.55 0.47
query33 3.09 2.99 3.03
query34 15.68 5.12 4.50
query35 4.49 4.48 4.53
query36 0.67 0.49 0.48
query37 0.08 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.03
query40 0.18 0.14 0.14
query41 0.08 0.03 0.02
query42 0.03 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 103.88 s
Total hot run time: 29.49 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) set max row count for building runtime filter [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #50422:
URL: https://github.com/apache/doris/pull/50422#issuecomment-2829894304
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org