You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morrySnow (via GitHub)" <gi...@apache.org> on 2023/12/29 09:49:53 UTC
[PR] [fix](Nereids) topn runtime filter only support simplest case [doris]
morrySnow opened a new pull request, #29312:
URL: https://github.com/apache/doris/pull/29312
only support simple case: select ... from tbl [where ...] order by ... limit ...
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) topn runtime filter only support simplest case [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29312:
URL: https://github.com/apache/doris/pull/29312#issuecomment-1871930898
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 70559108d00cbbc908ca48d53b6019c6d83db45c, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5456 5139 5126 5126
q2 393 186 168 168
q3 1478 1201 1179 1179
q4 1103 821 749 749
q5 3172 3180 3109 3109
q6 232 139 140 139
q7 989 592 532 532
q8 2165 2282 2230 2230
q9 6717 6685 6677 6677
q10 3203 3154 3129 3129
q11 354 237 233 233
q12 395 247 248 247
q13 4455 3682 3668 3668
q14 264 226 232 226
q15 643 580 559 559
q16 459 409 425 409
q17 1043 589 533 533
q18 7132 6823 6857 6823
q19 1646 1559 1494 1494
q20 626 360 378 360
q21 2881 2473 2544 2473
q22 405 332 336 332
Total cold run time: 45211 ms
Total hot run time: 40395 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5153 5142 5114 5114
q2 336 257 235 235
q3 3399 3341 3327 3327
q4 2163 2033 2052 2033
q5 6006 5942 5936 5936
q6 239 139 135 135
q7 2420 1951 1942 1942
q8 3567 3684 3689 3684
q9 9094 9039 9026 9026
q10 3886 3907 3948 3907
q11 597 486 496 486
q12 829 652 632 632
q13 3882 3225 3203 3203
q14 296 276 290 276
q15 636 562 577 562
q16 546 546 527 527
q17 2063 1835 1830 1830
q18 8874 8375 8426 8375
q19 1752 1698 1687 1687
q20 2309 2000 2017 2000
q21 5763 5475 5494 5475
q22 586 513 502 502
Total cold run time: 64396 ms
Total hot run time: 60894 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) topn runtime filter only support simplest case [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29312:
URL: https://github.com/apache/doris/pull/29312#issuecomment-1871927322
TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools
```
TPC-DS sf100 test result on commit 70559108d00cbbc908ca48d53b6019c6d83db45c, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 932 364 344 344
query2 4091 1961 2104 1961
query3 6577 232 223 223
query4 27171 22477 22452 22452
query5 4912 558 569 558
query6 298 179 184 179
query7 4584 279 265 265
query8 227 207 214 207
query9 8146 2652 2691 2652
query10 372 247 266 247
query11 16362 15738 15507 15507
query12 139 82 80 80
query13 1638 327 327 327
query14 10725 7431 7404 7404
query15 227 194 194 194
query16 5519 308 302 302
query17 1654 512 499 499
query18 1915 278 270 270
query19 270 145 141 141
query20 87 82 79 79
query21 187 102 96 96
query22 4836 4830 4814 4814
query23 32316 31398 31308 31308
query24 12036 2831 2814 2814
query25 588 349 352 349
query26 1467 151 151 151
query27 2829 284 284 284
query28 7147 1984 1973 1973
query29 1727 415 397 397
query30 287 141 145 141
query31 951 787 795 787
query32 93 65 62 62
query33 725 282 283 282
query34 858 463 449 449
query35 873 774 755 755
query36 1377 1236 1269 1236
query37 192 87 90 87
query38 3365 3284 3237 3237
query39 1313 1305 1283 1283
query40 304 94 94 94
query41 38 36 34 34
query42 99 94 97 94
query43 528 534 512 512
query44 1106 767 788 767
query45 209 196 188 188
query46 1084 655 665 655
query47 1685 1673 1592 1592
query48 337 270 263 263
query49 1209 340 335 335
query50 764 381 344 344
query51 5410 5257 5372 5257
query52 99 96 86 86
query53 217 159 146 146
query54 1488 602 638 602
query55 105 89 89 89
query56 221 203 206 203
query57 1056 976 951 951
query58 238 214 204 204
query59 2925 2663 2680 2663
query60 263 229 232 229
query61 82 82 82 82
query62 654 467 480 467
query63 166 164 162 162
query64 5265 1722 1710 1710
query65 3352 3282 3266 3266
query66 1307 330 333 330
query67 15331 15008 15235 15008
query68 10961 540 512 512
query69 615 277 261 261
query70 1703 1534 1570 1534
query71 565 236 233 233
query72 4846 3541 3505 3505
query73 1436 322 317 317
query74 7018 6526 6472 6472
query75 5590 2283 2273 2273
query76 5338 1125 1098 1098
query77 879 292 291 291
query78 9133 8627 8588 8588
query79 4725 507 519 507
query80 2851 380 373 373
query81 515 210 212 210
query82 315 128 126 126
query83 315 137 136 136
query84 255 53 60 53
query85 2530 280 275 275
query86 404 385 402 385
query87 3591 3377 3352 3352
query88 3268 2451 2447 2447
query89 344 277 273 273
query90 1942 248 242 242
query91 119 92 91 91
query92 72 57 54 54
query93 2765 500 430 430
query94 896 219 212 212
query95 520 475 473 473
query96 655 341 341 341
query97 4287 4180 4175 4175
query98 213 195 193 193
query99 1134 837 840 837
Total cold run time: 294355 ms
Total hot run time: 180998 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) topn runtime filter only support simplest case [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #29312:
URL: https://github.com/apache/doris/pull/29312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) topn runtime filter only support simplest case [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29312:
URL: https://github.com/apache/doris/pull/29312#issuecomment-1871927250
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.39 seconds
stream load tsv: 574 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17184200364 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) topn runtime filter only support simplest case [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29312:
URL: https://github.com/apache/doris/pull/29312#issuecomment-1874154897
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) topn runtime filter only support simplest case [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29312:
URL: https://github.com/apache/doris/pull/29312#issuecomment-1871912280
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 70559108d00cbbc908ca48d53b6019c6d83db45c, data reload: false
------ Round 1 ----------------------------------
q1 17629 5286 5321 5286
q2 2024 162 153 153
q3 10573 1150 1317 1150
q4 10189 900 945 900
q5 7807 3142 3049 3049
q6 234 142 140 140
q7 983 532 544 532
q8 9358 2236 2232 2232
q9 7057 6713 6646 6646
q10 8352 3122 3182 3122
q11 450 240 210 210
q12 436 245 243 243
q13 17993 3603 3629 3603
q14 256 216 212 212
q15 597 571 547 547
q16 462 414 426 414
q17 1018 603 529 529
q18 7590 6777 6760 6760
q19 1784 1580 1522 1522
q20 714 359 391 359
q21 2988 2534 2551 2534
q22 370 330 327 327
Total cold run time: 108864 ms
Total hot run time: 40470 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5264 5251 5204 5204
q2 357 257 279 257
q3 3369 3360 3350 3350
q4 2245 2093 2080 2080
q5 5958 5968 5951 5951
q6 244 136 135 135
q7 2387 1909 1930 1909
q8 3577 3655 3691 3655
q9 9010 8955 8906 8906
q10 3893 3950 3951 3950
q11 612 528 498 498
q12 836 688 657 657
q13 9506 3183 3211 3183
q14 293 283 282 282
q15 609 553 558 553
q16 578 524 525 524
q17 2061 1830 1824 1824
q18 8952 8501 8465 8465
q19 1832 1764 1785 1764
q20 2257 1973 1976 1973
q21 5895 5405 5365 5365
q22 538 491 463 463
Total cold run time: 70273 ms
Total hot run time: 60948 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) topn runtime filter only support simplest case [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #29312:
URL: https://github.com/apache/doris/pull/29312#issuecomment-1871888212
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) topn runtime filter only support simplest case [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29312:
URL: https://github.com/apache/doris/pull/29312#issuecomment-1872740324
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org