You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2024/06/13 07:47:05 UTC
[PR] [opt](nereids) compute tuple size by output slots [doris]
englefly opened a new pull request, #36246:
URL: https://github.com/apache/doris/pull/36246
## Proposed changes
in previous version, nereids compute output tuple size of an operator by sum all avgSize of column stats in stats.expressionToColumnStats. This is not accurate, because we may put some intermediate expression's column stats into stats.expressionToColumnStats, and hence the tuple size may becomes larger.
Issue Number: close #xxx
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) compute tuple size by output slots [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #36246:
URL: https://github.com/apache/doris/pull/36246#issuecomment-2164880084
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) compute tuple size by output slots [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #36246:
URL: https://github.com/apache/doris/pull/36246#issuecomment-2164899204
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) compute tuple size by output slots [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36246:
URL: https://github.com/apache/doris/pull/36246#issuecomment-2165006609
<details>
<summary>ClickBench: <b>Total hot run time: 30.21 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2d39c822097e4d38769ce8fdbd99979256fbc70b, data reload: false
query1 0.04 0.04 0.03
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.68 0.09 0.09
query5 0.48 0.50 0.50
query6 1.12 0.72 0.73
query7 0.02 0.02 0.01
query8 0.05 0.04 0.05
query9 0.54 0.49 0.49
query10 0.55 0.56 0.54
query11 0.15 0.11 0.11
query12 0.14 0.12 0.12
query13 0.59 0.59 0.59
query14 0.81 0.82 0.78
query15 0.83 0.82 0.81
query16 0.34 0.37 0.37
query17 1.00 0.99 1.02
query18 0.24 0.24 0.27
query19 1.80 1.66 1.69
query20 0.02 0.01 0.02
query21 15.43 0.66 0.65
query22 4.28 8.24 1.63
query23 18.28 1.34 1.37
query24 2.13 0.21 0.23
query25 0.15 0.08 0.07
query26 0.27 0.17 0.17
query27 0.08 0.07 0.08
query28 13.17 1.01 0.99
query29 12.68 3.31 3.22
query30 0.27 0.06 0.05
query31 2.87 0.39 0.38
query32 3.28 0.48 0.46
query33 2.93 2.89 2.87
query34 17.16 4.42 4.38
query35 4.56 4.46 4.46
query36 0.65 0.46 0.47
query37 0.19 0.15 0.15
query38 0.15 0.14 0.14
query39 0.05 0.04 0.03
query40 0.17 0.15 0.15
query41 0.09 0.05 0.05
query42 0.05 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 109.64 s
Total hot run time: 30.21 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) compute tuple size by output slots [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36246:
URL: https://github.com/apache/doris/pull/36246#issuecomment-2164966480
<details>
<summary>TPC-H: <b>Total hot run time: 40075 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2d39c822097e4d38769ce8fdbd99979256fbc70b, data reload: false
------ Round 1 ----------------------------------
q1 17961 4543 4437 4437
q2 2669 205 198 198
q3 11416 1091 1107 1091
q4 10490 860 795 795
q5 8451 2760 2687 2687
q6 226 140 141 140
q7 969 625 596 596
q8 9372 2048 2083 2048
q9 8947 6436 6477 6436
q10 8893 3719 3734 3719
q11 478 235 239 235
q12 404 234 232 232
q13 18451 2972 2985 2972
q14 256 218 215 215
q15 513 477 469 469
q16 529 407 379 379
q17 982 786 732 732
q18 8000 7559 7372 7372
q19 6226 1488 1570 1488
q20 667 324 321 321
q21 4991 3186 3281 3186
q22 386 327 327 327
Total cold run time: 121277 ms
Total hot run time: 40075 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4403 4258 4275 4258
q2 366 263 272 263
q3 2931 2722 2684 2684
q4 1869 1573 1649 1573
q5 5242 5233 5267 5233
q6 213 127 131 127
q7 2121 1757 1749 1749
q8 3166 3334 3288 3288
q9 8262 8300 8339 8300
q10 3871 3601 3688 3601
q11 582 482 501 482
q12 791 605 567 567
q13 16747 2986 2938 2938
q14 293 255 251 251
q15 516 472 471 471
q16 467 404 410 404
q17 1769 1462 1454 1454
q18 7533 7463 7394 7394
q19 1698 1643 1437 1437
q20 1973 1795 1760 1760
q21 4803 4796 4616 4616
q22 589 546 539 539
Total cold run time: 70205 ms
Total hot run time: 53389 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) compute tuple size by output slots [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36246:
URL: https://github.com/apache/doris/pull/36246#issuecomment-2173130456
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) compute tuple size by output slots [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36246:
URL: https://github.com/apache/doris/pull/36246#issuecomment-2164878963
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) compute tuple size by output slots [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly merged PR #36246:
URL: https://github.com/apache/doris/pull/36246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) compute tuple size by output slots [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36246:
URL: https://github.com/apache/doris/pull/36246#issuecomment-2173130521
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) compute tuple size by output slots [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36246:
URL: https://github.com/apache/doris/pull/36246#issuecomment-2164994505
<details>
<summary>TPC-DS: <b>Total hot run time: 172382 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2d39c822097e4d38769ce8fdbd99979256fbc70b, data reload: false
query1 928 377 370 370
query2 6459 2388 2151 2151
query3 6654 205 207 205
query4 19691 17248 17194 17194
query5 4135 473 466 466
query6 251 156 167 156
query7 4593 305 294 294
query8 323 286 283 283
query9 8487 2482 2465 2465
query10 596 317 293 293
query11 10625 10012 10056 10012
query12 146 96 88 88
query13 1664 367 384 367
query14 10122 7067 7658 7067
query15 239 192 190 190
query16 7782 266 261 261
query17 1917 528 563 528
query18 1859 279 285 279
query19 193 158 153 153
query20 93 82 81 81
query21 215 128 124 124
query22 4305 4009 4008 4008
query23 33610 33041 33155 33041
query24 12021 2817 2875 2817
query25 659 357 358 357
query26 1790 156 154 154
query27 3000 324 327 324
query28 7445 2099 2070 2070
query29 1129 614 605 605
query30 268 152 147 147
query31 962 736 738 736
query32 93 55 56 55
query33 772 307 340 307
query34 917 467 468 467
query35 732 598 622 598
query36 1094 902 945 902
query37 285 74 69 69
query38 2841 2689 2700 2689
query39 854 813 801 801
query40 281 125 126 125
query41 56 51 51 51
query42 120 99 104 99
query43 624 567 536 536
query44 1250 732 743 732
query45 192 162 167 162
query46 1081 714 709 709
query47 1843 1756 1771 1756
query48 392 293 296 293
query49 1187 407 407 407
query50 774 395 397 395
query51 6762 6656 6604 6604
query52 101 90 95 90
query53 360 287 290 287
query54 1038 477 444 444
query55 74 76 73 73
query56 285 263 259 259
query57 1144 1009 1027 1009
query58 259 241 256 241
query59 3510 3088 3163 3088
query60 303 294 276 276
query61 94 88 91 88
query62 676 436 439 436
query63 319 278 287 278
query64 9882 2243 1745 1745
query65 3232 3165 3089 3089
query66 1376 339 324 324
query67 15357 14954 15026 14954
query68 4597 548 556 548
query69 487 316 322 316
query70 1170 1177 1142 1142
query71 406 284 278 278
query72 7214 5255 5542 5255
query73 753 379 330 330
query74 6007 5460 5511 5460
query75 3479 2664 2643 2643
query76 2730 982 935 935
query77 505 322 315 315
query78 10322 9795 9802 9795
query79 2553 521 513 513
query80 1161 473 469 469
query81 584 222 218 218
query82 770 107 102 102
query83 237 170 171 170
query84 244 87 86 86
query85 1605 288 275 275
query86 478 315 284 284
query87 3265 3031 3101 3031
query88 3939 2459 2450 2450
query89 474 373 393 373
query90 1780 193 196 193
query91 203 100 99 99
query92 63 51 49 49
query93 1765 512 510 510
query94 1182 189 189 189
query95 432 325 323 323
query96 594 274 275 274
query97 3198 3102 2999 2999
query98 212 201 194 194
query99 1194 859 840 840
Total cold run time: 275924 ms
Total hot run time: 172382 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org