You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hust-hhb (via GitHub)" <gi...@apache.org> on 2024/02/07 04:20:17 UTC
[PR] [fix](group_commit)Add bounds checking when reading wal file on group commit [doris]
hust-hhb opened a new pull request, #30940:
URL: https://github.com/apache/doris/pull/30940
## Proposed changes
read wal file need a bound checking
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Add bounds checking when reading wal file on group commit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30940:
URL: https://github.com/apache/doris/pull/30940#issuecomment-1931282394
<details>
<summary>ClickBench: <b>Total hot run time: 30.19 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3f754f36b17314ef60ee6ff321ec0bafa0668e25, data reload: false
query1 0.04 0.03 0.03
query2 0.06 0.02 0.02
query3 0.23 0.06 0.06
query4 1.67 0.10 0.10
query5 0.53 0.52 0.54
query6 1.21 0.63 0.62
query7 0.01 0.01 0.01
query8 0.04 0.03 0.03
query9 0.56 0.49 0.51
query10 0.56 0.57 0.56
query11 0.12 0.09 0.08
query12 0.12 0.09 0.10
query13 0.61 0.61 0.60
query14 0.78 0.82 0.81
query15 0.78 0.78 0.78
query16 0.39 0.40 0.39
query17 1.03 1.02 1.04
query18 0.21 0.29 0.24
query19 1.82 1.83 1.75
query20 0.01 0.00 0.01
query21 15.41 0.59 0.56
query22 2.48 2.39 1.65
query23 17.12 0.85 0.78
query24 2.59 1.49 0.39
query25 0.31 0.16 0.12
query26 0.60 0.14 0.14
query27 0.05 0.04 0.05
query28 11.34 0.84 0.85
query29 12.52 3.16 3.10
query30 0.63 0.52 0.50
query31 2.78 0.34 0.36
query32 3.34 0.48 0.47
query33 3.23 3.21 3.20
query34 15.72 4.30 4.27
query35 4.29 4.26 4.32
query36 1.10 1.08 1.07
query37 0.07 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.02 0.01
query40 0.16 0.13 0.13
query41 0.07 0.02 0.01
query42 0.03 0.01 0.02
query43 0.02 0.02 0.02
Total cold run time: 104.7 s
Total hot run time: 30.19 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Add bounds checking when reading wal file on group commit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30940:
URL: https://github.com/apache/doris/pull/30940#issuecomment-1931269394
<details>
<summary>TPC-H: <b>Total hot run time: 37416 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f754f36b17314ef60ee6ff321ec0bafa0668e25, data reload: false
------ Round 1 ----------------------------------
q1 17643 4511 4468 4468
q2 2046 142 140 140
q3 10603 973 931 931
q4 4910 823 736 736
q5 8421 2931 2938 2931
q6 188 124 123 123
q7 1203 764 744 744
q8 9701 2062 2041 2041
q9 8056 6542 6411 6411
q10 8158 2436 2451 2436
q11 423 203 216 203
q12 810 280 287 280
q13 18006 3326 3319 3319
q14 283 250 255 250
q15 537 506 489 489
q16 473 409 419 409
q17 944 513 479 479
q18 6656 5948 5987 5948
q19 1562 1361 1417 1361
q20 566 327 328 327
q21 7082 3146 3077 3077
q22 815 319 313 313
Total cold run time: 109086 ms
Total hot run time: 37416 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4477 4408 4351 4351
q2 323 240 226 226
q3 2981 2870 2842 2842
q4 1883 1726 1628 1628
q5 5188 5253 5219 5219
q6 192 115 117 115
q7 2149 1785 1800 1785
q8 3116 3241 3239 3239
q9 8348 8264 8314 8264
q10 5852 3572 3607 3572
q11 543 461 458 458
q12 729 560 578 560
q13 7382 3100 3072 3072
q14 273 248 267 248
q15 534 492 480 480
q16 492 471 482 471
q17 1824 1700 1692 1692
q18 8024 7883 7501 7501
q19 8620 1504 1543 1504
q20 2108 1915 1895 1895
q21 4693 4550 4587 4550
q22 567 497 441 441
Total cold run time: 70298 ms
Total hot run time: 54113 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Add bounds checking when reading wal file on group commit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30940:
URL: https://github.com/apache/doris/pull/30940#issuecomment-1931261681
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Add bounds checking when reading wal file on group commit [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #30940:
URL: https://github.com/apache/doris/pull/30940#issuecomment-1931254453
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Add bounds checking when reading wal file on group commit [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #30940:
URL: https://github.com/apache/doris/pull/30940
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Add bounds checking when reading wal file on group commit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30940:
URL: https://github.com/apache/doris/pull/30940#issuecomment-1932038748
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Add bounds checking when reading wal file on group commit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30940:
URL: https://github.com/apache/doris/pull/30940#issuecomment-1931270799
TeamCity be ut coverage result:
Function Coverage: 36.17% (8638/23882)
Line Coverage: 28.19% (70623/250485)
Region Coverage: 27.18% (36435/134051)
Branch Coverage: 23.97% (18663/77856)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3f754f36b17314ef60ee6ff321ec0bafa0668e25_3f754f36b17314ef60ee6ff321ec0bafa0668e25/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Add bounds checking when reading wal file on group commit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30940:
URL: https://github.com/apache/doris/pull/30940#issuecomment-1932038796
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Add bounds checking when reading wal file on group commit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30940:
URL: https://github.com/apache/doris/pull/30940#issuecomment-1931278013
<details>
<summary>TPC-DS: <b>Total hot run time: 174757 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3f754f36b17314ef60ee6ff321ec0bafa0668e25, data reload: false
query1 929 351 333 333
query2 6547 2140 1842 1842
query3 6719 206 205 205
query4 31714 22050 21985 21985
query5 4266 357 365 357
query6 262 163 175 163
query7 4609 291 278 278
query8 249 172 186 172
query9 9085 2271 2244 2244
query10 425 216 209 209
query11 18649 15438 15348 15348
query12 127 80 76 76
query13 1620 409 418 409
query14 9577 7241 7196 7196
query15 276 184 179 179
query16 8135 259 252 252
query17 2058 529 500 500
query18 2116 270 267 267
query19 363 142 141 141
query20 86 78 74 74
query21 193 135 121 121
query22 4949 4723 4635 4635
query23 31382 30080 30059 30059
query24 10551 2750 2728 2728
query25 572 338 350 338
query26 1211 150 149 149
query27 2882 301 304 301
query28 7562 1838 1820 1820
query29 895 622 605 605
query30 277 133 146 133
query31 920 704 718 704
query32 89 53 52 52
query33 733 220 219 219
query34 1093 471 457 457
query35 848 775 749 749
query36 1085 917 886 886
query37 120 59 59 59
query38 3271 3137 3087 3087
query39 1293 1262 1256 1256
query40 182 94 93 93
query41 39 36 36 36
query42 99 92 94 92
query43 519 504 479 479
query44 1107 684 698 684
query45 193 183 179 179
query46 1071 644 676 644
query47 1571 1512 1550 1512
query48 449 355 347 347
query49 1086 286 284 284
query50 765 372 378 372
query51 5319 5197 5150 5150
query52 97 88 86 86
query53 345 266 271 266
query54 280 221 219 219
query55 80 75 73 73
query56 221 198 200 198
query57 981 897 915 897
query58 212 176 184 176
query59 2508 2434 2422 2422
query60 229 212 213 212
query61 85 91 82 82
query62 680 367 350 350
query63 297 271 272 271
query64 5181 3194 3256 3194
query65 3254 3252 3223 3223
query66 806 313 301 301
query67 14539 14049 14107 14049
query68 4103 558 538 538
query69 467 335 349 335
query70 1253 1222 1199 1199
query71 327 247 248 247
query72 5978 2813 2680 2680
query73 691 330 332 330
query74 6621 6212 6247 6212
query75 3039 2307 2347 2307
query76 2555 908 935 908
query77 339 230 222 222
query78 9157 8767 8929 8767
query79 2882 499 493 493
query80 2109 346 354 346
query81 555 194 197 194
query82 720 81 86 81
query83 251 127 129 127
query84 288 92 81 81
query85 2310 337 324 324
query86 472 306 285 285
query87 3360 3233 3236 3233
query88 3901 2379 2377 2377
query89 438 367 356 356
query90 2076 165 163 163
query91 153 120 124 120
query92 59 43 47 43
query93 3010 481 499 481
query94 1286 178 170 170
query95 8206 366 353 353
query96 596 277 278 277
query97 4191 4117 4125 4117
query98 222 194 195 194
query99 1175 718 729 718
Total cold run time: 292723 ms
Total hot run time: 174757 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Add bounds checking when reading wal file on group commit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30940:
URL: https://github.com/apache/doris/pull/30940#issuecomment-1931254398
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group_commit)Add bounds checking when reading wal file on group commit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30940:
URL: https://github.com/apache/doris/pull/30940#issuecomment-1931285177
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 3f754f36b17314ef60ee6ff321ec0bafa0668e25 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.6 seconds inserted 10000000 Rows, about 735K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org