You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HHoflittlefish777 (via GitHub)" <gi...@apache.org> on 2024/01/05 06:12:50 UTC
[PR] [improve](insert-into) record rows info in log for check [doris]
HHoflittlefish777 opened a new pull request, #29581:
URL: https://github.com/apache/doris/pull/29581
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](insert-into) record rows info in log for check [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29581:
URL: https://github.com/apache/doris/pull/29581#issuecomment-1878282310
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](insert-into) record rows info in log for check [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29581:
URL: https://github.com/apache/doris/pull/29581#issuecomment-1878177790
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](insert-into) record rows info in log for check [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29581:
URL: https://github.com/apache/doris/pull/29581#issuecomment-1878186996
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](insert-into) record rows info in log for check [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29581:
URL: https://github.com/apache/doris/pull/29581#issuecomment-1878192847
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](insert-into) record rows info in log for check [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29581:
URL: https://github.com/apache/doris/pull/29581#issuecomment-1878222685
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 2b776734d45d5d5f689e02f6f1bb48de81e66c28, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5511 5186 5088 5088
q2 391 172 158 158
q3 1474 1205 1182 1182
q4 1080 811 840 811
q5 3100 3103 3092 3092
q6 221 135 138 135
q7 997 571 512 512
q8 2153 2267 2281 2267
q9 6708 6654 6639 6639
q10 3207 3171 3117 3117
q11 347 227 216 216
q12 389 238 236 236
q13 4392 3663 3637 3637
q14 254 224 219 219
q15 596 545 546 545
q16 450 382 401 382
q17 1051 535 479 479
q18 7107 6709 6760 6709
q19 1630 1530 1586 1530
q20 610 353 331 331
q21 2892 2433 2499 2433
q22 383 337 351 337
Total cold run time: 44943 ms
Total hot run time: 40055 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5055 5104 5086 5086
q2 332 233 236 233
q3 3366 3351 3329 3329
q4 2152 2083 2088 2083
q5 5944 5966 5935 5935
q6 224 128 130 128
q7 2395 1925 1914 1914
q8 3589 3640 3694 3640
q9 9072 8964 8943 8943
q10 3875 3903 3878 3878
q11 585 469 491 469
q12 808 649 636 636
q13 3896 3178 3208 3178
q14 291 295 278 278
q15 580 534 532 532
q16 545 506 499 499
q17 2022 1793 1810 1793
q18 8773 8462 8341 8341
q19 1740 1659 1678 1659
q20 2269 1973 1983 1973
q21 5644 5408 5279 5279
q22 576 465 524 465
Total cold run time: 63733 ms
Total hot run time: 60271 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](insert-into) record rows info in log for check [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29581:
URL: https://github.com/apache/doris/pull/29581#issuecomment-1878282277
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](insert-into) record rows info in log for check [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #29581:
URL: https://github.com/apache/doris/pull/29581
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](insert-into) record rows info in log for check [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #29581:
URL: https://github.com/apache/doris/pull/29581#discussion_r1442533635
##########
be/src/runtime/tablets_channel.h:
##########
@@ -109,6 +109,10 @@
void refresh_profile();
+ size_t total_received_rows() const { return _total_received_rows; }
+
+ size_t num_rows_filtered() const { return _num_rows_filtered; }
Review Comment:
warning: method 'num_rows_filtered' can be made const [readability-make-member-function-const]
```suggestion
size_t num_rows_filtered() const { return _num_rows_filtered; }
```
##########
be/src/runtime/tablets_channel.h:
##########
@@ -109,6 +109,10 @@ class BaseTabletsChannel {
void refresh_profile();
+ size_t total_received_rows() const { return _total_received_rows; }
Review Comment:
warning: method 'total_received_rows' can be made const [readability-make-member-function-const]
```suggestion
size_t total_received_rows() const { return _total_received_rows; }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](insert-into) record rows info in log for check [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29581:
URL: https://github.com/apache/doris/pull/29581#issuecomment-1878212303
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 2b776734d45d5d5f689e02f6f1bb48de81e66c28, data reload: false
------ Round 1 ----------------------------------
q1 18977 5534 5156 5156
q2 2033 163 148 148
q3 10610 1095 1060 1060
q4 10206 802 836 802
q5 7791 2979 2877 2877
q6 226 132 131 131
q7 918 530 543 530
q8 9284 2049 2053 2049
q9 6898 6431 6439 6431
q10 8242 3046 3041 3041
q11 428 225 221 221
q12 395 237 236 236
q13 17995 3635 3628 3628
q14 245 229 208 208
q15 564 530 519 519
q16 451 400 389 389
q17 981 527 498 498
q18 7389 6784 6697 6697
q19 1601 1356 1394 1356
q20 689 329 350 329
q21 2837 2484 2420 2420
q22 389 336 339 336
Total cold run time: 109149 ms
Total hot run time: 39062 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5229 5098 5085 5085
q2 336 231 242 231
q3 3330 3303 3302 3302
q4 2123 2074 2061 2061
q5 5854 5804 5829 5804
q6 215 121 127 121
q7 2366 1906 1907 1906
q8 3423 3463 3487 3463
q9 8816 8795 8725 8725
q10 3794 3830 3833 3830
q11 588 474 469 469
q12 821 624 631 624
q13 8888 3249 3177 3177
q14 276 278 273 273
q15 580 518 516 516
q16 574 511 487 487
q17 1958 1782 1787 1782
q18 8707 8410 8440 8410
q19 1640 1614 1565 1565
q20 2214 1936 1954 1936
q21 5622 5386 5351 5351
q22 556 483 504 483
Total cold run time: 67910 ms
Total hot run time: 59601 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](insert-into) record rows info in log for check [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29581:
URL: https://github.com/apache/doris/pull/29581#issuecomment-1878206706
TeamCity be ut coverage result:
Function Coverage: 36.63% (8617/23524)
Line Coverage: 28.67% (70031/244290)
Region Coverage: 27.64% (36239/131122)
Branch Coverage: 24.34% (18515/76078)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2b776734d45d5d5f689e02f6f1bb48de81e66c28_2b776734d45d5d5f689e02f6f1bb48de81e66c28/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](insert-into) record rows info in log for check [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29581:
URL: https://github.com/apache/doris/pull/29581#issuecomment-1878228346
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49.43 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.1 seconds inserted 10000000 Rows, about 369K ops/s
storage size: 17188249818 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](insert-into) record rows info in log for check [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29581:
URL: https://github.com/apache/doris/pull/29581#issuecomment-1878222227
TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools
```
TPC-DS sf100 test result on commit 2b776734d45d5d5f689e02f6f1bb48de81e66c28, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 912 350 339 339
query2 5827 1990 1938 1938
query3 6149 208 202 202
query4 25855 22351 22345 22345
query5 4057 561 519 519
query6 257 176 180 176
query7 4439 266 263 263
query8 221 200 197 197
query9 8115 2587 2585 2585
query10 441 243 238 238
query11 16181 15516 15361 15361
query12 129 77 74 74
query13 1622 315 315 315
query14 11399 7034 6994 6994
query15 235 193 194 193
query16 6402 271 266 266
query17 1814 494 477 477
query18 1923 266 250 250
query19 193 134 131 131
query20 79 73 75 73
query21 183 97 95 95
query22 5066 4711 4531 4531
query23 32099 31510 31240 31240
query24 11920 2813 2801 2801
query25 576 340 332 332
query26 1686 142 139 139
query27 2866 272 275 272
query28 6774 1877 1875 1875
query29 1885 389 381 381
query30 281 143 149 143
query31 969 744 791 744
query32 82 56 54 54
query33 716 261 261 261
query34 902 437 439 437
query35 871 748 774 748
query36 1348 1208 1259 1208
query37 171 70 80 70
query38 3432 3312 3228 3228
query39 1323 1288 1276 1276
query40 285 87 86 86
query41 38 34 36 34
query42 83 88 78 78
query43 527 503 497 497
query44 1046 692 712 692
query45 191 182 177 177
query46 1085 641 619 619
query47 1603 1547 1506 1506
query48 323 251 253 251
query49 1137 304 321 304
query50 744 349 357 349
query51 5392 5273 5304 5273
query52 97 87 87 87
query53 208 143 145 143
query54 1215 571 551 551
query55 96 82 80 80
query56 186 186 188 186
query57 1039 878 965 878
query58 208 190 198 190
query59 2935 2591 2691 2591
query60 221 222 218 218
query61 84 80 80 80
query62 652 446 466 446
query63 168 152 148 148
query64 5663 1693 1651 1651
query65 3327 3251 3261 3251
query66 1361 344 340 340
query67 15739 15329 15313 15313
query68 11381 514 510 510
query69 521 252 252 252
query70 1766 1447 1545 1447
query71 491 228 208 208
query72 5838 3513 3529 3513
query73 2434 316 314 314
query74 7038 6574 6453 6453
query75 5009 2329 2284 2284
query76 6334 1105 1093 1093
query77 675 252 246 246
query78 9094 8622 8567 8567
query79 1016 530 487 487
query80 547 354 338 338
query81 460 207 205 205
query82 203 93 97 93
query83 160 138 136 136
query84 246 54 53 53
query85 921 264 256 256
query86 396 417 382 382
query87 3603 3376 3372 3372
query88 2894 2263 2272 2263
query89 324 258 257 257
query90 1902 199 199 199
query91 118 87 88 87
query92 58 52 52 52
query93 1328 490 481 481
query94 805 185 180 180
query95 458 414 411 411
query96 627 322 317 317
query97 4265 4187 4154 4154
query98 209 211 198 198
query99 1146 848 796 796
Total cold run time: 287320 ms
Total hot run time: 178394 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](insert-into) record rows info in log for check [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #29581:
URL: https://github.com/apache/doris/pull/29581#discussion_r1442530182
##########
be/src/runtime/tablets_channel.h:
##########
@@ -109,6 +109,10 @@ class BaseTabletsChannel {
void refresh_profile();
+ size_t total_received_rows() { return _total_received_rows; }
+
+ size_t num_rows_filtered() { return _num_rows_filtered; }
Review Comment:
warning: method 'num_rows_filtered' can be made const [readability-make-member-function-const]
```suggestion
size_t num_rows_filtered() const { return _num_rows_filtered; }
```
##########
be/src/runtime/tablets_channel.h:
##########
@@ -109,6 +109,10 @@ class BaseTabletsChannel {
void refresh_profile();
+ size_t total_received_rows() { return _total_received_rows; }
Review Comment:
warning: method 'total_received_rows' can be made const [readability-make-member-function-const]
```suggestion
size_t total_received_rows() const { return _total_received_rows; }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org