You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Yukang-Lian (via GitHub)" <gi...@apache.org> on 2024/02/06 04:50:02 UTC
[PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Yukang-Lian opened a new pull request, #30893:
URL: https://github.com/apache/doris/pull/30893
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #30893:
URL: https://github.com/apache/doris/pull/30893#discussion_r1482398656
##########
be/src/olap/wal/wal_dirs_info.cpp:
##########
@@ -50,20 +50,21 @@ void WalDirInfo::set_used(size_t used) {
_used = used;
}
-size_t WalDirInfo::get_pre_allocated() {
+size_t WalDirInfo::get_estimated_wal_bytes() {
Review Comment:
warning: method 'get_estimated_wal_bytes' can be made const [readability-make-member-function-const]
be/src/olap/wal/wal_dirs_info.h:43:
```diff
- size_t get_estimated_wal_bytes();
+ size_t get_estimated_wal_bytes() const;
```
```suggestion
size_t WalDirInfo::get_estimated_wal_bytes() const {
```
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -538,7 +534,7 @@ Status LoadBlockQueue::close_wal() {
return Status::OK();
}
-bool LoadBlockQueue::has_enough_wal_disk_space(size_t pre_allocated) {
+bool LoadBlockQueue::has_enough_wal_disk_space(size_t estimated_wal_bytes) {
Review Comment:
warning: method 'has_enough_wal_disk_space' can be made static [readability-convert-member-functions-to-static]
be/src/runtime/group_commit_mgr.h:73:
```diff
- bool has_enough_wal_disk_space(size_t estimated_wal_bytes);
+ static bool has_enough_wal_disk_space(size_t estimated_wal_bytes);
```
##########
be/test/olap/wal/wal_dirs_info_test.cpp:
##########
@@ -57,31 +57,32 @@ class WalDirsInfoTest : public testing::Test {
EXPECT_EQ(st, Status::OK());
EXPECT_EQ(wal_dir_info->get_used(), used);
- st = wal_dirs_info.update_wal_dir_pre_allocated(wal_dir, pre_allocated, 0);
+ st = wal_dirs_info.update_wal_dir_estimated_wal_bytes(wal_dir, estimated_wal_bytes, 0);
EXPECT_EQ(st, Status::OK());
st = wal_dirs_info.get_wal_dir_info(wal_dir, wal_dir_info);
EXPECT_NE(wal_dir_info, nullptr);
EXPECT_EQ(st, Status::OK());
- EXPECT_EQ(wal_dir_info->get_pre_allocated(), pre_allocated);
+ EXPECT_EQ(wal_dir_info->get_estimated_wal_bytes(), estimated_wal_bytes);
- st = wal_dirs_info.update_wal_dir_pre_allocated(wal_dir, 0, pre_allocated);
+ st = wal_dirs_info.update_wal_dir_estimated_wal_bytes(wal_dir, 0, estimated_wal_bytes);
EXPECT_EQ(st, Status::OK());
st = wal_dirs_info.get_wal_dir_info(wal_dir, wal_dir_info);
EXPECT_NE(wal_dir_info, nullptr);
EXPECT_EQ(st, Status::OK());
- EXPECT_EQ(wal_dir_info->get_pre_allocated(), 0);
+ EXPECT_EQ(wal_dir_info->get_estimated_wal_bytes(), 0);
- st = wal_dirs_info.update_wal_dir_pre_allocated(wal_dir, pre_allocated, 0);
+ st = wal_dirs_info.update_wal_dir_estimated_wal_bytes(wal_dir, estimated_wal_bytes, 0);
EXPECT_EQ(st, Status::OK());
st = wal_dirs_info.get_wal_dir_info(wal_dir, wal_dir_info);
EXPECT_NE(wal_dir_info, nullptr);
EXPECT_EQ(st, Status::OK());
- EXPECT_EQ(wal_dir_info->get_pre_allocated(), pre_allocated);
+ EXPECT_EQ(wal_dir_info->get_estimated_wal_bytes(), estimated_wal_bytes);
- EXPECT_EQ(wal_dir_info->available(), limit - used - pre_allocated);
+ EXPECT_EQ(wal_dir_info->available(), limit - used - estimated_wal_bytes);
}
- void set_and_check_fail(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated) {
+ void set_and_check_fail(std::string wal_dir, size_t limit, size_t used,
Review Comment:
warning: method 'set_and_check_fail' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void set_and_check_fail(std::string wal_dir, size_t limit, size_t used,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1933785024
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1933838473
TeamCity be ut coverage result:
Function Coverage: 35.83% (8553/23874)
Line Coverage: 27.77% (69442/250057)
Region Coverage: 26.91% (36030/133900)
Branch Coverage: 23.69% (18422/77762)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3598d0d2e1b7ee9b5880a6218301f78d5244228b_3598d0d2e1b7ee9b5880a6218301f78d5244228b/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1929061210
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1929119601
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1928780249
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1933873433
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 3598d0d2e1b7ee9b5880a6218301f78d5244228b with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.6 seconds inserted 10000000 Rows, about 735K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1933839801
<details>
<summary>TPC-H: <b>Total hot run time: 41414 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3598d0d2e1b7ee9b5880a6218301f78d5244228b, data reload: false
------ Round 1 ----------------------------------
q1 17651 5071 4883 4883
q2 2045 142 136 136
q3 10574 1041 1008 1008
q4 4652 964 974 964
q5 7709 3211 3268 3211
q6 203 135 134 134
q7 1280 785 770 770
q8 9241 2070 2047 2047
q9 7577 6708 6672 6672
q10 8316 2667 2650 2650
q11 417 217 211 211
q12 767 332 341 332
q13 18022 3667 3714 3667
q14 286 260 258 258
q15 597 545 507 507
q16 464 427 419 419
q17 915 875 856 856
q18 7377 6538 6658 6538
q19 1558 1497 1492 1492
q20 652 354 365 354
q21 6909 3973 3997 3973
q22 871 346 332 332
Total cold run time: 108083 ms
Total hot run time: 41414 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4912 4973 4936 4936
q2 306 191 194 191
q3 3699 3660 3648 3648
q4 2573 2557 2543 2543
q5 5819 5799 5772 5772
q6 226 124 125 124
q7 2265 1687 1741 1687
q8 3077 3155 3152 3152
q9 8736 8729 9223 8729
q10 7056 4498 4549 4498
q11 597 439 411 411
q12 891 637 656 637
q13 4425 3493 3502 3493
q14 278 245 261 245
q15 648 546 555 546
q16 495 452 483 452
q17 1799 1686 1958 1686
q18 13780 7695 7646 7646
q19 1634 1655 1637 1637
q20 2129 1895 1856 1856
q21 6668 6270 6253 6253
q22 581 508 508 508
Total cold run time: 72594 ms
Total hot run time: 60650 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1928939093
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1929186819
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1933860840
<details>
<summary>TPC-DS: <b>Total hot run time: 180500 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3598d0d2e1b7ee9b5880a6218301f78d5244228b, data reload: false
query1 920 357 346 346
query2 6528 1693 1802 1693
query3 6693 208 204 204
query4 24607 22565 22396 22396
query5 4283 388 384 384
query6 257 175 165 165
query7 4608 302 314 302
query8 252 193 213 193
query9 8434 2886 2895 2886
query10 427 218 223 218
query11 16228 15531 15548 15531
query12 148 86 82 82
query13 1701 426 415 415
query14 9984 7913 7722 7722
query15 217 186 196 186
query16 7600 253 251 251
query17 1517 577 527 527
query18 1958 270 265 265
query19 197 144 149 144
query20 85 79 85 79
query21 188 127 110 110
query22 4898 4693 4726 4693
query23 32587 31698 31589 31589
query24 12658 3469 3437 3437
query25 618 381 361 361
query26 1784 158 167 158
query27 3041 316 308 308
query28 6571 1818 1820 1818
query29 1122 624 638 624
query30 274 139 152 139
query31 950 751 772 751
query32 98 63 57 57
query33 727 238 228 228
query34 1064 485 504 485
query35 949 828 824 824
query36 995 906 871 871
query37 129 61 61 61
query38 3457 3279 3304 3279
query39 1393 1323 1315 1315
query40 280 108 102 102
query41 36 35 35 35
query42 103 104 101 101
query43 470 447 449 447
query44 1068 700 713 700
query45 197 181 184 181
query46 1032 798 788 788
query47 1665 1551 1592 1551
query48 408 338 343 338
query49 1154 316 300 300
query50 799 396 379 379
query51 5327 5148 5127 5127
query52 105 90 95 90
query53 403 311 309 309
query54 313 229 225 225
query55 85 78 82 78
query56 224 203 207 203
query57 1041 932 903 903
query58 222 198 197 197
query59 2222 2181 2225 2181
query60 257 232 230 230
query61 85 83 84 83
query62 580 379 373 373
query63 348 311 287 287
query64 6331 3117 3187 3117
query65 3281 3266 3266 3266
query66 1339 330 325 325
query67 14499 13978 14335 13978
query68 5093 558 529 529
query69 523 356 359 356
query70 1232 1197 1268 1197
query71 389 259 260 259
query72 6333 2809 2634 2634
query73 695 310 325 310
query74 7111 6673 6715 6673
query75 3209 2565 2549 2549
query76 3283 1160 1217 1160
query77 354 239 237 237
query78 9409 8829 8757 8757
query79 985 511 504 504
query80 517 357 351 351
query81 404 207 206 206
query82 246 91 88 88
query83 150 124 122 122
query84 221 79 81 79
query85 760 341 339 339
query86 289 320 290 290
query87 3595 3400 3396 3396
query88 2722 2277 2292 2277
query89 453 377 365 365
query90 1788 168 166 166
query91 155 129 128 128
query92 53 49 46 46
query93 975 476 495 476
query94 1105 183 189 183
query95 8642 8467 387 387
query96 612 262 274 262
query97 4403 4268 4298 4268
query98 218 202 194 194
query99 1031 721 737 721
Total cold run time: 280976 ms
Total hot run time: 180500 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1933636774
<details>
<summary>TPC-DS: <b>Total hot run time: 188597 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0b0213d1dc795ea97e1552562893355f50b65e4, data reload: false
query1 925 358 348 348
query2 6524 1674 1660 1660
query3 6700 207 203 203
query4 24533 22483 22383 22383
query5 4314 464 463 463
query6 267 182 162 162
query7 4608 298 298 298
query8 236 200 213 200
query9 8430 2827 2835 2827
query10 419 228 230 228
query11 16216 15583 15476 15476
query12 147 84 83 83
query13 1700 411 436 411
query14 9381 7682 7821 7682
query15 208 182 190 182
query16 7520 249 247 247
query17 1428 558 539 539
query18 1953 283 269 269
query19 196 143 150 143
query20 88 82 85 82
query21 192 122 112 112
query22 4981 4697 4817 4697
query23 32616 31568 31684 31568
query24 12699 3466 3395 3395
query25 645 361 365 361
query26 1866 156 159 156
query27 3048 303 310 303
query28 6572 1855 1840 1840
query29 1105 614 614 614
query30 284 138 148 138
query31 944 766 760 760
query32 93 58 53 53
query33 725 236 243 236
query34 1062 479 488 479
query35 932 846 819 819
query36 997 922 864 864
query37 166 62 63 62
query38 3394 3388 3338 3338
query39 1378 1325 1318 1318
query40 282 108 108 108
query41 37 35 35 35
query42 108 95 100 95
query43 456 448 432 432
query44 1080 684 702 684
query45 200 185 179 179
query46 1058 764 760 760
query47 1671 1578 1533 1533
query48 426 346 343 343
query49 1221 306 294 294
query50 788 387 382 382
query51 5333 5146 5138 5138
query52 112 88 102 88
query53 394 308 292 292
query54 291 222 238 222
query55 83 81 75 75
query56 217 205 199 199
query57 1081 944 942 942
query58 226 198 199 198
query59 2269 2179 2097 2097
query60 243 232 234 232
query61 86 90 84 84
query62 600 374 351 351
query63 336 288 284 284
query64 6207 3110 3166 3110
query65 3299 3267 3245 3245
query66 1337 331 321 321
query67 14690 14390 14496 14390
query68 5102 541 537 537
query69 513 368 344 344
query70 1280 1240 1215 1215
query71 452 259 271 259
query72 6285 2785 2619 2619
query73 693 307 303 303
query74 7092 6756 6709 6709
query75 3241 2575 2550 2550
query76 3287 1146 1203 1146
query77 355 242 229 229
query78 9432 8865 8740 8740
query79 963 498 495 495
query80 533 348 347 347
query81 438 205 208 205
query82 168 82 86 82
query83 141 124 122 122
query84 227 77 77 77
query85 1045 355 347 347
query86 289 304 270 270
query87 3621 3393 3423 3393
query88 2696 2273 2270 2270
query89 440 374 351 351
query90 2001 166 169 166
query91 154 125 127 125
query92 53 48 47 47
query93 973 509 464 464
query94 1266 181 187 181
query95 8637 8510 8624 8510
query96 582 279 264 264
query97 4441 4237 4318 4237
query98 205 196 196 196
query99 1067 685 702 685
Total cold run time: 281295 ms
Total hot run time: 188597 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1929008378
<details>
<summary>ClickBench: <b>Total hot run time: 30.62 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5d946f7ebb4b89f17e7553eaab592f30608450b4, data reload: false
query1 0.04 0.03 0.04
query2 0.06 0.02 0.02
query3 0.22 0.06 0.06
query4 1.67 0.10 0.10
query5 0.53 0.52 0.51
query6 1.18 0.68 0.64
query7 0.02 0.01 0.01
query8 0.03 0.02 0.02
query9 0.56 0.51 0.50
query10 0.56 0.56 0.55
query11 0.12 0.09 0.09
query12 0.11 0.09 0.09
query13 0.61 0.61 0.60
query14 0.82 0.80 0.79
query15 0.79 0.77 0.77
query16 0.39 0.39 0.39
query17 1.01 1.04 0.96
query18 0.23 0.24 0.26
query19 1.87 1.78 1.77
query20 0.02 0.01 0.01
query21 15.42 0.58 0.59
query22 2.65 2.51 1.52
query23 17.13 0.81 0.84
query24 2.52 1.41 0.98
query25 0.40 0.12 0.14
query26 0.53 0.14 0.13
query27 0.05 0.05 0.04
query28 11.46 0.83 0.84
query29 12.59 3.11 3.15
query30 0.61 0.53 0.54
query31 2.78 0.35 0.35
query32 3.36 0.48 0.48
query33 3.20 3.20 3.25
query34 15.88 4.27 4.23
query35 4.30 4.27 4.32
query36 1.09 1.05 1.06
query37 0.06 0.05 0.05
query38 0.04 0.02 0.02
query39 0.02 0.01 0.01
query40 0.17 0.12 0.14
query41 0.08 0.01 0.02
query42 0.02 0.01 0.02
query43 0.03 0.02 0.02
Total cold run time: 105.23 s
Total hot run time: 30.62 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1933649471
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit a0b0213d1dc795ea97e1552562893355f50b65e4 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.3 seconds inserted 10000000 Rows, about 699K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1929013089
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 5d946f7ebb4b89f17e7553eaab592f30608450b4 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.9 seconds inserted 10000000 Rows, about 719K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1929153150
<details>
<summary>TPC-H: <b>Total hot run time: 37203 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aa0143a9000e1ee5ac23fb87fa90fd07423d9485, data reload: false
------ Round 1 ----------------------------------
q1 17725 4697 4616 4616
q2 2694 153 139 139
q3 11726 967 924 924
q4 4967 798 753 753
q5 7918 2901 2849 2849
q6 187 121 131 121
q7 1142 730 724 724
q8 9327 2004 2016 2004
q9 7252 6419 6353 6353
q10 8145 2372 2441 2372
q11 421 212 214 212
q12 739 277 279 277
q13 18015 3329 3304 3304
q14 277 245 259 245
q15 529 499 480 480
q16 472 409 412 409
q17 935 541 491 491
q18 6912 5886 5881 5881
q19 1594 1389 1383 1383
q20 604 349 332 332
q21 6716 3033 3081 3033
q22 811 321 301 301
Total cold run time: 109108 ms
Total hot run time: 37203 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4476 4426 4476 4426
q2 323 240 232 232
q3 3014 2884 2853 2853
q4 1833 1626 1684 1626
q5 5182 5230 5222 5222
q6 195 112 112 112
q7 2147 1778 1781 1778
q8 3139 3248 3262 3248
q9 8404 8346 8316 8316
q10 5877 3605 3532 3532
q11 550 442 469 442
q12 739 593 596 593
q13 7159 3114 3073 3073
q14 265 270 251 251
q15 528 488 485 485
q16 512 477 463 463
q17 1849 1652 1711 1652
q18 7992 7809 7592 7592
q19 9248 1480 1552 1480
q20 2115 1932 1897 1897
q21 4757 4597 4575 4575
q22 568 474 451 451
Total cold run time: 70872 ms
Total hot run time: 54299 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1928787078
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #30893:
URL: https://github.com/apache/doris/pull/30893#discussion_r1480188858
##########
be/src/vec/sink/group_commit_block_sink.cpp:
##########
@@ -292,5 +298,26 @@ size_t GroupCommitBlockSink::_pre_allocated(bool is_blocks_contain_all_load_data
: _state->content_length());
}
+void GroupCommitBlockSink::_remove_pre_allocated() {
+ if (!_is_pre_allocated_removed) {
+ std::string wal_path;
+ Status st = ExecEnv::GetInstance()->wal_mgr()->get_wal_path(_load_block_queue->txn_id,
+ wal_path);
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed to get wal path in remove pre allocated, reason: "
+ << st.to_string();
+ return;
+ }
+ st = ExecEnv::GetInstance()->wal_mgr()->update_wal_dir_pre_allocated(
+ WalManager::get_base_wal_path(wal_path), 0, _pre_allocated);
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed to remove pre allocated, reason: " << st.to_string();
+ return;
+ }
+ _is_pre_allocated_removed = true;
+ _load_block_queue->block_queue_pre_allocated.fetch_sub(_pre_allocated);
+ }
+};
Review Comment:
Check to return to reduce indention.
```suggestion
if (_is_pre_allocated_removed) return;
std::string wal_path;
Status st = ExecEnv::GetInstance()->wal_mgr()->get_wal_path(_load_block_queue->txn_id,
wal_path);
if (!st.ok()) {
LOG(WARNING) << "Failed to get wal path in remove pre allocated, reason: "
<< st.to_string();
return;
}
st = ExecEnv::GetInstance()->wal_mgr()->update_wal_dir_pre_allocated(
WalManager::get_base_wal_path(wal_path), 0, _pre_allocated);
if (!st.ok()) {
LOG(WARNING) << "Failed to remove pre allocated, reason: " << st.to_string();
return;
}
_is_pre_allocated_removed = true;
_load_block_queue->block_queue_pre_allocated.fetch_sub(_pre_allocated);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1930330029
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1929136158
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1929156009
TeamCity be ut coverage result:
Function Coverage: 36.19% (8627/23839)
Line Coverage: 28.20% (70547/250145)
Region Coverage: 27.22% (36415/133798)
Branch Coverage: 24.00% (18651/77728)
Coverage Report: http://coverage.selectdb-in.cc/coverage/aa0143a9000e1ee5ac23fb87fa90fd07423d9485_aa0143a9000e1ee5ac23fb87fa90fd07423d9485/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1933619457
<details>
<summary>TPC-H: <b>Total hot run time: 41687 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0b0213d1dc795ea97e1552562893355f50b65e4, data reload: false
------ Round 1 ----------------------------------
q1 17634 4986 4904 4904
q2 2057 140 137 137
q3 10579 1024 1029 1024
q4 4663 996 995 995
q5 7713 3282 3272 3272
q6 198 139 142 139
q7 1287 795 785 785
q8 9247 2083 2072 2072
q9 7592 6705 6674 6674
q10 8319 2650 2668 2650
q11 415 212 225 212
q12 824 339 335 335
q13 18005 3686 3673 3673
q14 299 263 259 259
q15 623 526 532 526
q16 478 429 413 413
q17 934 852 864 852
q18 7449 6816 6604 6604
q19 1572 1490 1503 1490
q20 646 372 351 351
q21 7095 4006 3984 3984
q22 867 355 336 336
Total cold run time: 108496 ms
Total hot run time: 41687 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4867 4838 4849 4838
q2 291 184 181 181
q3 3610 3597 3599 3597
q4 2531 2518 2515 2515
q5 5759 5758 5765 5758
q6 211 128 126 126
q7 2242 1683 1670 1670
q8 3023 3110 3117 3110
q9 8737 8709 8722 8709
q10 6787 4249 4253 4249
q11 505 362 357 357
q12 780 546 548 546
q13 4356 3420 3457 3420
q14 277 266 238 238
q15 594 510 503 503
q16 496 456 459 456
q17 1688 1625 1622 1622
q18 8348 7764 7584 7584
q19 1641 1646 1632 1632
q20 2131 1828 1839 1828
q21 6580 6183 6201 6183
q22 580 502 513 502
Total cold run time: 66034 ms
Total hot run time: 59624 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1929174151
<details>
<summary>TPC-DS: <b>Total hot run time: 174090 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aa0143a9000e1ee5ac23fb87fa90fd07423d9485, data reload: false
query1 940 338 341 338
query2 6526 1942 1975 1942
query3 6726 211 210 210
query4 31859 21999 21933 21933
query5 4248 354 346 346
query6 249 164 156 156
query7 4602 288 278 278
query8 246 166 200 166
query9 9107 2279 2286 2279
query10 413 214 217 214
query11 18801 15417 15321 15321
query12 130 76 74 74
query13 1625 428 410 410
query14 9293 6877 6967 6877
query15 227 190 182 182
query16 8113 264 254 254
query17 1871 516 493 493
query18 2117 267 256 256
query19 361 142 136 136
query20 79 81 78 78
query21 198 125 124 124
query22 4881 4786 4769 4769
query23 31324 30131 30001 30001
query24 11646 2720 2812 2720
query25 601 336 332 332
query26 1739 144 152 144
query27 2935 301 300 300
query28 7732 1837 1824 1824
query29 978 607 609 607
query30 288 132 141 132
query31 940 687 728 687
query32 95 52 47 47
query33 733 225 224 224
query34 1098 444 458 444
query35 860 745 737 737
query36 1009 922 941 922
query37 223 60 56 56
query38 3239 3055 3126 3055
query39 1304 1246 1242 1242
query40 268 94 94 94
query41 37 35 34 34
query42 103 92 91 91
query43 549 499 492 492
query44 1123 684 704 684
query45 193 186 174 174
query46 1055 650 647 647
query47 1576 1449 1497 1449
query48 442 348 372 348
query49 1222 277 270 270
query50 769 374 372 372
query51 5277 5173 5138 5138
query52 103 85 83 83
query53 332 263 275 263
query54 296 211 222 211
query55 81 76 75 75
query56 213 205 205 205
query57 998 902 875 875
query58 202 171 180 171
query59 2390 2287 2331 2287
query60 240 208 206 206
query61 83 82 87 82
query62 651 371 354 354
query63 296 271 278 271
query64 6509 3662 3317 3317
query65 3249 3229 3218 3218
query66 1354 318 318 318
query67 14323 14240 14076 14076
query68 4507 522 544 522
query69 460 326 320 320
query70 1256 1273 1164 1164
query71 322 246 241 241
query72 5912 2843 2652 2652
query73 716 323 330 323
query74 6520 6218 6308 6218
query75 3073 2363 2305 2305
query76 2677 930 945 930
query77 432 226 227 226
query78 9079 8796 8718 8718
query79 3391 482 501 482
query80 2180 346 335 335
query81 527 198 198 198
query82 1255 81 86 81
query83 395 123 131 123
query84 284 77 82 77
query85 1759 332 316 316
query86 472 314 286 286
query87 3351 3273 3304 3273
query88 4384 2316 2296 2296
query89 440 373 352 352
query90 2143 169 165 165
query91 152 120 134 120
query92 56 45 47 45
query93 4408 493 451 451
query94 1453 175 177 175
query95 7991 358 7720 358
query96 593 273 270 270
query97 4209 4087 4121 4087
query98 214 201 192 192
query99 1152 706 674 674
Total cold run time: 299053 ms
Total hot run time: 174090 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #30893:
URL: https://github.com/apache/doris/pull/30893#discussion_r1480188858
##########
be/src/vec/sink/group_commit_block_sink.cpp:
##########
@@ -292,5 +298,26 @@ size_t GroupCommitBlockSink::_pre_allocated(bool is_blocks_contain_all_load_data
: _state->content_length());
}
+void GroupCommitBlockSink::_remove_pre_allocated() {
+ if (!_is_pre_allocated_removed) {
+ std::string wal_path;
+ Status st = ExecEnv::GetInstance()->wal_mgr()->get_wal_path(_load_block_queue->txn_id,
+ wal_path);
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed to get wal path in remove pre allocated, reason: "
+ << st.to_string();
+ return;
+ }
+ st = ExecEnv::GetInstance()->wal_mgr()->update_wal_dir_pre_allocated(
+ WalManager::get_base_wal_path(wal_path), 0, _pre_allocated);
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed to remove pre allocated, reason: " << st.to_string();
+ return;
+ }
+ _is_pre_allocated_removed = true;
+ _load_block_queue->block_queue_pre_allocated.fetch_sub(_pre_allocated);
+ }
+};
Review Comment:
```suggestion
if (_is_pre_allocated_removed) return;
std::string wal_path;
Status st = ExecEnv::GetInstance()->wal_mgr()->get_wal_path(_load_block_queue->txn_id,
wal_path);
if (!st.ok()) {
LOG(WARNING) << "Failed to get wal path in remove pre allocated, reason: "
<< st.to_string();
return;
}
st = ExecEnv::GetInstance()->wal_mgr()->update_wal_dir_pre_allocated(
WalManager::get_base_wal_path(wal_path), 0, _pre_allocated);
if (!st.ok()) {
LOG(WARNING) << "Failed to remove pre allocated, reason: " << st.to_string();
return;
}
_is_pre_allocated_removed = true;
_load_block_queue->block_queue_pre_allocated.fetch_sub(_pre_allocated);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1933370078
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1928803352
<details>
<summary>TPC-H: <b>Total hot run time: 37584 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 68d282845a28baab08be249fc5deaa5b8e3fea67, data reload: false
------ Round 1 ----------------------------------
q1 17676 4525 4599 4525
q2 2167 149 142 142
q3 11079 978 952 952
q4 5420 819 858 819
q5 7933 2870 2868 2868
q6 197 131 131 131
q7 1223 762 760 760
q8 9567 2037 2050 2037
q9 8238 6383 6400 6383
q10 8132 2420 2391 2391
q11 426 211 242 211
q12 737 293 273 273
q13 18011 3380 3331 3331
q14 278 252 256 252
q15 537 488 488 488
q16 480 403 421 403
q17 943 566 519 519
q18 6796 6005 6115 6005
q19 1571 1480 1393 1393
q20 592 363 337 337
q21 6974 3121 3084 3084
q22 831 295 280 280
Total cold run time: 109808 ms
Total hot run time: 37584 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4564 4554 4345 4345
q2 330 225 247 225
q3 3007 2847 2808 2808
q4 1832 1649 1691 1649
q5 5257 5235 5261 5235
q6 195 116 117 116
q7 2118 1761 1791 1761
q8 3145 3288 3292 3288
q9 8404 8325 8331 8325
q10 5941 3633 3609 3609
q11 543 464 464 464
q12 731 626 580 580
q13 8779 3100 3082 3082
q14 270 265 255 255
q15 548 493 496 493
q16 514 482 474 474
q17 1848 1703 1711 1703
q18 8000 7795 7694 7694
q19 11020 1559 1546 1546
q20 2158 1921 1910 1910
q21 4862 4720 4433 4433
q22 526 467 457 457
Total cold run time: 74592 ms
Total hot run time: 54452 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1928800078
TeamCity be ut coverage result:
Function Coverage: 36.18% (8627/23846)
Line Coverage: 28.20% (70543/250185)
Region Coverage: 27.21% (36413/133827)
Branch Coverage: 23.99% (18653/77738)
Coverage Report: http://coverage.selectdb-in.cc/coverage/68d282845a28baab08be249fc5deaa5b8e3fea67_68d282845a28baab08be249fc5deaa5b8e3fea67/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1928984862
TeamCity be ut coverage result:
Function Coverage: 36.19% (8627/23840)
Line Coverage: 28.20% (70550/250140)
Region Coverage: 27.22% (36417/133796)
Branch Coverage: 24.00% (18653/77726)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5d946f7ebb4b89f17e7553eaab592f30608450b4_5d946f7ebb4b89f17e7553eaab592f30608450b4/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1933544834
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1933657410
TeamCity be ut coverage result:
Function Coverage: 35.83% (8554/23874)
Line Coverage: 27.78% (69462/250057)
Region Coverage: 26.91% (36037/133900)
Branch Coverage: 23.70% (18428/77762)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a0b0213d1dc795ea97e1552562893355f50b65e4_a0b0213d1dc795ea97e1552562893355f50b65e4/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1929183653
<details>
<summary>ClickBench: <b>Total hot run time: 31.49 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aa0143a9000e1ee5ac23fb87fa90fd07423d9485, data reload: false
query1 0.03 0.03 0.03
query2 0.05 0.02 0.02
query3 0.22 0.06 0.06
query4 1.68 0.10 0.10
query5 0.53 0.51 0.51
query6 1.22 0.63 0.63
query7 0.01 0.02 0.01
query8 0.04 0.02 0.02
query9 0.56 0.50 0.52
query10 0.53 0.55 0.54
query11 0.11 0.08 0.08
query12 0.10 0.08 0.09
query13 0.60 0.61 0.60
query14 0.80 0.79 0.80
query15 0.79 0.77 0.78
query16 0.37 0.40 0.39
query17 1.00 1.03 0.97
query18 0.22 0.27 0.25
query19 1.91 1.77 1.82
query20 0.02 0.01 0.01
query21 15.45 0.56 0.57
query22 2.64 2.39 2.00
query23 16.99 0.84 0.77
query24 2.43 1.69 1.21
query25 0.34 0.21 0.16
query26 0.66 0.13 0.13
query27 0.05 0.04 0.05
query28 10.76 0.85 0.82
query29 12.52 3.14 3.19
query30 0.57 0.53 0.54
query31 2.90 0.34 0.35
query32 3.39 0.48 0.47
query33 3.19 3.27 3.23
query34 15.81 4.33 4.31
query35 4.34 4.35 4.34
query36 1.09 1.05 1.07
query37 0.07 0.04 0.04
query38 0.04 0.03 0.02
query39 0.02 0.02 0.02
query40 0.16 0.13 0.13
query41 0.07 0.02 0.02
query42 0.02 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 104.32 s
Total hot run time: 31.49 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1929188717
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit aa0143a9000e1ee5ac23fb87fa90fd07423d9485 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.5 seconds inserted 10000000 Rows, about 740K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1928984019
<details>
<summary>TPC-H: <b>Total hot run time: 36818 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5d946f7ebb4b89f17e7553eaab592f30608450b4, data reload: false
------ Round 1 ----------------------------------
q1 17646 4506 4523 4506
q2 2032 140 131 131
q3 10610 936 910 910
q4 4646 746 689 689
q5 7698 2815 2791 2791
q6 187 121 119 119
q7 1173 742 718 718
q8 9265 2034 2011 2011
q9 7305 6460 6351 6351
q10 8109 2444 2422 2422
q11 404 202 205 202
q12 781 264 269 264
q13 17996 3282 3298 3282
q14 275 245 263 245
q15 539 493 488 488
q16 474 407 420 407
q17 934 555 506 506
q18 6662 5859 5840 5840
q19 1569 1355 1333 1333
q20 572 362 311 311
q21 7032 2989 3134 2989
q22 829 340 303 303
Total cold run time: 106738 ms
Total hot run time: 36818 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4396 4435 4352 4352
q2 343 240 231 231
q3 2970 2847 2792 2792
q4 1886 1643 1654 1643
q5 5147 5183 5223 5183
q6 189 113 116 113
q7 2154 1762 1714 1714
q8 3107 3235 3274 3235
q9 8342 8239 8301 8239
q10 5895 3555 3588 3555
q11 534 459 461 459
q12 729 579 586 579
q13 13123 3101 3053 3053
q14 277 242 266 242
q15 538 487 490 487
q16 504 450 465 450
q17 1837 1689 1725 1689
q18 7920 7711 7538 7538
q19 8597 1540 1558 1540
q20 2141 1891 1895 1891
q21 4791 4583 4475 4475
q22 537 433 466 433
Total cold run time: 75957 ms
Total hot run time: 53893 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1928780337
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1928818843
<details>
<summary>ClickBench: <b>Total hot run time: 31.1 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 68d282845a28baab08be249fc5deaa5b8e3fea67, data reload: false
query1 0.04 0.03 0.02
query2 0.06 0.03 0.02
query3 0.24 0.06 0.06
query4 1.67 0.09 0.10
query5 0.52 0.52 0.53
query6 1.19 0.65 0.64
query7 0.02 0.02 0.01
query8 0.04 0.03 0.03
query9 0.52 0.51 0.50
query10 0.57 0.57 0.55
query11 0.12 0.09 0.08
query12 0.11 0.09 0.10
query13 0.60 0.61 0.60
query14 0.79 0.81 0.81
query15 0.79 0.77 0.78
query16 0.40 0.40 0.39
query17 1.03 1.00 0.99
query18 0.22 0.26 0.24
query19 1.91 1.75 1.75
query20 0.01 0.01 0.01
query21 15.40 0.59 0.59
query22 2.39 2.10 1.84
query23 17.63 0.80 0.86
query24 2.27 0.89 1.05
query25 0.32 0.24 0.15
query26 0.51 0.14 0.13
query27 0.05 0.05 0.05
query28 12.13 0.85 0.82
query29 12.48 3.22 3.30
query30 0.69 0.56 0.55
query31 2.79 0.34 0.35
query32 3.36 0.48 0.48
query33 3.21 3.22 3.25
query34 15.78 4.30 4.27
query35 4.32 4.28 4.30
query36 1.09 1.04 1.06
query37 0.06 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.02 0.01
query40 0.16 0.16 0.13
query41 0.06 0.01 0.02
query42 0.02 0.02 0.01
query43 0.03 0.02 0.02
Total cold run time: 105.66 s
Total hot run time: 31.1 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1928821299
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 68d282845a28baab08be249fc5deaa5b8e3fea67 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.8 seconds inserted 10000000 Rows, about 724K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #30893:
URL: https://github.com/apache/doris/pull/30893#discussion_r1480188858
##########
be/src/vec/sink/group_commit_block_sink.cpp:
##########
@@ -292,5 +298,26 @@ size_t GroupCommitBlockSink::_pre_allocated(bool is_blocks_contain_all_load_data
: _state->content_length());
}
+void GroupCommitBlockSink::_remove_pre_allocated() {
+ if (!_is_pre_allocated_removed) {
+ std::string wal_path;
+ Status st = ExecEnv::GetInstance()->wal_mgr()->get_wal_path(_load_block_queue->txn_id,
+ wal_path);
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed to get wal path in remove pre allocated, reason: "
+ << st.to_string();
+ return;
+ }
+ st = ExecEnv::GetInstance()->wal_mgr()->update_wal_dir_pre_allocated(
+ WalManager::get_base_wal_path(wal_path), 0, _pre_allocated);
+ if (!st.ok()) {
+ LOG(WARNING) << "Failed to remove pre allocated, reason: " << st.to_string();
+ return;
+ }
+ _is_pre_allocated_removed = true;
+ _load_block_queue->block_queue_pre_allocated.fetch_sub(_pre_allocated);
+ }
+};
Review Comment:
Check to return first to reduce indention.
```suggestion
if (_is_pre_allocated_removed) return;
std::string wal_path;
Status st = ExecEnv::GetInstance()->wal_mgr()->get_wal_path(_load_block_queue->txn_id,
wal_path);
if (!st.ok()) {
LOG(WARNING) << "Failed to get wal path in remove pre allocated, reason: "
<< st.to_string();
return;
}
st = ExecEnv::GetInstance()->wal_mgr()->update_wal_dir_pre_allocated(
WalManager::get_base_wal_path(wal_path), 0, _pre_allocated);
if (!st.ok()) {
LOG(WARNING) << "Failed to remove pre allocated, reason: " << st.to_string();
return;
}
_is_pre_allocated_removed = true;
_load_block_queue->block_queue_pre_allocated.fetch_sub(_pre_allocated);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1931117418
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1933868969
<details>
<summary>ClickBench: <b>Total hot run time: 31.25 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3598d0d2e1b7ee9b5880a6218301f78d5244228b, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.02 0.02
query3 0.23 0.07 0.07
query4 1.64 0.10 0.10
query5 0.47 0.48 0.48
query6 1.36 0.60 0.63
query7 0.02 0.02 0.01
query8 0.04 0.03 0.03
query9 0.55 0.46 0.47
query10 0.48 0.49 0.49
query11 0.14 0.09 0.09
query12 0.12 0.10 0.10
query13 0.59 0.59 0.59
query14 0.76 0.80 0.80
query15 0.83 0.80 0.79
query16 0.33 0.33 0.34
query17 0.91 0.93 0.95
query18 0.19 0.18 0.16
query19 1.79 1.72 1.70
query20 0.01 0.00 0.01
query21 15.41 0.67 0.57
query22 3.41 3.39 2.63
query23 17.57 1.11 0.94
query24 2.09 0.67 0.38
query25 0.64 0.07 0.06
query26 0.15 0.14 0.15
query27 0.06 0.05 0.06
query28 11.81 0.86 0.83
query29 12.58 3.32 3.47
query30 0.51 0.47 0.44
query31 2.77 0.36 0.37
query32 3.35 0.48 0.48
query33 3.12 3.13 3.21
query34 15.36 4.44 4.49
query35 4.50 4.51 4.49
query36 1.08 0.95 0.96
query37 0.06 0.05 0.05
query38 0.05 0.03 0.04
query39 0.02 0.01 0.02
query40 0.17 0.14 0.15
query41 0.08 0.01 0.01
query42 0.03 0.02 0.01
query43 0.03 0.02 0.02
Total cold run time: 105.4 s
Total hot run time: 31.25 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1928814510
<details>
<summary>TPC-DS: <b>Total hot run time: 174022 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 68d282845a28baab08be249fc5deaa5b8e3fea67, data reload: false
query1 942 350 339 339
query2 6537 1935 1908 1908
query3 6714 216 213 213
query4 31703 21999 22004 21999
query5 4272 433 362 362
query6 252 174 168 168
query7 4602 289 280 280
query8 241 179 183 179
query9 9003 2266 2255 2255
query10 408 218 211 211
query11 19039 15326 15452 15326
query12 129 77 77 77
query13 1624 414 413 413
query14 9201 7139 6977 6977
query15 231 172 182 172
query16 8161 267 250 250
query17 1918 535 508 508
query18 2101 263 252 252
query19 364 145 141 141
query20 80 77 82 77
query21 204 130 133 130
query22 4554 4506 4600 4506
query23 31325 30014 30145 30014
query24 11029 2803 2754 2754
query25 591 344 344 344
query26 1482 141 148 141
query27 2954 301 300 300
query28 7630 1828 1822 1822
query29 918 623 615 615
query30 283 133 141 133
query31 879 699 702 699
query32 97 58 57 57
query33 723 224 211 211
query34 1069 460 469 460
query35 866 758 778 758
query36 1071 942 891 891
query37 115 57 54 54
query38 3299 3144 3135 3135
query39 1297 1262 1263 1262
query40 268 94 90 90
query41 36 34 35 34
query42 98 89 95 89
query43 508 501 505 501
query44 1052 684 703 684
query45 202 181 175 175
query46 1061 673 636 636
query47 1632 1468 1525 1468
query48 421 343 353 343
query49 1176 290 278 278
query50 765 376 391 376
query51 5206 5213 5147 5147
query52 93 89 81 81
query53 343 257 274 257
query54 268 220 218 218
query55 80 72 75 72
query56 226 212 200 200
query57 944 915 914 914
query58 203 176 180 176
query59 2381 2206 2345 2206
query60 238 216 218 216
query61 93 83 83 83
query62 687 370 361 361
query63 296 269 265 265
query64 6260 3245 3581 3245
query65 3284 3219 3205 3205
query66 1064 319 311 311
query67 14465 14118 14137 14118
query68 4458 536 531 531
query69 462 310 325 310
query70 1310 1241 1243 1241
query71 339 268 230 230
query72 5964 2869 2665 2665
query73 703 324 320 320
query74 6595 6217 6247 6217
query75 3117 2359 2323 2323
query76 2845 909 940 909
query77 381 227 234 227
query78 9123 8790 8501 8501
query79 2769 517 505 505
query80 2002 348 342 342
query81 524 198 199 198
query82 845 90 81 81
query83 253 123 122 122
query84 289 85 82 82
query85 2178 339 328 328
query86 487 320 291 291
query87 3427 3213 3183 3183
query88 4347 2349 2366 2349
query89 464 360 378 360
query90 2020 166 165 165
query91 152 123 122 122
query92 58 46 44 44
query93 4070 499 484 484
query94 1282 176 175 175
query95 8063 352 7732 352
query96 598 277 280 277
query97 4238 4102 4088 4088
query98 221 195 188 188
query99 1110 670 672 670
Total cold run time: 295952 ms
Total hot run time: 174022 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1928949766
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1929000359
<details>
<summary>TPC-DS: <b>Total hot run time: 173429 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5d946f7ebb4b89f17e7553eaab592f30608450b4, data reload: false
query1 958 349 338 338
query2 6543 2047 1950 1950
query3 6695 215 210 210
query4 31671 21990 21982 21982
query5 4237 353 402 353
query6 250 163 171 163
query7 4606 285 277 277
query8 244 170 184 170
query9 9121 2297 2266 2266
query10 406 202 226 202
query11 17280 15378 15346 15346
query12 142 77 72 72
query13 1614 412 441 412
query14 9339 6928 6755 6755
query15 235 179 178 178
query16 8118 253 249 249
query17 1885 535 508 508
query18 2096 268 263 263
query19 357 138 136 136
query20 81 76 84 76
query21 205 117 127 117
query22 4902 4433 4479 4433
query23 30781 29964 29999 29964
query24 10364 2730 2765 2730
query25 538 343 340 340
query26 708 141 145 141
query27 2189 295 299 295
query28 5940 1850 1851 1850
query29 864 610 604 604
query30 283 133 142 133
query31 895 692 711 692
query32 92 52 47 47
query33 624 221 219 219
query34 814 461 475 461
query35 839 764 749 749
query36 1083 959 888 888
query37 91 57 57 57
query38 3243 3151 3089 3089
query39 1319 1241 1237 1237
query40 184 99 91 91
query41 39 34 35 34
query42 98 88 92 88
query43 500 468 482 468
query44 1019 686 705 686
query45 189 183 173 173
query46 1044 640 655 640
query47 1548 1482 1497 1482
query48 436 363 362 362
query49 1048 284 287 284
query50 762 386 371 371
query51 5278 5126 5162 5126
query52 101 78 87 78
query53 339 285 265 265
query54 282 212 211 211
query55 80 72 73 72
query56 219 211 200 200
query57 1005 903 917 903
query58 204 172 178 172
query59 2576 2496 2292 2292
query60 232 210 219 210
query61 88 93 84 84
query62 641 351 347 347
query63 303 263 274 263
query64 4885 3649 3366 3366
query65 3258 3224 3199 3199
query66 813 302 304 302
query67 14557 14068 13818 13818
query68 4283 537 539 537
query69 465 318 338 318
query70 1276 1201 1241 1201
query71 323 256 261 256
query72 6000 2858 2690 2690
query73 700 321 333 321
query74 6631 6243 6351 6243
query75 3040 2331 2333 2331
query76 2582 906 910 906
query77 365 230 227 227
query78 9139 8841 8512 8512
query79 3652 482 525 482
query80 2179 350 343 343
query81 548 196 195 195
query82 1264 86 83 83
query83 373 130 125 125
query84 285 77 89 77
query85 1569 341 330 330
query86 483 307 310 307
query87 3359 3200 3213 3200
query88 4465 2330 2307 2307
query89 437 360 367 360
query90 2059 164 162 162
query91 156 125 124 124
query92 55 45 42 42
query93 4965 477 461 461
query94 1280 176 171 171
query95 479 7787 356 356
query96 591 272 267 267
query97 4208 4096 4151 4096
query98 211 197 199 197
query99 1093 693 663 663
Total cold run time: 281895 ms
Total hot run time: 173429 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #30893:
URL: https://github.com/apache/doris/pull/30893#discussion_r1482400080
##########
be/src/olap/wal/wal_dirs_info.cpp:
##########
@@ -206,18 +208,18 @@ Status WalDirsInfo::update_all_wal_dir_used() {
return Status::OK();
}
-Status WalDirsInfo::update_wal_dir_pre_allocated(const std::string& wal_dir,
- size_t increase_pre_allocated,
- size_t decrease_pre_allocated) {
+Status WalDirsInfo::update_wal_dir_estimated_wal_bytes(const std::string& wal_dir,
Review Comment:
warning: method 'update_wal_dir_estimated_wal_bytes' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal/wal_dirs_info.h:76:
```diff
- Status update_wal_dir_estimated_wal_bytes(const std::string& wal_dir,
+ static Status update_wal_dir_estimated_wal_bytes(const std::string& wal_dir,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1933342250
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](group commit) optimize pre allocated calculation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30893:
URL: https://github.com/apache/doris/pull/30893#issuecomment-1933644869
<details>
<summary>ClickBench: <b>Total hot run time: 31.68 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0b0213d1dc795ea97e1552562893355f50b65e4, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.03 0.03
query3 0.23 0.08 0.08
query4 1.64 0.08 0.07
query5 0.48 0.49 0.50
query6 1.34 0.61 0.63
query7 0.02 0.02 0.01
query8 0.04 0.02 0.02
query9 0.54 0.45 0.46
query10 0.50 0.49 0.51
query11 0.13 0.10 0.09
query12 0.12 0.10 0.10
query13 0.59 0.60 0.58
query14 0.77 0.78 0.79
query15 0.83 0.80 0.79
query16 0.33 0.34 0.34
query17 0.91 0.94 0.90
query18 0.16 0.17 0.16
query19 1.81 1.69 1.65
query20 0.02 0.01 0.01
query21 15.42 0.66 0.57
query22 2.88 3.89 3.17
query23 17.35 0.99 1.07
query24 1.98 0.39 0.21
query25 0.58 0.06 0.07
query26 0.15 0.14 0.14
query27 0.05 0.06 0.04
query28 12.48 0.84 0.82
query29 12.53 3.42 3.36
query30 0.54 0.48 0.48
query31 2.77 0.37 0.38
query32 3.32 0.47 0.48
query33 3.16 3.16 3.13
query34 15.35 4.47 4.50
query35 4.48 4.50 4.48
query36 1.08 0.97 0.95
query37 0.08 0.05 0.05
query38 0.03 0.03 0.04
query39 0.02 0.02 0.02
query40 0.17 0.16 0.15
query41 0.07 0.02 0.02
query42 0.02 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 105.08 s
Total hot run time: 31.68 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org