You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "BiteTheDDDDt (via GitHub)" <gi...@apache.org> on 2024/02/04 03:59:31 UTC
[PR] [Improvement](memory) clear arena when finalize one row [doris]
BiteTheDDDDt opened a new pull request, #30788:
URL: https://github.com/apache/doris/pull/30788
## Proposed changes
clear arena when finalize one row
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](memory) clear arena when finalize one row [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30788:
URL: https://github.com/apache/doris/pull/30788#issuecomment-1931233100
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](memory) clear arena when finalize one row [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #30788:
URL: https://github.com/apache/doris/pull/30788
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](memory) clear arena when finalize one row [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30788:
URL: https://github.com/apache/doris/pull/30788#issuecomment-1925580016
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit b68ea417fae677f99f929c2971152bdb6a1ac704 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.5 seconds inserted 10000000 Rows, about 689K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](memory) clear arena when finalize one row [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30788:
URL: https://github.com/apache/doris/pull/30788#issuecomment-1925577222
<details>
<summary>TPC-H: <b>Total hot run time: 36780 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b68ea417fae677f99f929c2971152bdb6a1ac704, data reload: false
------ Round 1 ----------------------------------
q1 17721 4589 4434 4434
q2 2037 139 132 132
q3 10682 958 893 893
q4 4652 747 687 687
q5 7721 2758 2792 2758
q6 185 118 120 118
q7 1155 726 717 717
q8 9315 2005 2014 2005
q9 7545 6469 6453 6453
q10 8121 2432 2423 2423
q11 414 207 207 207
q12 741 267 280 267
q13 18278 3323 3375 3323
q14 292 255 262 255
q15 544 484 493 484
q16 483 413 420 413
q17 940 554 534 534
q18 6890 5936 5852 5852
q19 1549 1407 1227 1227
q20 627 325 344 325
q21 6631 3054 2977 2977
q22 817 300 296 296
Total cold run time: 107340 ms
Total hot run time: 36780 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4489 4360 4349 4349
q2 330 227 230 227
q3 2979 2884 2831 2831
q4 1788 1593 1590 1590
q5 5211 5258 5241 5241
q6 188 115 115 115
q7 2171 1759 1795 1759
q8 3128 3258 3288 3258
q9 8354 8300 8279 8279
q10 5824 3565 3509 3509
q11 545 468 451 451
q12 770 586 584 584
q13 12737 3077 3043 3043
q14 268 234 258 234
q15 532 483 494 483
q16 521 467 459 459
q17 1839 1677 1679 1677
q18 7941 7796 7631 7631
q19 7057 1522 1548 1522
q20 2118 1940 1915 1915
q21 4734 4555 4634 4555
q22 543 457 452 452
Total cold run time: 74067 ms
Total hot run time: 54164 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](memory) clear arena when finalize one row [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #30788:
URL: https://github.com/apache/doris/pull/30788#discussion_r1480861397
##########
be/src/olap/memtable.cpp:
##########
@@ -373,19 +373,29 @@ void MemTable::_finalize_one_row(RowInBlock* row,
// get value columns from agg_places
for (size_t i = _tablet_schema->num_key_columns(); i < _num_columns; ++i) {
auto function = _agg_functions[i];
- auto agg_place = row->agg_places(i);
- auto col_ptr = _output_mutable_block.get_column_by_position(i).get();
+ auto* agg_place = row->agg_places(i);
+ auto* col_ptr = _output_mutable_block.get_column_by_position(i).get();
function->insert_result_into(agg_place, *col_ptr);
+
if constexpr (is_final) {
function->destroy(agg_place);
} else {
function->reset(agg_place);
- function->add(agg_place, const_cast<const doris::vectorized::IColumn**>(&col_ptr),
- row_pos, _arena.get());
}
}
+
+ _arena->clear();
+
if constexpr (is_final) {
row->remove_init_agg();
+ } else {
+ for (size_t i = _tablet_schema->num_key_columns(); i < _num_columns; ++i) {
+ auto function = _agg_functions[i];
+ auto* agg_place = row->agg_places(i);
Review Comment:
why move code from line 383 to here??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](memory) clear arena when finalize one row [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30788:
URL: https://github.com/apache/doris/pull/30788#issuecomment-1925572030
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](memory) clear arena when finalize one row [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #30788:
URL: https://github.com/apache/doris/pull/30788#issuecomment-1925572033
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](memory) clear arena when finalize one row [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30788:
URL: https://github.com/apache/doris/pull/30788#issuecomment-1925579597
<details>
<summary>ClickBench: <b>Total hot run time: 30.71 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b68ea417fae677f99f929c2971152bdb6a1ac704, data reload: false
query1 0.03 0.03 0.03
query2 0.06 0.02 0.02
query3 0.22 0.06 0.06
query4 1.68 0.10 0.10
query5 0.53 0.53 0.51
query6 1.18 0.64 0.67
query7 0.02 0.01 0.02
query8 0.04 0.03 0.03
query9 0.56 0.50 0.49
query10 0.55 0.56 0.55
query11 0.12 0.09 0.09
query12 0.11 0.08 0.10
query13 0.60 0.62 0.61
query14 0.80 0.80 0.79
query15 0.80 0.78 0.78
query16 0.39 0.37 0.38
query17 1.03 0.99 1.00
query18 0.24 0.20 0.21
query19 1.83 1.73 1.79
query20 0.01 0.01 0.02
query21 15.40 0.60 0.58
query22 2.52 2.66 2.30
query23 17.38 0.69 0.80
query24 2.48 0.71 0.46
query25 0.37 0.17 0.07
query26 0.45 0.14 0.14
query27 0.05 0.04 0.05
query28 12.26 0.83 0.82
query29 12.52 3.11 3.19
query30 0.60 0.55 0.47
query31 2.79 0.34 0.34
query32 3.37 0.48 0.48
query33 3.21 3.21 3.23
query34 15.79 4.31 4.29
query35 4.31 4.30 4.34
query36 1.11 1.05 1.06
query37 0.06 0.05 0.05
query38 0.05 0.02 0.03
query39 0.02 0.02 0.02
query40 0.17 0.13 0.13
query41 0.07 0.02 0.01
query42 0.02 0.02 0.02
query43 0.02 0.02 0.02
Total cold run time: 105.82 s
Total hot run time: 30.71 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](memory) clear arena when finalize one row [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30788:
URL: https://github.com/apache/doris/pull/30788#issuecomment-1925577432
TeamCity be ut coverage result:
Function Coverage: 36.16% (8626/23858)
Line Coverage: 28.22% (70544/250009)
Region Coverage: 27.22% (36394/133684)
Branch Coverage: 24.01% (18641/77624)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b68ea417fae677f99f929c2971152bdb6a1ac704_b68ea417fae677f99f929c2971152bdb6a1ac704/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](memory) clear arena when finalize one row [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30788:
URL: https://github.com/apache/doris/pull/30788#issuecomment-1925574144
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](memory) clear arena when finalize one row [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #30788:
URL: https://github.com/apache/doris/pull/30788#issuecomment-1931235468
Very very important work, This PR will reduce many memory for agg table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](memory) clear arena when finalize one row [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30788:
URL: https://github.com/apache/doris/pull/30788#issuecomment-1931233123
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](memory) clear arena when finalize one row [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30788:
URL: https://github.com/apache/doris/pull/30788#issuecomment-1925578783
<details>
<summary>TPC-DS: <b>Total hot run time: 175040 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b68ea417fae677f99f929c2971152bdb6a1ac704, data reload: false
query1 932 344 330 330
query2 6560 1993 1910 1910
query3 6731 209 204 204
query4 31775 22129 22196 22129
query5 4291 363 418 363
query6 257 166 171 166
query7 4601 287 276 276
query8 245 190 182 182
query9 8920 2361 2335 2335
query10 419 243 216 216
query11 16940 15573 15607 15573
query12 142 80 89 80
query13 1738 433 443 433
query14 10396 7274 7154 7154
query15 272 188 198 188
query16 8247 275 253 253
query17 2197 571 523 523
query18 2368 275 263 263
query19 354 144 142 142
query20 85 76 79 76
query21 194 139 118 118
query22 4915 4705 4536 4536
query23 30785 30067 30040 30040
query24 10493 2773 2752 2752
query25 571 344 341 341
query26 1160 139 140 139
query27 2958 295 310 295
query28 7598 1865 1873 1865
query29 869 618 598 598
query30 279 132 139 132
query31 917 712 725 712
query32 88 53 46 46
query33 729 223 221 221
query34 1098 455 464 455
query35 850 783 747 747
query36 1070 918 940 918
query37 117 57 60 57
query38 3189 3115 3117 3115
query39 1307 1239 1244 1239
query40 190 97 94 94
query41 40 35 35 35
query42 100 89 93 89
query43 537 481 504 481
query44 1063 696 705 696
query45 189 182 175 175
query46 1063 654 650 650
query47 1489 1462 1515 1462
query48 461 352 348 348
query49 1084 294 283 283
query50 762 381 385 381
query51 5191 5125 5152 5125
query52 100 82 92 82
query53 356 264 273 264
query54 272 218 221 218
query55 84 72 77 72
query56 219 217 196 196
query57 977 911 878 878
query58 209 176 184 176
query59 2324 2260 2266 2260
query60 252 211 221 211
query61 85 88 83 83
query62 652 373 363 363
query63 293 258 282 258
query64 5266 3707 3470 3470
query65 3234 3224 3234 3224
query66 815 319 319 319
query67 14622 14359 14163 14163
query68 5334 536 545 536
query69 462 321 338 321
query70 1270 1271 1205 1205
query71 335 256 252 252
query72 6365 2856 2703 2703
query73 697 318 317 317
query74 6611 6103 6256 6103
query75 3058 2336 2352 2336
query76 3270 857 954 857
query77 475 226 226 226
query78 9212 8784 8522 8522
query79 2398 505 507 505
query80 1741 359 354 354
query81 541 193 196 193
query82 653 80 80 80
query83 281 127 132 127
query84 250 81 80 80
query85 2164 346 325 325
query86 484 293 300 293
query87 3384 3222 3246 3222
query88 3757 2371 2350 2350
query89 436 374 388 374
query90 2094 166 165 165
query91 169 120 138 120
query92 60 44 42 42
query93 2261 497 471 471
query94 1449 174 172 172
query95 478 366 8006 366
query96 598 270 278 270
query97 4273 4133 4097 4097
query98 229 206 184 184
query99 1153 694 699 694
Total cold run time: 284528 ms
Total hot run time: 175040 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](memory) clear arena when finalize one row [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #30788:
URL: https://github.com/apache/doris/pull/30788#discussion_r1480861397
##########
be/src/olap/memtable.cpp:
##########
@@ -373,19 +373,29 @@ void MemTable::_finalize_one_row(RowInBlock* row,
// get value columns from agg_places
for (size_t i = _tablet_schema->num_key_columns(); i < _num_columns; ++i) {
auto function = _agg_functions[i];
- auto agg_place = row->agg_places(i);
- auto col_ptr = _output_mutable_block.get_column_by_position(i).get();
+ auto* agg_place = row->agg_places(i);
+ auto* col_ptr = _output_mutable_block.get_column_by_position(i).get();
function->insert_result_into(agg_place, *col_ptr);
+
if constexpr (is_final) {
function->destroy(agg_place);
} else {
function->reset(agg_place);
- function->add(agg_place, const_cast<const doris::vectorized::IColumn**>(&col_ptr),
- row_pos, _arena.get());
}
}
+
+ _arena->clear();
+
if constexpr (is_final) {
row->remove_init_agg();
+ } else {
+ for (size_t i = _tablet_schema->num_key_columns(); i < _num_columns; ++i) {
+ auto function = _agg_functions[i];
+ auto* agg_place = row->agg_places(i);
Review Comment:
why move code from line 383 to here??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org