You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "jacktengg (via GitHub)" <gi...@apache.org> on 2024/03/21 13:01:07 UTC
[PR] [improvement](spill) spill trigger improvement [doris]
jacktengg opened a new pull request, #32641:
URL: https://github.com/apache/doris/pull/32641
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2017726728
TeamCity be ut coverage result:
Function Coverage: 35.23% (8733/24790)
Line Coverage: 27.03% (71513/264560)
Region Coverage: 26.28% (37103/141203)
Branch Coverage: 23.17% (18971/81894)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0e279b3fac84fecc43a2e6fc9829bd07ba245b4f_0e279b3fac84fecc43a2e6fc9829bd07ba245b4f/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2012234980
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542289829
##########
be/src/runtime/workload_group/workload_group.h:
##########
@@ -82,6 +82,31 @@ class WorkloadGroup : public std::enable_shared_from_this<WorkloadGroup> {
int64_t memory_used();
+ int spill_threshold_low_water_mark() const {
+ return _spill_low_watermark.load(std::memory_order_relaxed);
+ }
+ int spill_threashold_high_water_mark() const {
+ return _spill_high_watermark.load(std::memory_order_relaxed);
+ }
+
+ void set_weighted_memory_used(int64_t wg_total_mem_used, double ratio);
+
+ std::unordered_set<std::shared_ptr<MemTrackerLimiter>> get_all_mem_trackers();
+
+ int64_t get_weighted_memory_used() {
+ return _weighted_mem_used.load(std::memory_order_relaxed);
+ }
+
+ void check_mem_used(bool& is_low_wartermark, bool& is_high_wartermark) {
Review Comment:
add const to the function signature.
And do not use reference as return value. Use pointer instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1533875471
##########
be/src/pipeline/exec/partitioned_hash_join_sink_operator.cpp:
##########
@@ -48,15 +48,25 @@ Status PartitionedHashJoinSinkLocalState::open(RuntimeState* state) {
RETURN_IF_ERROR(PipelineXSinkLocalState::open(state));
return _partitioner->open(state);
}
+Status PartitionedHashJoinSinkLocalState::close(RuntimeState* state, Status exec_status) {
Review Comment:
warning: method 'close' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/exec/partitioned_hash_join_sink_operator.h:48:
```diff
- Status close(RuntimeState* state, Status exec_status) override;
+ static Status close(RuntimeState* state, Status exec_status) override;
```
##########
be/src/runtime/workload_group/workload_group.h:
##########
@@ -82,6 +82,24 @@ class WorkloadGroup : public std::enable_shared_from_this<WorkloadGroup> {
int64_t memory_used();
+ double spill_threshold_low_water_mark() const { return _spill_threshold_low_water_mark; }
+ double spill_threashold_high_water_mark() const { return _spill_threashold_high_water_mark; }
+
+ void set_weighted_memory_used(int64_t tg_total_mem_used, double ratio);
+
+ std::unordered_set<std::shared_ptr<MemTrackerLimiter>> get_all_mem_trackers();
+
+ int64_t get_weighted_memory_used() {
+ return _weighted_mem_used.load(std::memory_order_relaxed);
+ }
+
+ void check_mem_used(bool& is_low_wartermark, bool& is_high_wartermark) {
Review Comment:
warning: method 'check_mem_used' can be made const [readability-make-member-function-const]
```suggestion
void check_mem_used(bool& is_low_wartermark, bool& is_high_wartermark) const {
```
##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,121 @@
<< "ms, deleted group size:" << deleted_task_groups.size();
}
+struct WorkloadGroupMemInfo {
+ int64_t total_mem_used = 0;
+ int64_t weighted_mem_used = 0;
+ bool is_low_wartermark = false;
+ bool is_high_wartermark = false;
+ double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_used_memory() {
Review Comment:
warning: function 'refresh_wg_used_memory' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void WorkloadGroupMgr::refresh_wg_used_memory() {
^
```
<details>
<summary>Additional context</summary>
**be/src/runtime/workload_group/workload_group_manager.cpp:146:** 106 lines including whitespace and comments (threshold 80)
```cpp
void WorkloadGroupMgr::refresh_wg_used_memory() {
^
```
</details>
##########
be/src/runtime/memory/mem_tracker_limiter.h:
##########
@@ -244,6 +245,20 @@ class MemTrackerLimiter final : public MemTracker {
// Log the memory usage when memory limit is exceeded.
std::string tracker_limit_exceeded_str();
+ void set_weighted_mem(int64_t weighted_limit, double ratio) {
+ std::lock_guard<std::mutex> l(_weighted_mem_lock);
+ _weighted_consumption = _consumption->current_value() * ratio;
+ _weighted_limit = weighted_limit;
+ }
+ void get_weighted_mem_info(int64_t& weighted_limit, int64_t& weighted_consumption) {
Review Comment:
warning: method 'get_weighted_mem_info' can be made const [readability-make-member-function-const]
```suggestion
void get_weighted_mem_info(int64_t& weighted_limit, int64_t& weighted_consumption) const {
```
##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,121 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
<< "ms, deleted group size:" << deleted_task_groups.size();
}
+struct WorkloadGroupMemInfo {
+ int64_t total_mem_used = 0;
+ int64_t weighted_mem_used = 0;
+ bool is_low_wartermark = false;
+ bool is_high_wartermark = false;
+ double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_used_memory() {
Review Comment:
warning: method 'refresh_wg_used_memory' can be made static [readability-convert-member-functions-to-static]
be/src/runtime/workload_group/workload_group_manager.h:56:
```diff
- void refresh_wg_used_memory();
+ static void refresh_wg_used_memory();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1540421187
##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,125 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
<< "ms, deleted group size:" << deleted_task_groups.size();
}
+struct WorkloadGroupMemInfo {
+ int64_t total_mem_used = 0;
+ int64_t weighted_mem_used = 0;
+ bool is_low_wartermark = false;
+ bool is_high_wartermark = false;
+ double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {
Review Comment:
warning: function 'refresh_wg_memory_info' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void WorkloadGroupMgr::refresh_wg_memory_info() {
^
```
<details>
<summary>Additional context</summary>
**be/src/runtime/workload_group/workload_group_manager.cpp:146:** 110 lines including whitespace and comments (threshold 80)
```cpp
void WorkloadGroupMgr::refresh_wg_memory_info() {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026780194
TeamCity be ut coverage result:
Function Coverage: 35.56% (8839/24858)
Line Coverage: 27.29% (72455/265547)
Region Coverage: 26.51% (37528/141569)
Branch Coverage: 23.31% (19129/82066)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c8f6c475195c6fa5512777393f6824eb63ce1a44_c8f6c475195c6fa5512777393f6824eb63ce1a44/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024330793
<details>
<summary>TPC-DS: <b>Total hot run time: 181686 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 615fff13fc8d598fd9aad875a0fb730eabc282d0, data reload: false
query1 927 365 355 355
query2 6535 2044 1843 1843
query3 6700 215 207 207
query4 31645 21303 21350 21303
query5 4419 395 400 395
query6 269 188 172 172
query7 4634 297 294 294
query8 227 167 165 165
query9 9350 2287 2275 2275
query10 567 252 246 246
query11 17232 14208 14192 14192
query12 139 94 87 87
query13 1626 414 429 414
query14 9618 7643 7571 7571
query15 243 207 199 199
query16 8158 271 281 271
query17 1938 579 539 539
query18 2103 281 269 269
query19 283 148 147 147
query20 96 83 85 83
query21 199 129 123 123
query22 5141 4820 4812 4812
query23 33463 32680 33122 32680
query24 10744 2839 2819 2819
query25 586 364 357 357
query26 1219 150 151 150
query27 3002 353 356 353
query28 7598 1901 1862 1862
query29 856 614 604 604
query30 293 147 146 146
query31 941 714 737 714
query32 94 56 52 52
query33 760 256 252 252
query34 1062 480 498 480
query35 818 620 604 604
query36 1003 911 901 901
query37 124 66 66 66
query38 3560 3415 3421 3415
query39 1468 1440 1436 1436
query40 227 113 111 111
query41 47 45 45 45
query42 102 92 95 92
query43 492 461 458 458
query44 1206 732 727 727
query45 264 262 273 262
query46 1114 715 694 694
query47 1889 1840 1840 1840
query48 436 359 355 355
query49 1119 328 331 328
query50 753 364 365 364
query51 6662 6619 6597 6597
query52 95 90 89 89
query53 342 270 270 270
query54 284 230 230 230
query55 84 80 78 78
query56 243 216 214 214
query57 1214 1148 1130 1130
query58 227 203 199 199
query59 2800 2548 2591 2548
query60 258 232 239 232
query61 100 95 95 95
query62 655 453 457 453
query63 316 276 276 276
query64 5725 4202 4149 4149
query65 3059 3027 3051 3027
query66 949 386 369 369
query67 15297 14930 14908 14908
query68 5281 520 520 520
query69 568 377 371 371
query70 1246 1161 1177 1161
query71 405 271 266 266
query72 6510 2865 2682 2682
query73 714 316 324 316
query74 7221 6495 6542 6495
query75 2988 2266 2239 2239
query76 3483 847 859 847
query77 389 271 257 257
query78 10964 10142 10146 10142
query79 8327 515 517 515
query80 1896 395 408 395
query81 557 223 222 222
query82 1580 86 89 86
query83 307 155 149 149
query84 293 82 82 82
query85 1731 317 312 312
query86 496 307 289 289
query87 3723 3478 3590 3478
query88 5224 2383 2284 2284
query89 522 362 377 362
query90 1943 176 178 176
query91 171 138 137 137
query92 66 50 47 47
query93 6118 486 485 485
query94 1257 178 174 174
query95 444 333 336 333
query96 587 266 269 266
query97 2689 2497 2473 2473
query98 227 213 208 208
query99 1200 935 924 924
Total cold run time: 304726 ms
Total hot run time: 181686 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542711431
##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,127 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
<< "ms, deleted group size:" << deleted_task_groups.size();
}
+struct WorkloadGroupMemInfo {
+ int64_t total_mem_used = 0;
+ int64_t weighted_mem_used = 0;
+ bool is_low_wartermark = false;
+ bool is_high_wartermark = false;
+ double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {
Review Comment:
warning: function 'refresh_wg_memory_info' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void WorkloadGroupMgr::refresh_wg_memory_info() {
^
```
<details>
<summary>Additional context</summary>
**be/src/runtime/workload_group/workload_group_manager.cpp:146:** 112 lines including whitespace and comments (threshold 80)
```cpp
void WorkloadGroupMgr::refresh_wg_memory_info() {
^
```
</details>
##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -326,6 +304,48 @@ Status PipelineXTask::execute(bool* eos) {
return status;
}
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+ auto* query_ctx = state->get_query_ctx();
+ auto wg = query_ctx->workload_group();
+ if (!wg) {
+ LOG_ONCE(INFO) << "no workload group for query " << print_id(state->query_id());
+ return false;
+ }
+ bool is_wg_mem_low_water_mark = false;
+ bool is_wg_mem_high_water_mark = false;
+ wg->check_mem_used(&is_wg_mem_low_water_mark, &is_wg_mem_high_water_mark);
+ if (is_wg_mem_high_water_mark) {
+ if (revocable_mem_bytes > 0) {
+ LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
+ return true;
Review Comment:
warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:317:
```diff
- if (revocable_mem_bytes > 0) {
- LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
- return true;
- }
- return false;
+ return revocable_mem_bytes > 0;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026677979
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026766666
<details>
<summary>TPC-H: <b>Total hot run time: 37970 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c8f6c475195c6fa5512777393f6824eb63ce1a44, data reload: false
------ Round 1 ----------------------------------
q1 17638 4315 4162 4162
q2 2111 155 146 146
q3 10593 1139 1216 1139
q4 10226 786 818 786
q5 7469 2976 2913 2913
q6 205 124 128 124
q7 1038 584 578 578
q8 9328 2024 2028 2024
q9 6977 6395 6364 6364
q10 8422 3461 3549 3461
q11 419 230 218 218
q12 425 205 190 190
q13 17808 2841 2864 2841
q14 227 214 208 208
q15 509 479 461 461
q16 474 363 367 363
q17 954 530 622 530
q18 7192 6663 6546 6546
q19 1659 1462 1430 1430
q20 573 268 266 266
q21 3686 2936 2922 2922
q22 342 298 318 298
Total cold run time: 108275 ms
Total hot run time: 37970 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4169 4110 4098 4098
q2 327 232 226 226
q3 2955 2859 2865 2859
q4 1885 1538 1615 1538
q5 5202 5227 5240 5227
q6 199 119 119 119
q7 2192 1826 1818 1818
q8 3175 3340 3303 3303
q9 8478 8443 8452 8443
q10 3820 3858 3950 3858
q11 586 467 465 465
q12 757 575 579 575
q13 17028 3034 3082 3034
q14 292 270 279 270
q15 540 479 478 478
q16 494 466 449 449
q17 1793 1530 1550 1530
q18 8135 7797 7655 7655
q19 2884 1553 1588 1553
q20 2010 1883 1702 1702
q21 4947 4784 4885 4784
q22 557 476 486 476
Total cold run time: 72425 ms
Total hot run time: 54460 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026787628
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit c8f6c475195c6fa5512777393f6824eb63ce1a44 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.9 seconds inserted 10000000 Rows, about 719K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024333754
TeamCity be ut coverage result:
Function Coverage: 35.22% (8741/24817)
Line Coverage: 27.00% (71549/265007)
Region Coverage: 26.25% (37121/141401)
Branch Coverage: 23.15% (18981/81990)
Coverage Report: http://coverage.selectdb-in.cc/coverage/615fff13fc8d598fd9aad875a0fb730eabc282d0_615fff13fc8d598fd9aad875a0fb730eabc282d0/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1544065493
##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -327,6 +305,50 @@ Status PipelineXTask::execute(bool* eos) {
return status;
}
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+ auto* query_ctx = state->get_query_ctx();
+ auto wg = query_ctx->workload_group();
+ if (!wg) {
+ LOG_ONCE(INFO) << "no workload group for query " << print_id(state->query_id());
+ return false;
+ }
+ bool is_wg_mem_low_water_mark = false;
+ bool is_wg_mem_high_water_mark = false;
+ wg->check_mem_used(&is_wg_mem_low_water_mark, &is_wg_mem_high_water_mark);
+ if (is_wg_mem_high_water_mark) {
+ if (revocable_mem_bytes > 0) {
+ LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
+ return true;
Review Comment:
warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:318:
```diff
- if (revocable_mem_bytes > 0) {
- LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
- return true;
- }
- return false;
+ return revocable_mem_bytes > 0;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2021937403
<details>
<summary>TPC-DS: <b>Total hot run time: 181926 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e1783354448196d335626f926218759a3286ec2e, data reload: false
query1 926 369 352 352
query2 6547 1890 1840 1840
query3 6701 213 215 213
query4 31457 21417 21308 21308
query5 4318 413 400 400
query6 272 181 192 181
query7 4624 288 285 285
query8 229 169 171 169
query9 9277 2307 2277 2277
query10 557 243 249 243
query11 15277 14252 14268 14252
query12 135 90 93 90
query13 1623 415 429 415
query14 10233 7626 8040 7626
query15 243 204 191 191
query16 8233 266 262 262
query17 1960 591 556 556
query18 2104 301 286 286
query19 359 160 161 160
query20 100 89 88 88
query21 204 128 131 128
query22 5101 4849 4798 4798
query23 33344 32743 32746 32743
query24 10814 2879 2875 2875
query25 627 388 385 385
query26 1398 157 161 157
query27 2957 357 357 357
query28 7618 1913 1894 1894
query29 909 642 634 634
query30 302 156 149 149
query31 976 728 752 728
query32 86 57 57 57
query33 770 260 250 250
query34 1049 485 490 485
query35 847 609 605 605
query36 1013 871 887 871
query37 130 64 69 64
query38 3530 3394 3466 3394
query39 1466 1458 1460 1458
query40 205 111 114 111
query41 48 47 46 46
query42 104 95 94 94
query43 485 440 447 440
query44 1203 724 730 724
query45 253 265 264 264
query46 1108 748 711 711
query47 1935 1839 1836 1836
query48 447 387 347 347
query49 1101 330 332 330
query50 764 374 372 372
query51 6699 6631 6595 6595
query52 102 96 91 91
query53 351 272 271 271
query54 289 228 240 228
query55 87 85 83 83
query56 245 222 219 219
query57 1227 1141 1128 1128
query58 235 206 201 201
query59 2698 2567 2639 2567
query60 267 238 242 238
query61 96 91 91 91
query62 658 463 441 441
query63 303 281 274 274
query64 5804 4034 3993 3993
query65 3130 3047 3021 3021
query66 1425 372 347 347
query67 15263 15201 14903 14903
query68 5359 521 525 521
query69 574 374 362 362
query70 1196 1153 1071 1071
query71 405 264 272 264
query72 6343 2885 2675 2675
query73 703 316 320 316
query74 7020 6571 6513 6513
query75 2950 2266 2214 2214
query76 3545 911 908 908
query77 418 271 266 266
query78 10855 10307 10127 10127
query79 8706 527 528 527
query80 2120 400 395 395
query81 560 220 217 217
query82 1385 93 86 86
query83 290 149 149 149
query84 293 84 84 84
query85 2195 323 314 314
query86 502 287 293 287
query87 3702 3572 3548 3548
query88 4931 2322 2314 2314
query89 553 384 373 373
query90 1993 181 179 179
query91 169 140 139 139
query92 66 47 47 47
query93 6534 504 492 492
query94 1243 181 180 180
query95 439 331 334 331
query96 601 274 278 274
query97 2675 2498 2473 2473
query98 239 225 207 207
query99 1240 900 881 881
Total cold run time: 304652 ms
Total hot run time: 181926 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542643876
##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,135 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
<< "ms, deleted group size:" << deleted_task_groups.size();
}
+struct WorkloadGroupMemInfo {
+ int64_t total_mem_used = 0;
+ int64_t weighted_mem_used = 0;
+ bool is_low_wartermark = false;
+ bool is_high_wartermark = false;
+ double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {
+ std::shared_lock<std::shared_mutex> r_lock(_group_mutex);
+ // workload group id -> workload group queries
+ std::unordered_map<uint64_t, std::unordered_map<TUniqueId, std::weak_ptr<QueryContext>>>
+ all_wg_queries;
+ for (auto& [wg_id, wg] : _workload_groups) {
+ all_wg_queries.insert({wg_id, wg->queries()});
+ }
+
+ int64_t all_queries_mem_used = 0;
+
+ // calculate total memory used of each workload group and total memory used of all queries
+ std::unordered_map<uint64_t, WorkloadGroupMemInfo> wgs_mem_info;
+ for (auto& [wg_id, wg_queries] : all_wg_queries) {
+ int64_t wg_total_mem_used = 0;
+ for (const auto& [query_id, query_ctx_ptr] : wg_queries) {
+ if (auto query_ctx = query_ctx_ptr.lock()) {
+ wg_total_mem_used +=
+ query_ctx->is_cancelled() ? 0 : query_ctx->query_mem_tracker->consumption();
+ }
+ }
+ all_queries_mem_used += wg_total_mem_used;
+ wgs_mem_info[wg_id] = {wg_total_mem_used};
+ }
+
+ auto proc_vm_rss = PerfCounters::get_vm_rss();
+ if (all_queries_mem_used <= 0) {
+ return;
+ }
+
+ auto process_mem_used = doris::MemInfo::proc_mem_no_allocator_cache();
+ auto sys_mem_available = doris::MemInfo::sys_mem_available();
+ std::string debug_msg = fmt::format(
+ "\nProcess Memory Summary: process_vm_rss: {}, process mem: {}, sys mem available: "
+ "{}, all quries mem: {}",
+ PrettyPrinter::print(proc_vm_rss, TUnit::BYTES),
+ PrettyPrinter::print(process_mem_used, TUnit::BYTES),
+ PrettyPrinter::print(sys_mem_available, TUnit::BYTES),
+ PrettyPrinter::print(all_queries_mem_used, TUnit::BYTES));
+ LOG_EVERY_N(INFO, 5) << debug_msg;
+
+ if (proc_vm_rss < all_queries_mem_used) {
+ all_queries_mem_used = proc_vm_rss;
+ }
+
+ // process memory used is actually bigger than all_queries_mem_used,
+ // because memory of page cache, allocator cache, segment cache etc. are included
+ // in process_mem_used.
+ // we count these cache memories equally on workload groups.
+ double ratio = (double)proc_vm_rss / (double)all_queries_mem_used;
+
+ for (auto& wg : _workload_groups) {
+ auto wg_mem_limit = wg.second->memory_limit();
+ auto& wg_mem_info = wgs_mem_info[wg.first];
+ wg_mem_info.weighted_mem_used = wg_mem_info.total_mem_used * ratio;
+ wg_mem_info.mem_used_ratio = (double)wg_mem_info.weighted_mem_used / wg_mem_limit;
+
+ wg.second->set_weighted_memory_used(wg_mem_info.total_mem_used, ratio);
+
+ auto spill_low_water_mark = wg.second->spill_threshold_low_water_mark();
+ auto spill_high_water_mark = wg.second->spill_threashold_high_water_mark();
+ wg_mem_info.is_high_wartermark = (wg_mem_info.weighted_mem_used >
+ ((double)wg_mem_limit * spill_high_water_mark / 100));
+ wg_mem_info.is_low_wartermark = (wg_mem_info.weighted_mem_used >
+ ((double)wg_mem_limit * spill_low_water_mark / 100));
+
+ // calculate query weighted memory limit of task group
+ const auto& wg_queries = all_wg_queries[wg.first];
+ auto wg_query_count = wg_queries.size();
Review Comment:
这里如果query count < 1 感觉可以直接continue了;
那么216 行直接就是wg_mem_limit + wg_query_count) / wg_query_count
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024972418
<details>
<summary>TPC-H: <b>Total hot run time: 37769 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit df0ddde4d7f8ba7db99389a85bb281abe6306fe4, data reload: false
------ Round 1 ----------------------------------
q1 17620 4264 4122 4122
q2 2109 160 155 155
q3 10779 1174 1271 1174
q4 10697 771 761 761
q5 7467 2966 2924 2924
q6 203 126 125 125
q7 942 536 497 497
q8 9357 2072 2012 2012
q9 6944 6367 6326 6326
q10 8470 3444 3586 3444
q11 435 236 223 223
q12 388 209 200 200
q13 17814 2911 2881 2881
q14 233 193 205 193
q15 508 475 466 466
q16 495 384 373 373
q17 968 521 608 521
q18 7230 6589 6508 6508
q19 2381 1449 1515 1449
q20 547 275 264 264
q21 3598 2995 2859 2859
q22 362 292 309 292
Total cold run time: 109547 ms
Total hot run time: 37769 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4177 4061 4062 4061
q2 329 227 227 227
q3 2993 2835 2823 2823
q4 1856 1600 1571 1571
q5 5238 5269 5257 5257
q6 198 116 119 116
q7 1753 1369 1367 1367
q8 3197 3312 3324 3312
q9 8518 8441 8525 8441
q10 3784 3848 3770 3770
q11 547 449 446 446
q12 723 565 540 540
q13 16924 2836 2865 2836
q14 275 248 243 243
q15 503 474 460 460
q16 488 429 413 413
q17 1749 1511 1468 1468
q18 7427 7128 7085 7085
q19 1613 1491 1557 1491
q20 1905 1721 1721 1721
q21 4874 4729 4655 4655
q22 518 444 444 444
Total cold run time: 69589 ms
Total hot run time: 52747 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024321329
<details>
<summary>TPC-H: <b>Total hot run time: 37958 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 615fff13fc8d598fd9aad875a0fb730eabc282d0, data reload: false
------ Round 1 ----------------------------------
q1 17601 4755 4141 4141
q2 2108 154 157 154
q3 10654 1158 1223 1158
q4 10232 829 772 772
q5 7453 3067 2958 2958
q6 200 123 124 123
q7 1101 594 572 572
q8 9355 2018 2030 2018
q9 7370 6689 6609 6609
q10 8462 3488 3563 3488
q11 438 224 213 213
q12 425 193 193 193
q13 17789 2892 2854 2854
q14 241 200 208 200
q15 512 462 465 462
q16 503 378 387 378
q17 959 565 611 565
q18 7086 6574 6358 6358
q19 3989 1378 1430 1378
q20 555 258 262 258
q21 3548 2813 2990 2813
q22 365 293 300 293
Total cold run time: 110946 ms
Total hot run time: 37958 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4147 4124 4041 4041
q2 332 233 227 227
q3 3082 2880 2847 2847
q4 1875 1552 1561 1552
q5 5302 5367 5330 5330
q6 195 117 117 117
q7 2225 1849 1853 1849
q8 3190 3319 3317 3317
q9 8716 8702 8670 8670
q10 3825 3787 3777 3777
q11 530 452 438 438
q12 711 560 550 550
q13 16928 2867 2856 2856
q14 285 249 254 249
q15 501 467 464 464
q16 484 423 419 419
q17 1740 1527 1471 1471
q18 7471 7148 7000 7000
q19 1624 1547 1548 1547
q20 1911 1739 1709 1709
q21 4936 4770 4693 4693
q22 542 427 460 427
Total cold run time: 70552 ms
Total hot run time: 53550 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542288795
##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -326,6 +304,46 @@ Status PipelineXTask::execute(bool* eos) {
return status;
}
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+ auto* query_ctx = state->get_query_ctx();
+ bool need_revoke = query_ctx->query_mem_tracker->need_revoke();
+ if (!need_revoke) {
+ return false;
+ }
+
+ const auto min_revocable_mem_bytes = state->min_revocable_mem();
+
+ auto tg = query_ctx->workload_group();
Review Comment:
If tg == nullptr then return false; to avoid core when there is no workload group bind to this query. And print log in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542289053
##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -326,6 +304,46 @@ Status PipelineXTask::execute(bool* eos) {
return status;
}
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+ auto* query_ctx = state->get_query_ctx();
+ bool need_revoke = query_ctx->query_mem_tracker->need_revoke();
+ if (!need_revoke) {
+ return false;
+ }
+
+ const auto min_revocable_mem_bytes = state->min_revocable_mem();
+
+ auto tg = query_ctx->workload_group();
Review Comment:
do not use tg. tg is old name "task group", use wg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2025006677
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit df0ddde4d7f8ba7db99389a85bb281abe6306fe4 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.0 seconds inserted 10000000 Rows, about 714K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2021868148
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024860541
TeamCity be ut coverage result:
Function Coverage: 35.58% (8832/24825)
Line Coverage: 27.29% (72377/265175)
Region Coverage: 26.53% (37532/141479)
Branch Coverage: 23.34% (19145/82032)
Coverage Report: http://coverage.selectdb-in.cc/coverage/09207c21e85cb9bb7da144b2960060fec2a8e348_09207c21e85cb9bb7da144b2960060fec2a8e348/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026784311
<details>
<summary>ClickBench: <b>Total hot run time: 29.21 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c8f6c475195c6fa5512777393f6824eb63ce1a44, data reload: false
query1 0.04 0.03 0.03
query2 0.06 0.02 0.02
query3 0.22 0.06 0.06
query4 1.65 0.09 0.10
query5 0.50 0.48 0.47
query6 1.14 0.66 0.63
query7 0.01 0.01 0.02
query8 0.04 0.02 0.03
query9 0.55 0.50 0.50
query10 0.54 0.54 0.55
query11 0.12 0.09 0.09
query12 0.11 0.09 0.09
query13 0.58 0.58 0.59
query14 0.74 0.76 0.77
query15 0.81 0.78 0.79
query16 0.36 0.40 0.37
query17 0.92 1.04 0.95
query18 0.21 0.22 0.24
query19 1.79 1.75 1.68
query20 0.01 0.02 0.02
query21 15.54 0.57 0.56
query22 2.68 2.73 1.15
query23 17.13 0.84 0.73
query24 3.32 2.16 1.38
query25 0.19 0.14 0.20
query26 0.63 0.14 0.12
query27 0.04 0.03 0.03
query28 10.43 0.86 0.83
query29 12.52 3.24 3.20
query30 0.27 0.09 0.08
query31 2.82 0.36 0.36
query32 3.33 0.43 0.45
query33 2.77 2.78 2.84
query34 16.16 4.38 4.32
query35 4.41 4.43 4.45
query36 0.66 0.47 0.46
query37 0.08 0.06 0.06
query38 0.05 0.03 0.04
query39 0.04 0.02 0.02
query40 0.17 0.13 0.14
query41 0.07 0.02 0.02
query42 0.02 0.02 0.02
query43 0.02 0.02 0.03
Total cold run time: 103.75 s
Total hot run time: 29.21 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024866370
<details>
<summary>TPC-H: <b>Total hot run time: 37716 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 09207c21e85cb9bb7da144b2960060fec2a8e348, data reload: false
------ Round 1 ----------------------------------
q1 17613 4536 4120 4120
q2 2108 159 151 151
q3 10587 1176 1207 1176
q4 10242 831 803 803
q5 7470 2941 2940 2940
q6 200 124 127 124
q7 1039 609 577 577
q8 9332 2013 1969 1969
q9 6921 6320 6316 6316
q10 8456 3533 3582 3533
q11 434 223 225 223
q12 393 200 189 189
q13 17801 2853 2828 2828
q14 229 200 207 200
q15 505 470 457 457
q16 495 368 365 365
q17 956 612 524 524
q18 7099 6453 6409 6409
q19 3027 1384 1372 1372
q20 570 249 251 249
q21 3554 2906 2904 2904
q22 339 298 287 287
Total cold run time: 109370 ms
Total hot run time: 37716 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4108 4082 4051 4051
q2 335 228 235 228
q3 2971 2873 2869 2869
q4 1806 1572 1535 1535
q5 5185 5229 5234 5229
q6 196 117 121 117
q7 2220 1862 1837 1837
q8 3169 3301 3289 3289
q9 8465 8424 8418 8418
q10 3824 3831 3794 3794
q11 541 439 447 439
q12 739 568 543 543
q13 16937 2832 2823 2823
q14 283 251 256 251
q15 502 451 450 450
q16 455 411 409 409
q17 1735 1477 1440 1440
q18 7502 7173 7220 7173
q19 1621 1516 1535 1516
q20 1905 1739 1702 1702
q21 4814 4643 4695 4643
q22 495 426 445 426
Total cold run time: 69808 ms
Total hot run time: 53182 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026354187
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542627384
##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,135 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
<< "ms, deleted group size:" << deleted_task_groups.size();
}
+struct WorkloadGroupMemInfo {
+ int64_t total_mem_used = 0;
+ int64_t weighted_mem_used = 0;
+ bool is_low_wartermark = false;
+ bool is_high_wartermark = false;
+ double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {
+ std::shared_lock<std::shared_mutex> r_lock(_group_mutex);
+ // workload group id -> workload group queries
+ std::unordered_map<uint64_t, std::unordered_map<TUniqueId, std::weak_ptr<QueryContext>>>
+ all_wg_queries;
+ for (auto& [wg_id, wg] : _workload_groups) {
+ all_wg_queries.insert({wg_id, wg->queries()});
+ }
+
+ int64_t all_queries_mem_used = 0;
+
+ // calculate total memory used of each workload group and total memory used of all queries
+ std::unordered_map<uint64_t, WorkloadGroupMemInfo> wgs_mem_info;
+ for (auto& [wg_id, wg_queries] : all_wg_queries) {
+ int64_t wg_total_mem_used = 0;
+ for (const auto& [query_id, query_ctx_ptr] : wg_queries) {
+ if (auto query_ctx = query_ctx_ptr.lock()) {
+ wg_total_mem_used +=
+ query_ctx->is_cancelled() ? 0 : query_ctx->query_mem_tracker->consumption();
Review Comment:
这样可能不行,如果把canceled 设置为0,那么可能会oom,因为此时canceled的query 的内存还没释放。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024719391
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542643876
##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,135 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
<< "ms, deleted group size:" << deleted_task_groups.size();
}
+struct WorkloadGroupMemInfo {
+ int64_t total_mem_used = 0;
+ int64_t weighted_mem_used = 0;
+ bool is_low_wartermark = false;
+ bool is_high_wartermark = false;
+ double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {
+ std::shared_lock<std::shared_mutex> r_lock(_group_mutex);
+ // workload group id -> workload group queries
+ std::unordered_map<uint64_t, std::unordered_map<TUniqueId, std::weak_ptr<QueryContext>>>
+ all_wg_queries;
+ for (auto& [wg_id, wg] : _workload_groups) {
+ all_wg_queries.insert({wg_id, wg->queries()});
+ }
+
+ int64_t all_queries_mem_used = 0;
+
+ // calculate total memory used of each workload group and total memory used of all queries
+ std::unordered_map<uint64_t, WorkloadGroupMemInfo> wgs_mem_info;
+ for (auto& [wg_id, wg_queries] : all_wg_queries) {
+ int64_t wg_total_mem_used = 0;
+ for (const auto& [query_id, query_ctx_ptr] : wg_queries) {
+ if (auto query_ctx = query_ctx_ptr.lock()) {
+ wg_total_mem_used +=
+ query_ctx->is_cancelled() ? 0 : query_ctx->query_mem_tracker->consumption();
+ }
+ }
+ all_queries_mem_used += wg_total_mem_used;
+ wgs_mem_info[wg_id] = {wg_total_mem_used};
+ }
+
+ auto proc_vm_rss = PerfCounters::get_vm_rss();
+ if (all_queries_mem_used <= 0) {
+ return;
+ }
+
+ auto process_mem_used = doris::MemInfo::proc_mem_no_allocator_cache();
+ auto sys_mem_available = doris::MemInfo::sys_mem_available();
+ std::string debug_msg = fmt::format(
+ "\nProcess Memory Summary: process_vm_rss: {}, process mem: {}, sys mem available: "
+ "{}, all quries mem: {}",
+ PrettyPrinter::print(proc_vm_rss, TUnit::BYTES),
+ PrettyPrinter::print(process_mem_used, TUnit::BYTES),
+ PrettyPrinter::print(sys_mem_available, TUnit::BYTES),
+ PrettyPrinter::print(all_queries_mem_used, TUnit::BYTES));
+ LOG_EVERY_N(INFO, 5) << debug_msg;
+
+ if (proc_vm_rss < all_queries_mem_used) {
+ all_queries_mem_used = proc_vm_rss;
+ }
+
+ // process memory used is actually bigger than all_queries_mem_used,
+ // because memory of page cache, allocator cache, segment cache etc. are included
+ // in process_mem_used.
+ // we count these cache memories equally on workload groups.
+ double ratio = (double)proc_vm_rss / (double)all_queries_mem_used;
+
+ for (auto& wg : _workload_groups) {
+ auto wg_mem_limit = wg.second->memory_limit();
+ auto& wg_mem_info = wgs_mem_info[wg.first];
+ wg_mem_info.weighted_mem_used = wg_mem_info.total_mem_used * ratio;
+ wg_mem_info.mem_used_ratio = (double)wg_mem_info.weighted_mem_used / wg_mem_limit;
+
+ wg.second->set_weighted_memory_used(wg_mem_info.total_mem_used, ratio);
+
+ auto spill_low_water_mark = wg.second->spill_threshold_low_water_mark();
+ auto spill_high_water_mark = wg.second->spill_threashold_high_water_mark();
+ wg_mem_info.is_high_wartermark = (wg_mem_info.weighted_mem_used >
+ ((double)wg_mem_limit * spill_high_water_mark / 100));
+ wg_mem_info.is_low_wartermark = (wg_mem_info.weighted_mem_used >
+ ((double)wg_mem_limit * spill_low_water_mark / 100));
+
+ // calculate query weighted memory limit of task group
+ const auto& wg_queries = all_wg_queries[wg.first];
+ auto wg_query_count = wg_queries.size();
Review Comment:
这里如果query count < 1 感觉可以直接continue了;
那么216 行直接就是wg_mem_limit ) / wg_query_count
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542650160
##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,135 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
<< "ms, deleted group size:" << deleted_task_groups.size();
}
+struct WorkloadGroupMemInfo {
+ int64_t total_mem_used = 0;
+ int64_t weighted_mem_used = 0;
+ bool is_low_wartermark = false;
+ bool is_high_wartermark = false;
+ double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {
+ std::shared_lock<std::shared_mutex> r_lock(_group_mutex);
+ // workload group id -> workload group queries
+ std::unordered_map<uint64_t, std::unordered_map<TUniqueId, std::weak_ptr<QueryContext>>>
+ all_wg_queries;
+ for (auto& [wg_id, wg] : _workload_groups) {
+ all_wg_queries.insert({wg_id, wg->queries()});
+ }
+
+ int64_t all_queries_mem_used = 0;
+
+ // calculate total memory used of each workload group and total memory used of all queries
+ std::unordered_map<uint64_t, WorkloadGroupMemInfo> wgs_mem_info;
+ for (auto& [wg_id, wg_queries] : all_wg_queries) {
+ int64_t wg_total_mem_used = 0;
+ for (const auto& [query_id, query_ctx_ptr] : wg_queries) {
+ if (auto query_ctx = query_ctx_ptr.lock()) {
+ wg_total_mem_used +=
+ query_ctx->is_cancelled() ? 0 : query_ctx->query_mem_tracker->consumption();
+ }
+ }
+ all_queries_mem_used += wg_total_mem_used;
+ wgs_mem_info[wg_id] = {wg_total_mem_used};
+ }
+
+ auto proc_vm_rss = PerfCounters::get_vm_rss();
+ if (all_queries_mem_used <= 0) {
+ return;
+ }
+
+ auto process_mem_used = doris::MemInfo::proc_mem_no_allocator_cache();
+ auto sys_mem_available = doris::MemInfo::sys_mem_available();
+ std::string debug_msg = fmt::format(
+ "\nProcess Memory Summary: process_vm_rss: {}, process mem: {}, sys mem available: "
+ "{}, all quries mem: {}",
+ PrettyPrinter::print(proc_vm_rss, TUnit::BYTES),
+ PrettyPrinter::print(process_mem_used, TUnit::BYTES),
+ PrettyPrinter::print(sys_mem_available, TUnit::BYTES),
+ PrettyPrinter::print(all_queries_mem_used, TUnit::BYTES));
+ LOG_EVERY_N(INFO, 5) << debug_msg;
Review Comment:
这个日志感觉也会刷屏,如果内存不到80% 就别输出了。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026778442
<details>
<summary>TPC-DS: <b>Total hot run time: 182069 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c8f6c475195c6fa5512777393f6824eb63ce1a44, data reload: false
query1 1199 1112 1096 1096
query2 6352 2065 1975 1975
query3 6696 220 218 218
query4 34666 21377 21436 21377
query5 4268 375 373 373
query6 266 182 181 181
query7 4635 302 291 291
query8 215 181 177 177
query9 9790 2212 2202 2202
query10 447 264 243 243
query11 17133 14228 14228 14228
query12 124 83 85 83
query13 1623 421 416 416
query14 11057 7042 7040 7040
query15 207 170 171 170
query16 6823 250 271 250
query17 1702 592 555 555
query18 1390 292 278 278
query19 244 152 158 152
query20 90 86 84 84
query21 199 129 125 125
query22 4978 4794 4738 4738
query23 33462 32712 33024 32712
query24 11309 2882 2959 2882
query25 646 406 411 406
query26 1155 157 160 157
query27 3318 373 390 373
query28 7946 1878 1942 1878
query29 916 691 652 652
query30 305 152 150 150
query31 1014 756 732 732
query32 85 57 54 54
query33 706 263 248 248
query34 1250 507 516 507
query35 869 728 720 720
query36 1034 924 916 916
query37 110 64 67 64
query38 3701 3644 3634 3634
query39 1675 1588 1566 1566
query40 176 108 112 108
query41 50 57 45 45
query42 109 105 100 100
query43 508 461 471 461
query44 1344 759 757 757
query45 280 254 277 254
query46 1128 734 742 734
query47 2013 1928 1935 1928
query48 471 368 368 368
query49 883 312 344 312
query50 818 410 411 410
query51 6885 6768 6892 6768
query52 106 89 103 89
query53 354 280 282 280
query54 279 234 225 225
query55 88 84 77 77
query56 238 217 213 213
query57 1328 1165 1197 1165
query58 234 201 206 201
query59 2958 2532 2641 2532
query60 239 230 223 223
query61 98 93 104 93
query62 611 446 442 442
query63 305 276 275 275
query64 5684 4076 3984 3984
query65 3088 3003 2985 2985
query66 1182 338 328 328
query67 15545 14738 14806 14738
query68 9838 527 528 527
query69 700 374 371 371
query70 1366 1147 1130 1130
query71 575 268 270 268
query72 7051 2565 2361 2361
query73 1566 326 331 326
query74 8146 6466 6345 6345
query75 5191 2246 2266 2246
query76 6477 903 927 903
query77 575 228 231 228
query78 11800 10480 10189 10189
query79 12092 526 519 519
query80 1889 353 347 347
query81 497 210 205 205
query82 168 82 82 82
query83 228 137 141 137
query84 279 77 77 77
query85 921 325 317 317
query86 353 286 291 286
query87 3654 3487 3502 3487
query88 6185 2314 2306 2306
query89 483 371 360 360
query90 2555 171 167 167
query91 171 153 134 134
query92 60 43 45 43
query93 6036 487 482 482
query94 1616 177 172 172
query95 408 291 303 291
query96 628 277 263 263
query97 2689 2497 2461 2461
query98 222 213 219 213
query99 1043 828 854 828
Total cold run time: 326069 ms
Total hot run time: 182069 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2017713206
<details>
<summary>TPC-H: <b>Total hot run time: 38070 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0e279b3fac84fecc43a2e6fc9829bd07ba245b4f, data reload: false
------ Round 1 ----------------------------------
q1 17609 4243 4132 4132
q2 2119 165 166 165
q3 10565 1109 1197 1109
q4 10231 748 799 748
q5 7458 2965 2995 2965
q6 204 123 123 123
q7 1037 613 567 567
q8 9333 2032 1950 1950
q9 7469 6644 6598 6598
q10 8487 3430 3593 3430
q11 432 227 214 214
q12 432 204 199 199
q13 17809 2889 2866 2866
q14 231 207 213 207
q15 515 476 460 460
q16 501 379 372 372
q17 963 648 538 538
q18 7175 6607 6545 6545
q19 1542 1446 1468 1446
q20 560 253 240 240
q21 3555 2903 2903 2903
q22 352 296 293 293
Total cold run time: 108579 ms
Total hot run time: 38070 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4107 4089 4099 4089
q2 342 232 233 232
q3 2980 2863 2805 2805
q4 1900 1546 1552 1546
q5 5343 5354 5336 5336
q6 195 115 116 115
q7 2218 1871 1863 1863
q8 3159 3277 3280 3277
q9 8715 8729 8694 8694
q10 3822 3790 3715 3715
q11 555 461 445 445
q12 714 530 548 530
q13 12652 2853 2874 2853
q14 285 273 257 257
q15 502 459 464 459
q16 482 428 421 421
q17 1735 1508 1481 1481
q18 7403 7190 7086 7086
q19 1615 1567 1553 1553
q20 1899 1754 1717 1717
q21 4803 4746 4768 4746
q22 534 454 439 439
Total cold run time: 65960 ms
Total hot run time: 53659 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2017733857
<details>
<summary>TPC-DS: <b>Total hot run time: 186969 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0e279b3fac84fecc43a2e6fc9829bd07ba245b4f, data reload: false
query1 936 377 352 352
query2 7359 2040 1915 1915
query3 6702 209 214 209
query4 31799 21232 21659 21232
query5 4343 419 409 409
query6 278 175 181 175
query7 4625 295 301 295
query8 228 169 174 169
query9 9298 2251 2265 2251
query10 576 248 250 248
query11 17279 14404 14418 14404
query12 139 85 87 85
query13 1642 424 425 424
query14 14907 11366 11591 11366
query15 317 194 214 194
query16 8209 263 261 261
query17 2076 595 558 558
query18 2108 289 287 287
query19 368 160 158 158
query20 94 89 90 89
query21 210 130 130 130
query22 4982 4751 4726 4726
query23 33556 32542 32546 32542
query24 10721 2898 2897 2897
query25 632 413 393 393
query26 1386 157 163 157
query27 2949 364 364 364
query28 7430 1914 1885 1885
query29 892 623 640 623
query30 306 155 155 155
query31 954 740 727 727
query32 95 58 58 58
query33 741 329 253 253
query34 910 489 494 489
query35 828 600 618 600
query36 1008 876 916 876
query37 116 78 78 78
query38 3527 3491 3451 3451
query39 1486 1402 1420 1402
query40 224 119 112 112
query41 48 49 47 47
query42 103 96 100 96
query43 470 449 462 449
query44 1154 737 745 737
query45 276 272 265 265
query46 1129 704 700 700
query47 1917 1861 1862 1861
query48 461 356 365 356
query49 1110 361 338 338
query50 776 375 372 372
query51 6803 6748 6758 6748
query52 98 97 90 90
query53 350 275 271 271
query54 324 238 245 238
query55 81 78 80 78
query56 246 229 230 229
query57 1204 1123 1136 1123
query58 226 199 201 199
query59 2855 2653 2596 2596
query60 285 244 253 244
query61 98 107 99 99
query62 670 458 444 444
query63 307 278 283 278
query64 5407 4070 4034 4034
query65 3103 3016 3036 3016
query66 1480 376 355 355
query67 15361 14828 14675 14675
query68 7045 528 515 515
query69 617 380 382 380
query70 1251 1175 1146 1146
query71 531 288 274 274
query72 6248 2720 2542 2542
query73 755 321 311 311
query74 8543 6640 6736 6640
query75 4116 2784 2911 2784
query76 5002 902 884 884
query77 652 300 270 270
query78 10776 10286 10080 10080
query79 6889 521 527 521
query80 1316 422 419 419
query81 514 214 221 214
query82 364 213 208 208
query83 226 150 152 150
query84 298 87 85 85
query85 1142 398 367 367
query86 373 307 306 306
query87 3735 3554 3523 3523
query88 4813 2424 2414 2414
query89 475 376 364 364
query90 2088 178 180 178
query91 172 136 143 136
query92 65 49 47 47
query93 5743 493 483 483
query94 1220 179 176 176
query95 430 324 332 324
query96 620 273 280 273
query97 3059 2901 2872 2872
query98 232 218 222 218
query99 1151 930 928 928
Total cold run time: 313039 ms
Total hot run time: 186969 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542290286
##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -326,6 +304,46 @@ Status PipelineXTask::execute(bool* eos) {
return status;
}
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+ auto* query_ctx = state->get_query_ctx();
+ bool need_revoke = query_ctx->query_mem_tracker->need_revoke();
+ if (!need_revoke) {
+ return false;
+ }
+
+ const auto min_revocable_mem_bytes = state->min_revocable_mem();
+
+ auto tg = query_ctx->workload_group();
+ bool is_tg_mem_low_water_mark = false;
+ bool is_tg_mem_high_water_mark = false;
Review Comment:
use wg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542293320
##########
be/src/runtime/memory/mem_tracker_limiter.h:
##########
@@ -244,6 +245,20 @@ class MemTrackerLimiter final : public MemTracker {
// Log the memory usage when memory limit is exceeded.
std::string tracker_limit_exceeded_str();
+ void set_weighted_mem(int64_t weighted_limit, double ratio) {
Review Comment:
这个不要在这里搞。
这个信息应该是要记录到query context 中的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026354103
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024907750
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #32641:
URL: https://github.com/apache/doris/pull/32641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1534070171
##########
be/src/vec/exprs/vexpr.cpp:
##########
@@ -97,11 +97,11 @@ TExprNode create_texpr_node_from(const void* data, const PrimitiveType& type, in
break;
}
case TYPE_DATEV2: {
- THROW_IF_ERROR(create_texpr_literal_node<TYPE_DATEV2>(data, &node));
+ THROW_IF_ERROR(create_texpr_literal_node<TYPE_DATEV2>(data, &node, precision, scale));
Review Comment:
These code is not related with this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542283442
##########
be/src/pipeline/exec/partitioned_aggregation_source_operator.cpp:
##########
@@ -128,6 +129,7 @@ Status PartitionedAggSourceOperatorX::close(RuntimeState* state) {
Status PartitionedAggSourceOperatorX::get_block(RuntimeState* state, vectorized::Block* block,
bool* eos) {
auto& local_state = get_local_state(state);
+ local_state.inc_running_big_mem_op_num(state);
Review Comment:
why do we need check local state's count.
Only operator is ok?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542541083
##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -326,6 +304,47 @@ Status PipelineXTask::execute(bool* eos) {
return status;
}
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+ auto* query_ctx = state->get_query_ctx();
+ bool need_revoke = query_ctx->need_revoke();
+ if (!need_revoke) {
+ return false;
+ }
+
+ const auto min_revocable_mem_bytes = state->min_revocable_mem();
+
+ auto wg = query_ctx->workload_group();
+ if (!wg) {
+ LOG_ONCE(INFO) << "no workload group for query " << print_id(state->query_id());
+ return false;
+ }
+ bool is_wg_mem_low_water_mark = false;
+ bool is_wg_mem_high_water_mark = false;
+ wg->check_mem_used(&is_wg_mem_low_water_mark, &is_wg_mem_high_water_mark);
+ if (is_wg_mem_high_water_mark) {
+ if (revocable_mem_bytes > 0) {
+ LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
+ return true;
Review Comment:
warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:324:
```diff
- if (revocable_mem_bytes > 0) {
- LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
- return true;
- }
- return false;
+ return revocable_mem_bytes > 0;
```
##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,135 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
<< "ms, deleted group size:" << deleted_task_groups.size();
}
+struct WorkloadGroupMemInfo {
+ int64_t total_mem_used = 0;
+ int64_t weighted_mem_used = 0;
+ bool is_low_wartermark = false;
+ bool is_high_wartermark = false;
+ double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {
Review Comment:
warning: function 'refresh_wg_memory_info' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void WorkloadGroupMgr::refresh_wg_memory_info() {
^
```
<details>
<summary>Additional context</summary>
**be/src/runtime/workload_group/workload_group_manager.cpp:146:** 120 lines including whitespace and comments (threshold 80)
```cpp
void WorkloadGroupMgr::refresh_wg_memory_info() {
^
```
</details>
##########
be/src/runtime/query_context.h:
##########
@@ -247,6 +247,32 @@ class QueryContext {
bool is_nereids() const { return _is_nereids; }
+ WorkloadGroupPtr workload_group() const { return _workload_group; }
+
+ void inc_running_big_mem_op_num() {
+ _running_big_mem_op_num.fetch_add(1, std::memory_order_relaxed);
+ }
+ void dec_running_big_mem_op_num() {
+ _running_big_mem_op_num.fetch_sub(1, std::memory_order_relaxed);
+ }
+ int32_t get_running_big_mem_op_num() {
+ return _running_big_mem_op_num.load(std::memory_order_relaxed);
+ }
+
+ void set_weighted_mem(int64_t weighted_limit, int64_t weighted_consumption) {
+ std::lock_guard<std::mutex> l(_weighted_mem_lock);
+ _weighted_consumption = weighted_consumption;
+ _weighted_limit = weighted_limit;
+ }
+ void get_weighted_mem_info(int64_t& weighted_limit, int64_t& weighted_consumption) {
Review Comment:
warning: method 'get_weighted_mem_info' can be made const [readability-make-member-function-const]
```suggestion
void get_weighted_mem_info(int64_t& weighted_limit, int64_t& weighted_consumption) const {
```
##########
be/src/runtime/workload_group/workload_group.h:
##########
@@ -97,7 +121,7 @@
return _memory_limit > 0;
}
- Status add_query(TUniqueId query_id) {
+ Status add_query(TUniqueId query_id, std::shared_ptr<QueryContext> query_ctx) {
Review Comment:
warning: method 'add_query' can be made const [readability-make-member-function-const]
```suggestion
Status add_query(TUniqueId query_id, std::shared_ptr<QueryContext> query_ctx) const {
```
##########
be/src/runtime/workload_group/workload_group.h:
##########
@@ -82,6 +83,29 @@ class WorkloadGroup : public std::enable_shared_from_this<WorkloadGroup> {
int64_t memory_used();
+ int spill_threshold_low_water_mark() const {
+ return _spill_low_watermark.load(std::memory_order_relaxed);
+ }
+ int spill_threashold_high_water_mark() const {
+ return _spill_high_watermark.load(std::memory_order_relaxed);
+ }
+
+ void set_weighted_memory_used(int64_t wg_total_mem_used, double ratio);
+
+ int64_t get_weighted_memory_used() {
+ return _weighted_mem_used.load(std::memory_order_relaxed);
+ }
+
+ void check_mem_used(bool* is_low_wartermark, bool* is_high_wartermark) const {
Review Comment:
warning: method 'check_mem_used' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void check_mem_used(bool* is_low_wartermark, bool* is_high_wartermark) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542662525
##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,135 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
<< "ms, deleted group size:" << deleted_task_groups.size();
}
+struct WorkloadGroupMemInfo {
+ int64_t total_mem_used = 0;
+ int64_t weighted_mem_used = 0;
+ bool is_low_wartermark = false;
+ bool is_high_wartermark = false;
+ double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {
+ std::shared_lock<std::shared_mutex> r_lock(_group_mutex);
+ // workload group id -> workload group queries
+ std::unordered_map<uint64_t, std::unordered_map<TUniqueId, std::weak_ptr<QueryContext>>>
+ all_wg_queries;
+ for (auto& [wg_id, wg] : _workload_groups) {
+ all_wg_queries.insert({wg_id, wg->queries()});
+ }
+
+ int64_t all_queries_mem_used = 0;
+
+ // calculate total memory used of each workload group and total memory used of all queries
+ std::unordered_map<uint64_t, WorkloadGroupMemInfo> wgs_mem_info;
+ for (auto& [wg_id, wg_queries] : all_wg_queries) {
+ int64_t wg_total_mem_used = 0;
+ for (const auto& [query_id, query_ctx_ptr] : wg_queries) {
+ if (auto query_ctx = query_ctx_ptr.lock()) {
+ wg_total_mem_used +=
+ query_ctx->is_cancelled() ? 0 : query_ctx->query_mem_tracker->consumption();
+ }
+ }
+ all_queries_mem_used += wg_total_mem_used;
+ wgs_mem_info[wg_id] = {wg_total_mem_used};
+ }
+
+ auto proc_vm_rss = PerfCounters::get_vm_rss();
+ if (all_queries_mem_used <= 0) {
+ return;
+ }
+
+ auto process_mem_used = doris::MemInfo::proc_mem_no_allocator_cache();
+ auto sys_mem_available = doris::MemInfo::sys_mem_available();
+ std::string debug_msg = fmt::format(
+ "\nProcess Memory Summary: process_vm_rss: {}, process mem: {}, sys mem available: "
+ "{}, all quries mem: {}",
+ PrettyPrinter::print(proc_vm_rss, TUnit::BYTES),
+ PrettyPrinter::print(process_mem_used, TUnit::BYTES),
+ PrettyPrinter::print(sys_mem_available, TUnit::BYTES),
+ PrettyPrinter::print(all_queries_mem_used, TUnit::BYTES));
+ LOG_EVERY_N(INFO, 5) << debug_msg;
+
+ if (proc_vm_rss < all_queries_mem_used) {
+ all_queries_mem_used = proc_vm_rss;
+ }
+
+ // process memory used is actually bigger than all_queries_mem_used,
+ // because memory of page cache, allocator cache, segment cache etc. are included
+ // in process_mem_used.
+ // we count these cache memories equally on workload groups.
+ double ratio = (double)proc_vm_rss / (double)all_queries_mem_used;
+
+ for (auto& wg : _workload_groups) {
+ auto wg_mem_limit = wg.second->memory_limit();
+ auto& wg_mem_info = wgs_mem_info[wg.first];
+ wg_mem_info.weighted_mem_used = wg_mem_info.total_mem_used * ratio;
+ wg_mem_info.mem_used_ratio = (double)wg_mem_info.weighted_mem_used / wg_mem_limit;
+
+ wg.second->set_weighted_memory_used(wg_mem_info.total_mem_used, ratio);
+
+ auto spill_low_water_mark = wg.second->spill_threshold_low_water_mark();
+ auto spill_high_water_mark = wg.second->spill_threashold_high_water_mark();
+ wg_mem_info.is_high_wartermark = (wg_mem_info.weighted_mem_used >
+ ((double)wg_mem_limit * spill_high_water_mark / 100));
+ wg_mem_info.is_low_wartermark = (wg_mem_info.weighted_mem_used >
+ ((double)wg_mem_limit * spill_low_water_mark / 100));
+
+ // calculate query weighted memory limit of task group
+ const auto& wg_queries = all_wg_queries[wg.first];
+ auto wg_query_count = wg_queries.size();
+ int64_t query_weighted_mem_limit =
+ wg_query_count ? (wg_mem_limit + wg_query_count) / wg_query_count : wg_mem_limit;
+
+ debug_msg = "";
+ if (wg_mem_info.is_high_wartermark || wg_mem_info.is_low_wartermark) {
+ debug_msg = fmt::format(
+ "\nWorkload Group {}: mem limit: {}, mem used: {}, weighted mem used: {}, used "
+ "ratio: {}, query "
+ "count: {}, query_weighted_mem_limit: {}",
+ wg.second->name(), PrettyPrinter::print(wg_mem_limit, TUnit::BYTES),
+ PrettyPrinter::print(wg_mem_info.total_mem_used, TUnit::BYTES),
+ PrettyPrinter::print(wg_mem_info.weighted_mem_used, TUnit::BYTES),
+ wg_mem_info.mem_used_ratio, wg_query_count,
+ PrettyPrinter::print(query_weighted_mem_limit, TUnit::BYTES));
+
+ debug_msg += "\n Query Memory Summary:";
+ }
+ // check where queries need to revoke memory for task group
+ for (const auto& query : wg_queries) {
+ auto query_ctx = query.second.lock();
+ if (!query_ctx) {
+ continue;
+ }
+ auto query_consumption = query_ctx->query_mem_tracker->consumption();
+ int64_t query_weighted_consumption = query_consumption * ratio;
+ query_ctx->set_weighted_mem(query_weighted_mem_limit, query_weighted_consumption);
+
+ bool need_revoke = false;
+ if (wg_mem_info.is_high_wartermark) {
+ need_revoke = true;
+ } else if (wg_mem_info.is_low_wartermark) {
+ need_revoke = query_weighted_consumption > query_weighted_mem_limit;
+ }
+ query_ctx->set_need_revoke(need_revoke);
+
+ if (wg_mem_info.is_high_wartermark || wg_mem_info.is_low_wartermark) {
+ debug_msg += fmt::format(
+ "\n MemTracker Label={}, Parent Label={}, Used={}, WeightedUsed={}, "
+ "Peak={}",
+ query_ctx->query_mem_tracker->label(),
+ query_ctx->query_mem_tracker->parent_label(),
+ PrettyPrinter::print(query_consumption, TUnit::BYTES),
+ PrettyPrinter::print(query_weighted_consumption, TUnit::BYTES),
+ PrettyPrinter::print(query_ctx->query_mem_tracker->peak_consumption(),
+ TUnit::BYTES));
+ }
+ }
+ if (wg_mem_info.is_high_wartermark || wg_mem_info.is_low_wartermark) {
+ LOG_EVERY_N(INFO, 3) << debug_msg;
Review Comment:
改成10 吧
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542293516
##########
be/src/runtime/memory/mem_tracker_limiter.h:
##########
@@ -244,6 +245,20 @@ class MemTrackerLimiter final : public MemTracker {
// Log the memory usage when memory limit is exceeded.
std::string tracker_limit_exceeded_str();
+ void set_weighted_mem(int64_t weighted_limit, double ratio) {
+ std::lock_guard<std::mutex> l(_weighted_mem_lock);
+ _weighted_consumption = _consumption->current_value() * ratio;
+ _weighted_limit = weighted_limit;
+ }
+ void get_weighted_mem_info(int64_t& weighted_limit, int64_t& weighted_consumption) {
+ std::lock_guard<std::mutex> l(_weighted_mem_lock);
+ weighted_limit = _weighted_limit;
+ weighted_consumption = _weighted_consumption;
+ }
+
+ void set_need_revoke(bool b) { _need_revoke.store(b, std::memory_order_relaxed); }
Review Comment:
这些函数应该都是要再query ctx 中
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026869495
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1534078055
##########
be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:
##########
@@ -462,6 +472,10 @@ Status PartitionedHashJoinProbeOperatorX::open(RuntimeState* state) {
Status PartitionedHashJoinProbeOperatorX::push(RuntimeState* state, vectorized::Block* input_block,
bool eos) const {
auto& local_state = get_local_state(state);
+ if (!local_state._is_running) {
Review Comment:
You could implement 2 method in parent class:
inc_running_big_mem_op_num(){
if (not add) {
get_query_ctx()->inc_running_big_mem_op_num;
not_add=true;
}
}
dec_big_mem_op_num(){
if (not_add) {
log_fatal<<} else if (!_dec) {
query_ctx->dec;
_dec=true;
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2017575653
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1534078055
##########
be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:
##########
@@ -462,6 +472,10 @@ Status PartitionedHashJoinProbeOperatorX::open(RuntimeState* state) {
Status PartitionedHashJoinProbeOperatorX::push(RuntimeState* state, vectorized::Block* input_block,
bool eos) const {
auto& local_state = get_local_state(state);
+ if (!local_state._is_running) {
Review Comment:
You could implement 2 method in parent class:
inc_running_big_mem_op_num(){
if (not add) {
get_query_ctx()->inc_running_big_mem_op_num;
not_add=true;
}
}
dec_big_mem_op_num(){
if (not_add) {
log_fatal<<} else if (!_dec) {
query_ctx->dec;
_dec=true;
}
}
then all subclass could use this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2021926523
<details>
<summary>TPC-H: <b>Total hot run time: 38470 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e1783354448196d335626f926218759a3286ec2e, data reload: false
------ Round 1 ----------------------------------
q1 18359 4551 4252 4252
q2 2595 166 177 166
q3 11115 1158 1219 1158
q4 10864 825 849 825
q5 7774 3081 3066 3066
q6 218 129 128 128
q7 1078 643 588 588
q8 9590 2073 2042 2042
q9 7348 6672 6573 6573
q10 8380 3424 3574 3424
q11 418 233 217 217
q12 369 211 203 203
q13 17795 2904 2870 2870
q14 241 203 200 200
q15 505 460 454 454
q16 470 381 373 373
q17 967 529 626 529
q18 7174 6528 6574 6528
q19 1560 1434 1492 1434
q20 545 249 247 247
q21 3648 2978 2909 2909
q22 356 284 292 284
Total cold run time: 111369 ms
Total hot run time: 38470 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4114 4084 4107 4084
q2 328 227 231 227
q3 2983 2860 2961 2860
q4 1902 1593 1600 1593
q5 5331 5346 5356 5346
q6 194 115 117 115
q7 2288 1856 1885 1856
q8 3175 3292 3331 3292
q9 8810 8731 8788 8731
q10 3830 3793 3798 3793
q11 543 446 449 446
q12 714 544 541 541
q13 16567 2865 2880 2865
q14 279 248 270 248
q15 499 459 462 459
q16 480 429 430 429
q17 1747 1526 1496 1496
q18 7528 7201 7244 7201
q19 1623 1548 1511 1511
q20 1891 1737 1702 1702
q21 4893 4804 4733 4733
q22 548 430 427 427
Total cold run time: 70267 ms
Total hot run time: 53955 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2021945104
TeamCity be ut coverage result:
Function Coverage: 35.22% (8741/24816)
Line Coverage: 27.01% (71551/264859)
Region Coverage: 26.27% (37132/141340)
Branch Coverage: 23.16% (18982/81952)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e1783354448196d335626f926218759a3286ec2e_e1783354448196d335626f926218759a3286ec2e/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2021945803
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit e1783354448196d335626f926218759a3286ec2e with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.7 seconds inserted 10000000 Rows, about 729K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542290653
##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -326,6 +304,46 @@ Status PipelineXTask::execute(bool* eos) {
return status;
}
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+ auto* query_ctx = state->get_query_ctx();
+ bool need_revoke = query_ctx->query_mem_tracker->need_revoke();
+ if (!need_revoke) {
+ return false;
+ }
+
+ const auto min_revocable_mem_bytes = state->min_revocable_mem();
+
+ auto tg = query_ctx->workload_group();
+ bool is_tg_mem_low_water_mark = false;
+ bool is_tg_mem_high_water_mark = false;
+ if (tg) {
+ tg->check_mem_used(is_tg_mem_low_water_mark, is_tg_mem_high_water_mark);
+ }
+ if (is_tg_mem_high_water_mark) {
+ if (revocable_mem_bytes > 1024 * 1024) {
Review Comment:
change this to an session variable.
not use magic numbers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024274053
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024884658
<details>
<summary>TPC-DS: <b>Total hot run time: 180802 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 09207c21e85cb9bb7da144b2960060fec2a8e348, data reload: false
query1 947 363 353 353
query2 6555 1840 1919 1840
query3 6704 210 207 207
query4 31834 21276 21273 21273
query5 4319 413 385 385
query6 273 177 172 172
query7 4624 284 293 284
query8 226 167 173 167
query9 9413 2351 2349 2349
query10 572 246 255 246
query11 15396 14080 14224 14080
query12 135 97 87 87
query13 1641 426 411 411
query14 9458 7670 7935 7670
query15 261 187 185 185
query16 8232 265 269 265
query17 1958 552 527 527
query18 2095 274 282 274
query19 353 147 150 147
query20 91 85 88 85
query21 209 126 121 121
query22 4992 4829 4775 4775
query23 33739 32940 32803 32803
query24 11018 2811 2888 2811
query25 588 360 367 360
query26 1189 146 148 146
query27 2978 344 349 344
query28 7646 1948 1921 1921
query29 862 632 605 605
query30 300 151 162 151
query31 999 704 736 704
query32 90 58 51 51
query33 747 251 232 232
query34 1043 482 499 482
query35 820 612 599 599
query36 1014 888 877 877
query37 127 63 61 61
query38 3528 3389 3417 3389
query39 1468 1461 1432 1432
query40 218 109 110 109
query41 48 47 45 45
query42 100 98 93 93
query43 489 438 439 438
query44 1203 710 715 710
query45 282 255 266 255
query46 1094 704 695 695
query47 1899 1821 1840 1821
query48 447 366 358 358
query49 1094 325 323 323
query50 766 366 372 366
query51 6651 6533 6549 6533
query52 105 96 93 93
query53 349 270 274 270
query54 286 221 226 221
query55 94 78 76 76
query56 247 213 214 213
query57 1211 1129 1119 1119
query58 226 196 203 196
query59 2638 2504 2472 2472
query60 266 228 264 228
query61 111 108 113 108
query62 643 404 434 404
query63 310 282 279 279
query64 5715 4018 4077 4018
query65 3137 3031 2998 2998
query66 1451 377 378 377
query67 15188 14777 14781 14777
query68 6800 532 533 532
query69 609 395 371 371
query70 1183 1184 1115 1115
query71 462 264 265 264
query72 6499 2698 2463 2463
query73 732 324 323 323
query74 7751 6429 6407 6407
query75 3456 2207 2219 2207
query76 4002 906 888 888
query77 602 269 256 256
query78 10836 10140 10218 10140
query79 8798 519 519 519
query80 1555 391 358 358
query81 532 218 219 218
query82 864 84 85 84
query83 219 143 139 139
query84 289 82 81 81
query85 1502 316 308 308
query86 468 320 285 285
query87 3741 3508 3513 3508
query88 5106 2378 2369 2369
query89 495 370 364 364
query90 1937 180 179 179
query91 173 137 141 137
query92 61 47 46 46
query93 6948 493 482 482
query94 1154 175 175 175
query95 430 321 326 321
query96 599 274 275 274
query97 2630 2477 2466 2466
query98 237 214 212 212
query99 1257 845 855 845
Total cold run time: 306639 ms
Total hot run time: 180802 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024992428
<details>
<summary>TPC-DS: <b>Total hot run time: 180902 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit df0ddde4d7f8ba7db99389a85bb281abe6306fe4, data reload: false
query1 947 367 361 361
query2 6553 1960 1937 1937
query3 6705 208 204 204
query4 31963 21304 21329 21304
query5 4318 399 402 399
query6 270 180 173 173
query7 4648 293 290 290
query8 229 170 168 168
query9 9364 2319 2330 2319
query10 560 253 262 253
query11 17206 14160 14161 14160
query12 138 95 93 93
query13 1626 421 420 420
query14 9248 7587 7144 7144
query15 219 186 179 179
query16 8033 255 259 255
query17 1959 557 536 536
query18 2051 294 281 281
query19 235 148 146 146
query20 91 90 85 85
query21 200 123 123 123
query22 4979 4785 4768 4768
query23 33557 33171 32684 32684
query24 12097 2899 2845 2845
query25 665 400 390 390
query26 1841 153 153 153
query27 3184 320 314 314
query28 8179 1917 1900 1900
query29 1056 649 614 614
query30 303 152 147 147
query31 987 730 743 730
query32 95 58 58 58
query33 763 267 262 262
query34 1114 487 507 487
query35 920 700 683 683
query36 1044 881 915 881
query37 268 69 65 65
query38 3553 3443 3433 3433
query39 1575 1571 1515 1515
query40 283 106 112 106
query41 52 50 45 45
query42 103 92 98 92
query43 494 454 436 436
query44 1262 708 716 708
query45 268 266 239 239
query46 1096 706 687 687
query47 1944 1839 1839 1839
query48 451 359 358 358
query49 1232 333 345 333
query50 767 374 374 374
query51 6693 6638 6560 6560
query52 120 93 99 93
query53 341 274 276 274
query54 318 237 233 233
query55 84 83 80 80
query56 262 238 235 235
query57 1238 1116 1122 1116
query58 230 207 212 207
query59 2880 2565 2561 2561
query60 264 250 263 250
query61 123 114 114 114
query62 692 436 438 436
query63 297 272 275 272
query64 6908 4119 4107 4107
query65 3079 3017 3025 3017
query66 1417 379 369 369
query67 15660 14902 14783 14783
query68 9214 534 541 534
query69 708 376 377 376
query70 1420 1142 1194 1142
query71 524 267 261 261
query72 6525 2530 2380 2380
query73 1625 330 324 324
query74 8143 6467 6438 6438
query75 3895 2271 2206 2206
query76 5682 864 848 848
query77 646 251 254 251
query78 11020 10259 10058 10058
query79 10681 516 517 516
query80 1859 371 375 371
query81 509 213 227 213
query82 322 85 85 85
query83 216 144 143 143
query84 291 77 79 77
query85 1074 308 297 297
query86 356 305 300 300
query87 3742 3626 3536 3536
query88 4760 2349 2345 2345
query89 486 358 372 358
query90 2007 175 171 171
query91 178 141 137 137
query92 64 48 47 47
query93 6890 508 487 487
query94 1332 177 177 177
query95 424 326 321 321
query96 640 264 266 264
query97 2633 2515 2493 2493
query98 236 222 213 213
query99 1121 861 850 850
Total cold run time: 320524 ms
Total hot run time: 180902 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2017739697
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 0e279b3fac84fecc43a2e6fc9829bd07ba245b4f with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 22.0 seconds inserted 10000000 Rows, about 454K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1537294093
##########
be/src/runtime/workload_group/workload_group.h:
##########
@@ -82,6 +82,31 @@ class WorkloadGroup : public std::enable_shared_from_this<WorkloadGroup> {
int64_t memory_used();
+ int spill_threshold_low_water_mark() const {
+ return _spill_low_watermark.load(std::memory_order_relaxed);
+ }
+ int spill_threashold_high_water_mark() const {
+ return _spill_high_watermark.load(std::memory_order_relaxed);
+ }
+
+ void set_weighted_memory_used(int64_t wg_total_mem_used, double ratio);
+
+ std::unordered_set<std::shared_ptr<MemTrackerLimiter>> get_all_mem_trackers();
+
+ int64_t get_weighted_memory_used() {
+ return _weighted_mem_used.load(std::memory_order_relaxed);
+ }
+
+ void check_mem_used(bool& is_low_wartermark, bool& is_high_wartermark) {
Review Comment:
warning: method 'check_mem_used' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void check_mem_used(bool& is_low_wartermark, bool& is_high_wartermark) {
```
##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,121 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
<< "ms, deleted group size:" << deleted_task_groups.size();
}
+struct WorkloadGroupMemInfo {
+ int64_t total_mem_used = 0;
+ int64_t weighted_mem_used = 0;
+ bool is_low_wartermark = false;
+ bool is_high_wartermark = false;
+ double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {
Review Comment:
warning: method 'refresh_wg_memory_info' can be made static [readability-convert-member-functions-to-static]
be/src/runtime/workload_group/workload_group_manager.h:56:
```diff
- void refresh_wg_memory_info();
+ static void refresh_wg_memory_info();
```
##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -326,6 +304,46 @@ Status PipelineXTask::execute(bool* eos) {
return status;
}
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+ auto* query_ctx = state->get_query_ctx();
+ bool need_revoke = query_ctx->query_mem_tracker->need_revoke();
+ if (!need_revoke) {
+ return false;
+ }
+
+ const auto min_revocable_mem_bytes = state->min_revocable_mem();
+
+ auto tg = query_ctx->workload_group();
+ bool is_tg_mem_low_water_mark = false;
+ bool is_tg_mem_high_water_mark = false;
+ if (tg) {
+ tg->check_mem_used(is_tg_mem_low_water_mark, is_tg_mem_high_water_mark);
+ }
+ if (is_tg_mem_high_water_mark) {
+ if (revocable_mem_bytes > 1024 * 1024) {
+ LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
+ return true;
Review Comment:
warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:322:
```diff
- if (revocable_mem_bytes > 1024 * 1024) {
- LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
- return true;
- }
- return false;
+ return revocable_mem_bytes > 1024 * 1024;
```
##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,121 @@
<< "ms, deleted group size:" << deleted_task_groups.size();
}
+struct WorkloadGroupMemInfo {
+ int64_t total_mem_used = 0;
+ int64_t weighted_mem_used = 0;
+ bool is_low_wartermark = false;
+ bool is_high_wartermark = false;
+ double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {
Review Comment:
warning: function 'refresh_wg_memory_info' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void WorkloadGroupMgr::refresh_wg_memory_info() {
^
```
<details>
<summary>Additional context</summary>
**be/src/runtime/workload_group/workload_group_manager.cpp:146:** 106 lines including whitespace and comments (threshold 80)
```cpp
void WorkloadGroupMgr::refresh_wg_memory_info() {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1534069010
##########
be/src/vec/spill/spill_writer.cpp:
##########
@@ -35,13 +35,19 @@ Status SpillWriter::open() {
return Status::OK();
}
+SpillWriter::~SpillWriter() {
+ if (!closed_) {
+ LOG(WARNING) << "spill writer not closed correctly:\n" << boost::stacktrace::stacktrace();
Review Comment:
not use this way, boost has a static global lock.
Using doris::Status wrapped error stack, you could use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2025041379
TeamCity be ut coverage result:
Function Coverage: 35.59% (8837/24827)
Line Coverage: 27.30% (72396/265177)
Region Coverage: 26.54% (37555/141484)
Branch Coverage: 23.35% (19151/82034)
Coverage Report: http://coverage.selectdb-in.cc/coverage/df0ddde4d7f8ba7db99389a85bb281abe6306fe4_df0ddde4d7f8ba7db99389a85bb281abe6306fe4/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024336558
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 615fff13fc8d598fd9aad875a0fb730eabc282d0 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.8 seconds inserted 10000000 Rows, about 724K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542352340
##########
be/src/runtime/workload_group/workload_group.h:
##########
@@ -141,7 +166,8 @@ class WorkloadGroup : public std::enable_shared_from_this<WorkloadGroup> {
const uint64_t _id;
std::string _name;
int64_t _version;
- int64_t _memory_limit; // bytes
+ int64_t _memory_limit; // bytes
+ std::atomic_int64_t _weighted_mem_used = 0; // bytes
bool _enable_memory_overcommit;
std::atomic<uint64_t> _cpu_share;
Review Comment:
remove query_id_set and use std::map<queryid, weak_ptr<query context>> so that all memory limit info could be set to query context, not to memtracker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) spill trigger improvement [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024897391
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 09207c21e85cb9bb7da144b2960060fec2a8e348 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.7 seconds inserted 10000000 Rows, about 729K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org