You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wangbo (via GitHub)" <gi...@apache.org> on 2024/03/20 10:06:16 UTC
[PR] [Improvement](executor)Add spill property for workload group [doris]
wangbo opened a new pull request, #32554:
URL: https://github.com/apache/doris/pull/32554
## Proposed changes
Add SPILL_THRESHOLD_LOW_WATERMARK/SPILL_THRESHOLD_HIGH_WATERMARK
```
mysql [information_schema]>desc workload_groups;
+--------------------------------+--------------+------+-------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------------------------+--------------+------+-------+---------+-------+
| ID | BIGINT | Yes | false | NULL | |
| NAME | VARCHAR(256) | Yes | false | NULL | |
| CPU_SHARE | BIGINT | Yes | false | NULL | |
| MEMORY_LIMIT | VARCHAR(256) | Yes | false | NULL | |
| ENABLE_MEMORY_OVERCOMMIT | VARCHAR(256) | Yes | false | NULL | |
| MAX_CONCURRENCY | BIGINT | Yes | false | NULL | |
| MAX_QUEUE_SIZE | BIGINT | Yes | false | NULL | |
| QUEUE_TIMEOUT | BIGINT | Yes | false | NULL | |
| CPU_HARD_LIMIT | VARCHAR(256) | Yes | false | NULL | |
| SCAN_THREAD_NUM | BIGINT | Yes | false | NULL | |
| MAX_REMOTE_SCAN_THREAD_NUM | BIGINT | Yes | false | NULL | |
| MIN_REMOTE_SCAN_THREAD_NUM | BIGINT | Yes | false | NULL | |
| SPILL_THRESHOLD_LOW_WATERMARK | VARCHAR(256) | Yes | false | NULL | |
| SPILL_THRESHOLD_HIGH_WATERMARK | VARCHAR(256) | Yes | false | NULL | |
+--------------------------------+--------------+------+-------+---------+-------+
14 rows in set (0.01 sec)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32554:
URL: https://github.com/apache/doris/pull/32554#discussion_r1532284786
##########
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##########
@@ -120,6 +128,20 @@ private WorkloadGroup(long id, String name, Map<String, String> properties, long
this.cpuHardLimit = Integer.parseInt(cpuHardLimitStr);
this.properties.put(CPU_HARD_LIMIT, cpuHardLimitStr);
}
+ if (properties.containsKey(SPILL_THRESHOLD_LOW_WATERMARK)) {
+ String lowWatermarkStr = properties.get(SPILL_THRESHOLD_LOW_WATERMARK);
+ if (lowWatermarkStr.endsWith("%")) {
+ lowWatermarkStr = lowWatermarkStr.substring(0, lowWatermarkStr.length() - 1);
+ }
+ this.properties.put(SPILL_THRESHOLD_LOW_WATERMARK, lowWatermarkStr);
+ }
+ if (properties.containsKey(SPILL_THRESHOLD_HIGH_WATERMARK)) {
+ String highWatermarkStr = properties.get(SPILL_THRESHOLD_HIGH_WATERMARK);
+ if (highWatermarkStr.endsWith("%")) {
Review Comment:
check it could be parsed to int
and the high water mark > low water mark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2012187468
<details>
<summary>TPC-DS: <b>Total hot run time: 181202 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9352fc88715d81519e6f2e27b8e3a3b23d4aea03, data reload: false
query1 942 361 347 347
query2 7433 2045 2032 2032
query3 6707 214 214 214
query4 31355 20820 20760 20760
query5 4387 414 394 394
query6 266 182 183 182
query7 4620 294 280 280
query8 240 163 184 163
query9 9896 2227 2230 2227
query10 475 242 249 242
query11 17178 14320 14231 14231
query12 138 91 84 84
query13 1625 458 426 426
query14 10635 10248 11167 10248
query15 265 208 195 195
query16 8126 256 253 253
query17 1956 551 517 517
query18 2062 286 271 271
query19 237 147 162 147
query20 94 96 81 81
query21 199 135 125 125
query22 4670 4469 4476 4469
query23 31789 31096 30877 30877
query24 10338 2846 2875 2846
query25 588 375 391 375
query26 723 152 163 152
query27 2241 354 360 354
query28 6324 1873 1905 1873
query29 876 658 631 631
query30 304 146 147 146
query31 976 725 731 725
query32 102 59 58 58
query33 664 259 256 256
query34 873 471 478 471
query35 808 602 611 602
query36 991 851 863 851
query37 106 75 76 75
query38 3553 3443 3437 3437
query39 1407 1411 1370 1370
query40 215 119 114 114
query41 51 47 47 47
query42 107 95 98 95
query43 489 445 460 445
query44 1084 754 716 716
query45 280 254 251 251
query46 1090 686 699 686
query47 1665 1617 1599 1599
query48 456 368 362 362
query49 1092 330 328 328
query50 781 370 372 370
query51 6774 6787 6738 6738
query52 105 93 92 92
query53 341 276 275 275
query54 322 257 259 257
query55 83 79 75 75
query56 246 253 234 234
query57 1073 1010 1011 1010
query58 231 205 213 205
query59 2715 2532 2621 2532
query60 355 259 279 259
query61 118 119 128 119
query62 573 410 390 390
query63 300 284 273 273
query64 5354 3875 3939 3875
query65 3039 3021 2979 2979
query66 851 349 358 349
query67 14904 14609 14322 14322
query68 7618 542 545 542
query69 641 394 414 394
query70 1181 1173 1179 1173
query71 511 282 280 280
query72 6410 2646 2502 2502
query73 742 317 320 317
query74 6875 6546 6561 6546
query75 4592 3010 2958 2958
query76 4967 838 941 838
query77 656 263 254 254
query78 10437 9588 9583 9583
query79 10673 516 537 516
query80 1850 406 392 392
query81 502 216 211 211
query82 822 201 206 201
query83 213 147 144 144
query84 288 81 78 78
query85 1282 329 317 317
query86 417 319 286 286
query87 3770 3631 3570 3570
query88 4879 2354 2362 2354
query89 506 355 362 355
query90 1973 172 171 171
query91 165 135 131 131
query92 57 47 46 46
query93 7158 503 491 491
query94 1213 181 178 178
query95 436 328 327 327
query96 616 275 270 270
query97 3068 2844 2842 2842
query98 236 207 206 206
query99 1102 750 745 745
Total cold run time: 306719 ms
Total hot run time: 181202 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on code in PR #32554:
URL: https://github.com/apache/doris/pull/32554#discussion_r1533149083
##########
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##########
@@ -120,6 +128,20 @@ private WorkloadGroup(long id, String name, Map<String, String> properties, long
this.cpuHardLimit = Integer.parseInt(cpuHardLimitStr);
this.properties.put(CPU_HARD_LIMIT, cpuHardLimitStr);
}
+ if (properties.containsKey(SPILL_THRESHOLD_LOW_WATERMARK)) {
+ String lowWatermarkStr = properties.get(SPILL_THRESHOLD_LOW_WATERMARK);
+ if (lowWatermarkStr.endsWith("%")) {
+ lowWatermarkStr = lowWatermarkStr.substring(0, lowWatermarkStr.length() - 1);
Review Comment:
Value check in WorkloadGroup.checkProperties
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2012059362
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2009185588
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32554:
URL: https://github.com/apache/doris/pull/32554#discussion_r1532292883
##########
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##########
@@ -284,6 +306,51 @@ private static void checkProperties(Map<String, String> properties) throws DdlEx
throw new DdlException(QUEUE_TIMEOUT + " requires a positive integer");
}
}
+
+ int lowWaterMark = SPILL_LOW_WATERMARK_DEFAULT_VALUE;
+ if (properties.containsKey(SPILL_THRESHOLD_LOW_WATERMARK)) {
+ String lowVal = properties.get(SPILL_THRESHOLD_LOW_WATERMARK);
+ if (lowVal.endsWith("%")) {
+ lowVal = lowVal.substring(0, lowVal.length() - 1);
+ }
+ try {
+ int intValue = Integer.parseInt(lowVal);
+ if ((intValue < 1 || intValue > 100) && intValue != -1) {
+ throw new NumberFormatException();
+ }
+ lowWaterMark = intValue;
+ } catch (NumberFormatException e) {
+ throw new DdlException(
+ SPILL_THRESHOLD_LOW_WATERMARK
+ + " must be a positive integer(1 ~ 100) or -1. but input value is "
Review Comment:
remove -1, it should always between [1,100]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2011097430
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32554:
URL: https://github.com/apache/doris/pull/32554#discussion_r1532295846
##########
be/src/runtime/workload_group/workload_group.cpp:
##########
@@ -44,6 +44,8 @@ const static std::string MEMORY_LIMIT_DEFAULT_VALUE = "0%";
const static bool ENABLE_MEMORY_OVERCOMMIT_DEFAULT_VALUE = true;
const static int CPU_HARD_LIMIT_DEFAULT_VALUE = -1;
const static uint64_t CPU_SOFT_LIMIT_DEFAULT_VALUE = 1024;
+const static int SPILL_LOW_WATERMARK_DEFAULT_VALUE = -1;
Review Comment:
default value to 50
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2011233907
TeamCity be ut coverage result:
Function Coverage: 35.26% (8728/24750)
Line Coverage: 27.07% (71426/263860)
Region Coverage: 26.31% (37069/140871)
Branch Coverage: 23.22% (18954/81618)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bf80271e87d72e1b6a7d9e66befb1db93e896c5d_bf80271e87d72e1b6a7d9e66befb1db93e896c5d/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2009672692
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2011176730
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2012193345
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 9352fc88715d81519e6f2e27b8e3a3b23d4aea03 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 21.9 seconds inserted 10000000 Rows, about 456K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2014494582
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2014494627
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32554:
URL: https://github.com/apache/doris/pull/32554#discussion_r1532291388
##########
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##########
@@ -67,13 +67,21 @@ public class WorkloadGroup implements Writable, GsonPostProcessable {
public static final String MIN_REMOTE_SCAN_THREAD_NUM = "min_remote_scan_thread_num";
+ public static final String SPILL_THRESHOLD_LOW_WATERMARK = "spill_threshold_low_watermark";
+
+ public static final String SPILL_THRESHOLD_HIGH_WATERMARK = "spill_threshold_high_watermark";
+
// NOTE(wb): all property is not required, some properties default value is set in be
// default value is as followed
// cpu_share=1024, memory_limit=0%(0 means not limit), enable_memory_overcommit=true
private static final ImmutableSet<String> ALL_PROPERTIES_NAME = new ImmutableSet.Builder<String>()
.add(CPU_SHARE).add(MEMORY_LIMIT).add(ENABLE_MEMORY_OVERCOMMIT).add(MAX_CONCURRENCY)
.add(MAX_QUEUE_SIZE).add(QUEUE_TIMEOUT).add(CPU_HARD_LIMIT).add(SCAN_THREAD_NUM)
- .add(MAX_REMOTE_SCAN_THREAD_NUM).add(MIN_REMOTE_SCAN_THREAD_NUM).build();
+ .add(MAX_REMOTE_SCAN_THREAD_NUM).add(MIN_REMOTE_SCAN_THREAD_NUM)
+ .add(SPILL_THRESHOLD_LOW_WATERMARK).add(SPILL_THRESHOLD_HIGH_WATERMARK).build();
+
+ public static final int SPILL_LOW_WATERMARK_DEFAULT_VALUE = -1;
Review Comment:
change default value to 50
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32554:
URL: https://github.com/apache/doris/pull/32554#discussion_r1532283550
##########
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##########
@@ -120,6 +128,20 @@ private WorkloadGroup(long id, String name, Map<String, String> properties, long
this.cpuHardLimit = Integer.parseInt(cpuHardLimitStr);
this.properties.put(CPU_HARD_LIMIT, cpuHardLimitStr);
}
+ if (properties.containsKey(SPILL_THRESHOLD_LOW_WATERMARK)) {
+ String lowWatermarkStr = properties.get(SPILL_THRESHOLD_LOW_WATERMARK);
+ if (lowWatermarkStr.endsWith("%")) {
+ lowWatermarkStr = lowWatermarkStr.substring(0, lowWatermarkStr.length() - 1);
Review Comment:
Has to check the value :
1. could be parsed to int
2. the value is between -1,100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2009185529
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2012050375
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2012235445
TeamCity be ut coverage result:
Function Coverage: 35.27% (8730/24749)
Line Coverage: 27.09% (71507/263925)
Region Coverage: 26.33% (37097/140909)
Branch Coverage: 23.24% (18971/81646)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9352fc88715d81519e6f2e27b8e3a3b23d4aea03_9352fc88715d81519e6f2e27b8e3a3b23d4aea03/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #32554:
URL: https://github.com/apache/doris/pull/32554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on code in PR #32554:
URL: https://github.com/apache/doris/pull/32554#discussion_r1533149083
##########
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##########
@@ -120,6 +128,20 @@ private WorkloadGroup(long id, String name, Map<String, String> properties, long
this.cpuHardLimit = Integer.parseInt(cpuHardLimitStr);
this.properties.put(CPU_HARD_LIMIT, cpuHardLimitStr);
}
+ if (properties.containsKey(SPILL_THRESHOLD_LOW_WATERMARK)) {
+ String lowWatermarkStr = properties.get(SPILL_THRESHOLD_LOW_WATERMARK);
+ if (lowWatermarkStr.endsWith("%")) {
+ lowWatermarkStr = lowWatermarkStr.substring(0, lowWatermarkStr.length() - 1);
Review Comment:
Value check in WorkloadGroup.checkProperties.
What does 0 means?
##########
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##########
@@ -120,6 +128,20 @@ private WorkloadGroup(long id, String name, Map<String, String> properties, long
this.cpuHardLimit = Integer.parseInt(cpuHardLimitStr);
this.properties.put(CPU_HARD_LIMIT, cpuHardLimitStr);
}
+ if (properties.containsKey(SPILL_THRESHOLD_LOW_WATERMARK)) {
+ String lowWatermarkStr = properties.get(SPILL_THRESHOLD_LOW_WATERMARK);
+ if (lowWatermarkStr.endsWith("%")) {
+ lowWatermarkStr = lowWatermarkStr.substring(0, lowWatermarkStr.length() - 1);
+ }
+ this.properties.put(SPILL_THRESHOLD_LOW_WATERMARK, lowWatermarkStr);
+ }
+ if (properties.containsKey(SPILL_THRESHOLD_HIGH_WATERMARK)) {
+ String highWatermarkStr = properties.get(SPILL_THRESHOLD_HIGH_WATERMARK);
+ if (highWatermarkStr.endsWith("%")) {
Review Comment:
Value check in WorkloadGroup.checkProperties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2009194808
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2011088244
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2012159813
<details>
<summary>TPC-H: <b>Total hot run time: 37454 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9352fc88715d81519e6f2e27b8e3a3b23d4aea03, data reload: false
------ Round 1 ----------------------------------
q1 17659 4905 4095 4095
q2 2029 149 141 141
q3 10598 1090 1171 1090
q4 10225 791 751 751
q5 7452 3014 2953 2953
q6 201 123 123 123
q7 1025 577 551 551
q8 9338 1995 2016 1995
q9 7123 6430 6439 6430
q10 8403 3402 3504 3402
q11 434 223 217 217
q12 425 195 191 191
q13 17797 2856 2855 2855
q14 258 205 207 205
q15 511 453 460 453
q16 452 353 347 347
q17 951 529 597 529
q18 7099 6529 6400 6400
q19 3053 1353 1391 1353
q20 561 250 253 250
q21 3529 2818 2930 2818
q22 348 309 305 305
Total cold run time: 109471 ms
Total hot run time: 37454 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4144 4097 4088 4088
q2 322 223 222 222
q3 2952 2857 2845 2845
q4 1861 1576 1569 1569
q5 5215 5291 5244 5244
q6 195 115 117 115
q7 2269 1818 1854 1818
q8 3152 3321 3266 3266
q9 8599 8584 8623 8584
q10 3708 3682 3660 3660
q11 539 440 436 436
q12 721 530 532 530
q13 16932 2874 2820 2820
q14 289 258 256 256
q15 481 442 448 442
q16 458 405 406 405
q17 1726 1487 1465 1465
q18 7419 7336 7092 7092
q19 1612 1501 1542 1501
q20 1890 1696 1703 1696
q21 4795 4709 4639 4639
q22 513 473 461 461
Total cold run time: 69792 ms
Total hot run time: 53154 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2009344114
TeamCity be ut coverage result:
Function Coverage: 35.30% (8730/24731)
Line Coverage: 27.09% (71397/263549)
Region Coverage: 26.34% (37060/140714)
Branch Coverage: 23.24% (18944/81520)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9bee27aa905fc83ee0b3b3392879d2de20069737_9bee27aa905fc83ee0b3b3392879d2de20069737/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2009661292
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2011180370
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Add spill property for workload group [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32554:
URL: https://github.com/apache/doris/pull/32554#issuecomment-2009764992
TeamCity be ut coverage result:
Function Coverage: 35.27% (8727/24744)
Line Coverage: 27.08% (71424/263753)
Region Coverage: 26.32% (37065/140816)
Branch Coverage: 23.23% (18955/81588)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a8e6f6d23a1f1ba8066ba37be5a59bc933e45ceb_a8e6f6d23a1f1ba8066ba37be5a59bc933e45ceb/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org