You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wangbo (via GitHub)" <gi...@apache.org> on 2023/11/30 10:52:20 UTC
[PR] [Fix](exectuor)Fix Follower Fe query queue may not work when exec alter [doris]
wangbo opened a new pull request, #27831:
URL: https://github.com/apache/doris/pull/27831
## Proposed changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](exectuor)Fix Follower Fe query queue may not work when exec alter [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27831:
URL: https://github.com/apache/doris/pull/27831#issuecomment-1837110041
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](exectuor)Fix Follower Fe query queue may not work when exec alter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27831:
URL: https://github.com/apache/doris/pull/27831#issuecomment-1837127441
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.65 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.6 seconds inserted 10000000 Rows, about 337K ops/s
storage size: 17164003351 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](exectuor)Fix Follower Fe query queue may not work when exec alter [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #27831:
URL: https://github.com/apache/doris/pull/27831#issuecomment-1833530126
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](exectuor)Fix Follower Fe query queue may not work when exec alter [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #27831:
URL: https://github.com/apache/doris/pull/27831#issuecomment-1837105179
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](exectuor)Fix Follower Fe query queue may not work when exec alter [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #27831:
URL: https://github.com/apache/doris/pull/27831#discussion_r1410747153
##########
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/QueryQueue.java:
##########
@@ -44,6 +47,12 @@ public class QueryQueue {
public static final String RUNNING_QUERY_NUM = "running_query_num";
public static final String WAITING_QUERY_NUM = "waiting_query_num";
+ private Thread updatePropThread;
Review Comment:
我们写一个thread
1. 遍历所有的workloadgroup
2. 检查每个workload group 里的query queue的配置和workloadgroup 元数据对比，如果不同，那么就reset queue 的设置。
##########
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/QueryQueue.java:
##########
@@ -44,6 +47,12 @@ public class QueryQueue {
public static final String RUNNING_QUERY_NUM = "running_query_num";
public static final String WAITING_QUERY_NUM = "waiting_query_num";
+ private Thread updatePropThread;
Review Comment:
我们写一个thread
1. 遍历所有的workloadgroup
2. 检查每个workload group 里的query queue的配置和workloadgroup 元数据对比，如果不同，那么就reset queue 的设置。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](exectuor)Fix Follower Fe query queue may not work when exec alter [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #27831:
URL: https://github.com/apache/doris/pull/27831#discussion_r1412744471
##########
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java:
##########
@@ -75,9 +76,65 @@ public class WorkloadGroupMgr implements Writable, GsonPostProcessable {
@SerializedName(value = "idToWorkloadGroup")
private final Map<Long, WorkloadGroup> idToWorkloadGroup = Maps.newHashMap();
private final Map<String, WorkloadGroup> nameToWorkloadGroup = Maps.newHashMap();
+ private final Map<Long, QueryQueue> idToQueryQueue = Maps.newHashMap();
private final ResourceProcNode procNode = new ResourceProcNode();
private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
+ private Thread updatePropThread;
+ private volatile boolean stopped = false;
+
+ public void stopUpdateThread() {
Review Comment:
stop never called？
##########
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java:
##########
@@ -75,9 +76,65 @@ public class WorkloadGroupMgr implements Writable, GsonPostProcessable {
@SerializedName(value = "idToWorkloadGroup")
private final Map<Long, WorkloadGroup> idToWorkloadGroup = Maps.newHashMap();
private final Map<String, WorkloadGroup> nameToWorkloadGroup = Maps.newHashMap();
+ private final Map<Long, QueryQueue> idToQueryQueue = Maps.newHashMap();
private final ResourceProcNode procNode = new ResourceProcNode();
private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
+ private Thread updatePropThread;
+ private volatile boolean stopped = false;
+
+ public void stopUpdateThread() {
Review Comment:
stop never called？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](exectuor)Fix Follower Fe query queue may not work when exec alter [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #27831:
URL: https://github.com/apache/doris/pull/27831#issuecomment-1837039054
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](exectuor)Fix Follower Fe query queue may not work when exec alter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27831:
URL: https://github.com/apache/doris/pull/27831#issuecomment-1833732900
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.79 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17163937759 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](exectuor)Fix Follower Fe query queue may not work when exec alter [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27831:
URL: https://github.com/apache/doris/pull/27831#issuecomment-1837110033
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](exectuor)Fix Follower Fe query queue may not work when exec alter [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #27831:
URL: https://github.com/apache/doris/pull/27831
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](exectuor)Fix Follower Fe query queue may not work when exec alter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27831:
URL: https://github.com/apache/doris/pull/27831#issuecomment-1837063967
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.77 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17164375314 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](exectuor)Fix Follower Fe query queue may not work when exec alter [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #27831:
URL: https://github.com/apache/doris/pull/27831#discussion_r1412743928
##########
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java:
##########
@@ -75,9 +76,65 @@ public class WorkloadGroupMgr implements Writable, GsonPostProcessable {
@SerializedName(value = "idToWorkloadGroup")
private final Map<Long, WorkloadGroup> idToWorkloadGroup = Maps.newHashMap();
private final Map<String, WorkloadGroup> nameToWorkloadGroup = Maps.newHashMap();
+ private final Map<Long, QueryQueue> idToQueryQueue = Maps.newHashMap();
private final ResourceProcNode procNode = new ResourceProcNode();
private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
+ private Thread updatePropThread;
+ private volatile boolean stopped = false;
+
+ public void stopUpdateThread() {
+ this.stopped = true;
+ }
+
+ public void startUpdateThread() {
+ WorkloadGroupMgr wgMgr = this;
+ updatePropThread = new Thread(new Runnable() {
+ public void run() {
+ while (!wgMgr.stopped) {
+ try {
+ wgMgr.resetQueryQueueProp();
+ Thread.sleep(Config.query_queue_update_interval_ms);
+ } catch (Exception e) {
Review Comment:
catch Throwable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](exectuor)Fix Follower Fe query queue may not work when exec alter [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #27831:
URL: https://github.com/apache/doris/pull/27831#discussion_r1410747153
##########
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/QueryQueue.java:
##########
@@ -44,6 +47,12 @@ public class QueryQueue {
public static final String RUNNING_QUERY_NUM = "running_query_num";
public static final String WAITING_QUERY_NUM = "waiting_query_num";
+ private Thread updatePropThread;
Review Comment:
我们在WorkloadGroupMgr中写一个thread
1. 遍历所有的workloadgroup
2. 检查每个workload group 里的query queue的配置和workloadgroup 元数据对比，如果不同，那么就reset queue 的设置。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](exectuor)Fix Follower Fe query queue may not work when exec alter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27831:
URL: https://github.com/apache/doris/pull/27831#issuecomment-1833902570
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 21ed5a027b2178f0b2ec5be07185a8bf01e23d92, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4894 4663 4643 4643
q2 354 159 157 157
q3 1510 1324 1320 1320
q4 1154 968 999 968
q5 3257 3242 3240 3240
q6 258 127 127 127
q7 1028 507 559 507
q8 2220 2229 2222 2222
q9 6922 6942 6903 6903
q10 3302 3390 3363 3363
q11 346 214 203 203
q12 379 253 250 250
q13 4690 3927 3885 3885
q14 249 227 215 215
q15 590 543 523 523
q16 426 372 374 372
q17 1036 709 628 628
q18 8077 7584 7477 7477
q19 1565 1520 1518 1518
q20 586 306 321 306
q21 3402 2985 3014 2985
q22 371 293 310 293
Total cold run time: 46616 ms
Total hot run time: 42105 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4627 4584 4576 4576
q2 317 237 207 207
q3 3760 3754 3756 3754
q4 2524 2509 2509 2509
q5 6169 6180 6205 6180
q6 247 123 122 122
q7 2581 1978 1975 1975
q8 3725 3705 3712 3705
q9 9485 9365 9406 9365
q10 4072 4143 4145 4143
q11 613 502 499 499
q12 792 637 625 625
q13 4378 3593 3652 3593
q14 268 261 251 251
q15 589 522 516 516
q16 516 468 485 468
q17 2092 2067 2099 2067
q18 9437 9039 8788 8788
q19 1813 1792 1792 1792
q20 2313 1998 1967 1967
q21 7341 6880 7107 6880
q22 664 573 586 573
Total cold run time: 68323 ms
Total hot run time: 64555 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org