You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/12/23 13:11:36 UTC
[GitHub] [incubator-doris] tianhui5 opened a new pull request #7474: [Feature] Limit cluster resource usage in user granularity #7129
tianhui5 opened a new pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474
## Proposed changes
Fix #7129
## Types of changes
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation Update (if none of the other choices apply)
- [ ] Code refactor (Modify the code structure, format the code, etc...)
- [ ] Optimization. Including functional usability improvements and performance improvements.
- [ ] Dependency. Such as changes related to third-party components.
- [ ] Other.
## Checklist
- [x] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail
- [ ] Compiling and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] If these changes need document changes, I have updated the document
- [ ] Any dependent changes have been merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #7474: [Feature] Limit frequency of query and load in system level
Posted by GitBox <gi...@apache.org>.
morningman commented on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1037702437
Hi @tianhui5
Although we had some private discussions about this feature, we eventually needed a complete design and usage document to help all developers and users understand how this feature works and how to use it.
It is recommended that the final design be fully described in the issue #7129 (and the previous design be marked as Deprecated).
Also, I suggest a separate user document describing the functionality from a usage perspective, including the followings:
1. the purpose of the feature (Motivation)
2. what scenarios the feature is applicable to, how to use it, and what impact it will have.
3. whether there are shortcomings for further improvement.
In the previous Doris development, we also had a lot of features that were difficult to maintain due to lack of documentation, and I hope to be able to improve the process in this area in future projects. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #7474: [Feature] Limit frequency of operation in both cluster and user granularity
Posted by GitBox <gi...@apache.org>.
morningman commented on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1009002210
There are two main issues
1. all query requests need to send an RPC to Master FE, which is too much overhead.
2. Limiter is not set in the right place. For example, for query, it should be handled at the place where the query plan is finally sent to BE. For load, the frequency of transactions should be limited.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz edited a comment on pull request #7474: [Feature] Limit cluster resource usage in user granularity #7129
Posted by GitBox <gi...@apache.org>.
xinyiZzz edited a comment on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1000325263
Good idea,
If FE can get the memory consumption of each query, we can limit the memory used by users, which is what I plan to do in the future.
Impala’s AdmissionController does a similar thing, Introduction is here https://shimo.im/docs/6qxjctpyDHJgPwtw
In #7198, I have counted the real memory consumption of each Query, which may be implemented based on your PR later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #7474: [Feature] Limit cluster resource usage in user granularity #7129
Posted by GitBox <gi...@apache.org>.
morningman commented on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1000408597
Hi @tianhui5 , thanks for your contribution.
But better to add document to explain the usage or best practice.
Otherwise, it is hard to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #7474: [Feature] Limit frequency of query and load in system level
Posted by GitBox <gi...@apache.org>.
morningman commented on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1037702437
Hi @tianhui5
Although we had some private discussions about this feature, we eventually needed a complete design and usage document to help all developers and users understand how this feature works and how to use it.
It is recommended that the final design be fully described in the issue #7129 (and the previous design be marked as Deprecated).
Also, I suggest a separate user document describing the functionality from a usage perspective, including the followings:
1. the purpose of the feature (Motivation)
2. what scenarios the feature is applicable to, how to use it, and what impact it will have.
3. whether there are shortcomings for further improvement.
In the previous Doris development, we also had a lot of features that were difficult to maintain due to lack of documentation, and I hope to be able to improve the process in this area in future projects. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz edited a comment on pull request #7474: [Feature] Limit cluster resource usage in user granularity #7129
Posted by GitBox <gi...@apache.org>.
xinyiZzz edited a comment on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1000325263
good idea,
If FE can get the memory consumption of each query, we can limit the memory used by users, which is what I plan to do in the future.
Impala’s AdmissionController does a similar thing, Introduction is here https://shimo.im/docs/6qxjctpyDHJgPwtw
In #7198, I have counted the real memory consumption of each Query, which may be implemented based on this PR later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on pull request #7474: [Feature] Limit cluster resource usage in user granularity #7129
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1000325263
good idea,
If FE can get the memory consumption of each query, we can limit the memory used by users, which is what I plan to do in the future.
The actual AdmissionController of Impala does a similar thing, refer to: https://shimo.im/docs/6qxjctpyDHJgPwtw
In #7198, I have counted the real memory consumption of each Query, which may be implemented based on this PR later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz edited a comment on pull request #7474: [Feature] Limit cluster resource usage in user granularity #7129
Posted by GitBox <gi...@apache.org>.
xinyiZzz edited a comment on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1000325263
Good idea,
If FE can get the memory consumption of each Query and Load, we can limit the memory used by users, which is what I plan to do in the future.
Impala’s AdmissionController does a similar thing, Introduction is here https://shimo.im/docs/6qxjctpyDHJgPwtw
In #7198, I have counted the real memory consumption of each Query, which may be implemented based on your PR later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 edited a comment on pull request #7474: [Feature] Limit frequency of query and load in system level
Posted by GitBox <gi...@apache.org>.
tianhui5 edited a comment on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1038814985
> Hi @tianhui5
>
> Although we had some private discussions about this feature, we eventually needed a complete design and usage document to help all developers and users understand how this feature works and how to use it.
>
> It is recommended that the final design be fully described in the issue #7129 (and the previous design be marked as Deprecated).
>
> Also, I suggest a separate user document describing the functionality from a usage perspective, including the followings:
>
> 1. the purpose of the feature (Motivation)
> 2. what scenarios the feature is applicable to, how to use it, and what impact it will have.
> 3. whether there are shortcomings for further improvement.
>
> In the previous Doris development, we also had a lot of features that were difficult to maintain due to lack of documentation, and I hope to be able to improve the process in this area in future projects. Thanks!
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz edited a comment on pull request #7474: [Feature] Limit cluster resource usage in user granularity #7129
Posted by GitBox <gi...@apache.org>.
xinyiZzz edited a comment on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1000325263
Good idea,
If FE can get the memory consumption of each query, we can limit the memory used by users, which is what I plan to do in the future.
Impala’s AdmissionController does a similar thing, Introduction is here https://shimo.im/docs/6qxjctpyDHJgPwtw
In #7198, I have counted the real memory consumption of each Query, which may be implemented based on this PR later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on pull request #7474: [Feature] Limit frequency of query and load in system level
Posted by GitBox <gi...@apache.org>.
tianhui5 commented on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1038814296
>
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on pull request #7474: [Feature] Limit frequency of operation in both cluster and user granularity
Posted by GitBox <gi...@apache.org>.
tianhui5 commented on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1004706152
> Hi @tianhui5 , thanks for your contribution. But better to add document to explain the usage or best practice. Otherwise, it is hard to review.
Thanks for remind me. I've add relative doc and unit test, please review this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #7474: [Feature] Limit frequency of operation in both cluster and user granularity
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#discussion_r781212291
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -713,6 +735,20 @@
@ConfField(mutable = true, masterOnly = true)
public static int max_running_txn_num_per_db = 100;
+ /**
+ * maximum QPS of total system,
+ * new query will be rejected if QPS exceeds the threshold.
+ */
+ @ConfField(mutable = true, masterOnly = true, callback = QueryLimiterConfHandler.class)
Review comment:
I think this is not `masterOnly`?
##########
File path: fe/fe-core/src/main/java/org/apache/doris/http/rest/LoadAction.java
##########
@@ -116,6 +118,15 @@ public void executeWithoutPassword(BaseRequest request, BaseResponse response) t
}
redirectAddr = execEnv.getMultiLoadMgr().redirectAddr(fullDbName, label);
} else {
+ // Check rate limiter.
+ MasterRpcExecutor executor = new MasterRpcExecutor(ConnectContext.get());
+ try {
+ if (!executor.checkRateLimiter(TLimiterType.STREAM_LOAD)) {
+ throw new Exception("Check stream load rate limiter returned fail!");
Review comment:
How about return detail error msg?
And better to throw exception directly from `checkRateLimiter()`
##########
File path: fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
##########
@@ -335,6 +336,19 @@ private void dispatch() throws IOException {
LOG.warn("Unknown command(" + command + ")");
return;
}
+ // check rate limiter.
+ try {
+ MasterRpcExecutor executor = new MasterRpcExecutor(ctx);
Review comment:
1. `dispatch()` is not always QUERY
2. it is not good send rpc to master FE for each request.
##########
File path: fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserProperty.java
##########
@@ -501,6 +535,9 @@ public void write(DataOutput out) throws IOException {
// user resource
resource.write(out);
+ // user limiter
+ limiter.write(out);
Review comment:
You need to put all properties into `commonProperties`, so that you don't need to add new field for persistence.
You can refer to `cpu_resource_limit`
##########
File path: fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserProperty.java
##########
@@ -17,6 +17,7 @@
package org.apache.doris.mysql.privilege;
+import com.google.common.util.concurrent.AtomicDouble;
Review comment:
pay attention to the import order
##########
File path: fe/fe-core/src/main/java/org/apache/doris/system/RateLimiterMgr.java
##########
@@ -0,0 +1,100 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.system;
+
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.RateLimiter;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.mysql.privilege.UserProperty;
+import org.apache.doris.thrift.TLimiterType;
+
+import java.util.Map;
+
+/**
+ * RateLimiterMgr collect all rate limiter of the system.
+ */
+public class RateLimiterMgr {
+ private RateLimiter systemQPSLimiter;
+ private RateLimiter systemStreamLoadLimiter;
+ private Map<String, RateLimiter> userToQPSLimiter;
+ private Map<String, RateLimiter> userToStreamLoadLimiter;
+
+ public RateLimiterMgr(double QPS, double streamLoadPS) {
+ this.systemQPSLimiter = RateLimiter.create(QPS);
+ this.systemStreamLoadLimiter = RateLimiter.create(streamLoadPS);
+ this.userToQPSLimiter = Maps.newConcurrentMap();
+ this.userToStreamLoadLimiter = Maps.newConcurrentMap();
+ }
+
+ public synchronized boolean checkRateLimiter(TLimiterType type, String user) {
Review comment:
why using `synchronized`? the RateLimiter is thread safe
##########
File path: fe/fe-core/src/main/java/org/apache/doris/system/RateLimiterMgr.java
##########
@@ -0,0 +1,100 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.system;
+
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.RateLimiter;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.mysql.privilege.UserProperty;
+import org.apache.doris.thrift.TLimiterType;
+
+import java.util.Map;
+
+/**
+ * RateLimiterMgr collect all rate limiter of the system.
+ */
+public class RateLimiterMgr {
+ private RateLimiter systemQPSLimiter;
+ private RateLimiter systemStreamLoadLimiter;
+ private Map<String, RateLimiter> userToQPSLimiter;
+ private Map<String, RateLimiter> userToStreamLoadLimiter;
+
+ public RateLimiterMgr(double QPS, double streamLoadPS) {
Review comment:
It is not recommended to pass these values through the constructor's parameters. If we subsequently add more limiter, there will be more arguments to the constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 removed a comment on pull request #7474: [Feature] Limit frequency of query and load in system level
Posted by GitBox <gi...@apache.org>.
tianhui5 removed a comment on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1038814296
>
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on pull request #7474: [Feature] Limit frequency of operation in both cluster and user granularity
Posted by GitBox <gi...@apache.org>.
tianhui5 commented on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1024867902
> There are two main issues
>
> 1. all query requests need to send an RPC to Master FE, which is too much overhead.
> 2. Limiter is not set in the right place. For example, for query, it should be handled at the place where the query plan is finally sent to BE. For load, the frequency of transactions should be limited.
Yes, that seems to have too much RPC. I have rewrote the feature with statistic sychronize based on bdbje. Because I think it's fine to use bdbje to store some temperory statistics, and there is too much work to develop a middleware myself.
Please review this patch, thanks for your time!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on pull request #7474: [Feature] Limit frequency of query and load in system level
Posted by GitBox <gi...@apache.org>.
tianhui5 commented on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1044120193
The local current query number is reported to Master every certain time, and then send to all FE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on pull request #7474: [Feature] Limit frequency of query and load in system level
Posted by GitBox <gi...@apache.org>.
tianhui5 commented on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1038814985
> Hi @tianhui5
>
> Although we had some private discussions about this feature, we eventually needed a complete design and usage document to help all developers and users understand how this feature works and how to use it.
>
> It is recommended that the final design be fully described in the issue #7129 (and the previous design be marked as Deprecated).
>
> Also, I suggest a separate user document describing the functionality from a usage perspective, including the followings:
>
> 1. the purpose of the feature (Motivation)
> 2. what scenarios the feature is applicable to, how to use it, and what impact it will have.
> 3. whether there are shortcomings for further improvement.
>
> In the previous Doris development, we also had a lot of features that were difficult to maintain due to lack of documentation, and I hope to be able to improve the process in this area in future projects. Thanks!
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 edited a comment on pull request #7474: [Feature] Limit frequency of operation in both cluster and user granularity
Posted by GitBox <gi...@apache.org>.
tianhui5 edited a comment on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1004708964
> Good idea, If FE can get the memory consumption of each Query and Load, we can limit the memory used by users, which is what I plan to do in the future.
>
> Impala’s AdmissionController does a similar thing, Introduction is here https://shimo.im/docs/6qxjctpyDHJgPwtw
>
> In #7198, I have counted the real memory consumption of each Query, which may be implemented based on your PR later
Nice work.
In my PR, I've limit the system usage through a general way, like limit QPS. We can develop a more concise way in next move as you say.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on pull request #7474: [Feature] Limit frequency of operation in both cluster and user granularity
Posted by GitBox <gi...@apache.org>.
tianhui5 commented on pull request #7474:
URL: https://github.com/apache/incubator-doris/pull/7474#issuecomment-1004708964
> Good idea, If FE can get the memory consumption of each Query and Load, we can limit the memory used by users, which is what I plan to do in the future.
>
> Impala’s AdmissionController does a similar thing, Introduction is here https://shimo.im/docs/6qxjctpyDHJgPwtw
>
> In #7198, I have counted the real memory consumption of each Query, which may be implemented based on your PR later
Nice work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org