You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/06/27 07:55:41 UTC
[GitHub] [incubator-doris] caiconghui opened a new pull request #3955: Support checking database used data quota when data load job begin a new txn
caiconghui opened a new pull request #3955:
URL: https://github.com/apache/incubator-doris/pull/3955
Now, we only check database used data quota when create or alter table, or in some old type load job, but not for routine load job and stream load job. This PR provide a uniform solution to check db used data quota when data load job begin a new txn.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #3955: Support checking database used data quota when data load job begin a new txn
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3955:
URL: https://github.com/apache/incubator-doris/pull/3955#discussion_r449725096
##########
File path: fe/src/main/java/org/apache/doris/common/Config.java
##########
@@ -873,6 +873,18 @@
*/
@ConfField(mutable = true, masterOnly = true)
public static boolean disable_load_job = false;
+
+ /*
+ * if this is set to true, all load job will check db data quota when call begin txn api
+ */
+ @ConfField(mutable = true, masterOnly = true)
+ public static boolean enable_check_data_quota_on_load = false;
Review comment:
I think this can be set to true by default.
##########
File path: fe/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
##########
@@ -246,6 +249,10 @@ private void getTxnStateInfo(TransactionState txnState, List<String> info) {
public long beginTransaction(List<Long> tableIdList, String label, TUniqueId requestId,
TransactionState.TxnCoordinator coordinator, TransactionState.LoadJobSourceType sourceType, long listenerId, long timeoutSecond)
throws DuplicatedRequestException, LabelAlreadyUsedException, BeginTransactionException, AnalysisException {
+ if (Config.enable_check_data_quota_on_load) {
+ checkDatabaseDataQuota();
Review comment:
```suggestion
checkDatabaseDataQuota();
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli merged pull request #3955: Support checking database used data quota when data load job begin a new txn
Posted by GitBox <gi...@apache.org>.
chaoyli merged pull request #3955:
URL: https://github.com/apache/incubator-doris/pull/3955
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on a change in pull request #3955: Support checking database used data quota when data load job begin a new txn
Posted by GitBox <gi...@apache.org>.
caiconghui commented on a change in pull request #3955:
URL: https://github.com/apache/incubator-doris/pull/3955#discussion_r456926998
##########
File path: docs/en/administrator-guide/config/fe_config.md
##########
@@ -261,6 +267,12 @@ This configuration can play a role in certain scenarios. Assume that the initial
### `enable_auth_check`
+### `enable_check_data_quota_on_load`
Review comment:
@chaoyli
1. The main consideration is to make the user's load task not fail if they don't want use this load quota limit, but now it seems a bug?
2. TabletStatMgr will update replica rowcount and datasize every 5min now, but not compute data quota, data quota will be computed only when show data operation is executed or create table or alter table schema.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on a change in pull request #3955: Support checking database used data quota when data load job begin a new txn
Posted by GitBox <gi...@apache.org>.
caiconghui commented on a change in pull request #3955:
URL: https://github.com/apache/incubator-doris/pull/3955#discussion_r449755668
##########
File path: fe/src/main/java/org/apache/doris/common/Config.java
##########
@@ -873,6 +873,18 @@
*/
@ConfField(mutable = true, masterOnly = true)
public static boolean disable_load_job = false;
+
+ /*
+ * if this is set to true, all load job will check db data quota when call begin txn api
+ */
+ @ConfField(mutable = true, masterOnly = true)
+ public static boolean enable_check_data_quota_on_load = false;
Review comment:
done
##########
File path: fe/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
##########
@@ -246,6 +249,10 @@ private void getTxnStateInfo(TransactionState txnState, List<String> info) {
public long beginTransaction(List<Long> tableIdList, String label, TUniqueId requestId,
TransactionState.TxnCoordinator coordinator, TransactionState.LoadJobSourceType sourceType, long listenerId, long timeoutSecond)
throws DuplicatedRequestException, LabelAlreadyUsedException, BeginTransactionException, AnalysisException {
+ if (Config.enable_check_data_quota_on_load) {
+ checkDatabaseDataQuota();
Review comment:
fix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on a change in pull request #3955: Support checking database used data quota when data load job begin a new txn
Posted by GitBox <gi...@apache.org>.
chaoyli commented on a change in pull request #3955:
URL: https://github.com/apache/incubator-doris/pull/3955#discussion_r456903718
##########
File path: docs/en/administrator-guide/config/fe_config.md
##########
@@ -261,6 +267,12 @@ This configuration can play a role in certain scenarios. Assume that the initial
### `enable_auth_check`
+### `enable_check_data_quota_on_load`
Review comment:
1. I think check database quota can be checked upon every load. So we can check every time without this config.
2. How about data quota update frequency previously? It's used by table report, It seems that we can continue to use it or not ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on pull request #3955: Support checking database used data quota when data load job begin a new txn
Posted by GitBox <gi...@apache.org>.
caiconghui commented on pull request #3955:
URL: https://github.com/apache/incubator-doris/pull/3955#issuecomment-650519354
for #3956
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org