You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/12/15 10:17:31 UTC
[GitHub] [incubator-doris] Henry2SS opened a new pull request #7403: [feature] sql-block-rule: add partitionNum, tabletNum, cardinality in SqlBlockRule to block big/slow sql
Henry2SS opened a new pull request #7403:
URL: https://github.com/apache/incubator-doris/pull/7403
1. set partitionNum, tabletNum, cardinality as limitations to block sqls
2. compatible with lower version
3. add unit tests
4. add docs
## Proposed changes
There are many large queries that scan the entire table or large amounts of data, and most of them are triggered manually or accidentally. In any case, this poses a great threat to the stability of the cluster.
Therefore, we need more precise SQL block methods.
It should :
1. Just bolck query statement
2. Let the explain statement execute
3. Block large queries before being executed, but after being planned to get core information of OlapScanNode
Close related #7202 (replace it with issue number if it exists).
Describe the overview of changes, and introduce why we need it.
## Types of changes
What types of changes does your code introduce to Doris?
_Put an `x` in the boxes that apply_
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [x] Documentation Update (if none of the other choices apply)
- [x] Code refactor (Modify the code structure, format the code, etc...)
- [ ] Optimization. Including functional usability improvements and performance improvements.
- [ ] Dependency. Such as changes related to third-party components.
- [ ] Other.
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._
- [x] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail
- [ ] Compiling and unit tests pass locally with my changes
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] If these changes need document changes, I have updated the document
- [ ] Any dependent changes have been merged
## Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #7403: [feature] sql-block-rule: add partitionNum, tabletNum, cardinality in SqlBlockRule to block big/slow sql
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #7403:
URL: https://github.com/apache/incubator-doris/pull/7403#discussion_r774327223
##########
File path: docs/zh-CN/administrator-guide/block-rule/sql-block.md
##########
@@ -51,15 +59,27 @@ PROPERTIES(
mysql> select * from order_analysis;
ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule
```
+
+```sql
+CREATE SQL_BLOCK_RULE test_rule2 PROPERTIES("partitionNum" = "30", "cardinality"="10000000000","global"="false","enable"="true")
Review comment:
这个sql示例没有说明？
##########
File path: docs/zh-CN/administrator-guide/block-rule/sql-block.md
##########
@@ -26,15 +26,23 @@ under the License.
# SQL黑名单
-支持按用户配置SQL黑名单，通过正则匹配的方式拒绝指定SQL。
该功能仅用于限制查询语句，并且不会限制 explain 语句的执行。
+支持按用户配置SQL黑名单:
+
+1. 通过正则匹配的方式拒绝指定SQL
+
+2. 通过设置partitionNum, tabletNum, cardinality, 检查一个查询是否达到其中一个限制
+ - partitionNum, tabletNum, cardinality 可以一起设置，一旦一个查询达到其中一个限制，查询将会被拦截
## 规则
对SQL规则增删改查
- 创建SQL阻止规则
- - sql：匹配规则(基于正则匹配,特殊字符需要转译)，可选
- - sqlHash: sql hash值，用于完全匹配，我们会在`fe.audit.log`打印这个值，可选，这个参数和sql只能二选一,如果给空值，是空字符串不是null
+ - sql：匹配规则(基于正则匹配,特殊字符需要转译)，可选，默认值为 "NULL"
+ - sqlHash: sql hash值，用于完全匹配，我们会在`fe.audit.log`打印这个值，可选，这个参数和sql只能二选一，默认值为 "NULL"
+ - partitionNum: 一个扫描节点会扫描的最大partition数量，默认值为0L
+ - tabletNum: 一个扫描节点会扫描的最大扽tablet数量，默认值为0L
Review comment:
```suggestion
- tabletNum: 一个扫描节点会扫描的最大tablet数量，默认值为0L
```
##########
File path: docs/zh-CN/administrator-guide/block-rule/sql-block.md
##########
@@ -51,15 +59,27 @@ PROPERTIES(
mysql> select * from order_analysis;
ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule
```
+
+```sql
+CREATE SQL_BLOCK_RULE test_rule2 PROPERTIES("partitionNum" = "30", "cardinality"="10000000000","global"="false","enable"="true")
Review comment:
```suggestion
CREATE SQL_BLOCK_RULE test_rule2 PROPERTIES("partition_num" = "30", "cardinality"="10000000000","global"="false","enable"="true")
```
##########
File path: fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
##########
@@ -107,12 +108,22 @@ public void alterSqlBlockRule(AlterSqlBlockRuleStmt stmt) throws DdlException {
throw new DdlException("the sql block rule " + ruleName + " not exist");
}
SqlBlockRule originRule = nameToSqlBlockRuleMap.get(ruleName);
+ SqlBlockUtil.checkAlterValidate(sqlBlockRule, originRule);
if (StringUtils.isEmpty(sqlBlockRule.getSql())) {
sqlBlockRule.setSql(originRule.getSql());
}
if (StringUtils.isEmpty(sqlBlockRule.getSqlHash())) {
sqlBlockRule.setSqlHash(originRule.getSqlHash());
}
+ if (StringUtils.isEmpty(sqlBlockRule.getPartitionNum().toString())) {
Review comment:
sqlBlockRule.getPartitionNum() may be null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] Henry2SS commented on a change in pull request #7403: [feature] sql-block-rule: add partitionNum, tabletNum, cardinality in SqlBlockRule to block big/slow sql
Posted by GitBox <gi...@apache.org>.
Henry2SS commented on a change in pull request #7403:
URL: https://github.com/apache/incubator-doris/pull/7403#discussion_r775036629
##########
File path: fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
##########
@@ -107,12 +108,22 @@ public void alterSqlBlockRule(AlterSqlBlockRuleStmt stmt) throws DdlException {
throw new DdlException("the sql block rule " + ruleName + " not exist");
}
SqlBlockRule originRule = nameToSqlBlockRuleMap.get(ruleName);
+ SqlBlockUtil.checkAlterValidate(sqlBlockRule, originRule);
if (StringUtils.isEmpty(sqlBlockRule.getSql())) {
sqlBlockRule.setSql(originRule.getSql());
}
if (StringUtils.isEmpty(sqlBlockRule.getSqlHash())) {
sqlBlockRule.setSqlHash(originRule.getSqlHash());
}
+ if (StringUtils.isEmpty(sqlBlockRule.getPartitionNum().toString())) {
Review comment:
NO.
`sqlBlockRule ` in this place, is from `SqlBlockRule sqlBlockRule = SqlBlockRule.fromAlterStmt(stmt);`
and `stmt` is an instance of AlterSqlBlockRuleStmt.
In the method called `setProperties` in AlterSqlBlockRuleStmt.java, all the properties are set default values.
And I've tested ALTER:
1. from lower version to this version, it is compatible；
2. ALTER operation work well
3. ALTER operatio can report relevant exceptions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] Henry2SS commented on a change in pull request #7403: [feature] sql-block-rule: add partitionNum, tabletNum, cardinality in SqlBlockRule to block big/slow sql
Posted by GitBox <gi...@apache.org>.
Henry2SS commented on a change in pull request #7403:
URL: https://github.com/apache/incubator-doris/pull/7403#discussion_r775039218
##########
File path: docs/zh-CN/administrator-guide/block-rule/sql-block.md
##########
@@ -26,15 +26,23 @@ under the License.
# SQL黑名单
-支持按用户配置SQL黑名单，通过正则匹配的方式拒绝指定SQL。
该功能仅用于限制查询语句，并且不会限制 explain 语句的执行。
+支持按用户配置SQL黑名单:
+
+1. 通过正则匹配的方式拒绝指定SQL
+
+2. 通过设置partitionNum, tabletNum, cardinality, 检查一个查询是否达到其中一个限制
+ - partitionNum, tabletNum, cardinality 可以一起设置，一旦一个查询达到其中一个限制，查询将会被拦截
## 规则
对SQL规则增删改查
- 创建SQL阻止规则
- - sql：匹配规则(基于正则匹配,特殊字符需要转译)，可选
- - sqlHash: sql hash值，用于完全匹配，我们会在`fe.audit.log`打印这个值，可选，这个参数和sql只能二选一,如果给空值，是空字符串不是null
+ - sql：匹配规则(基于正则匹配,特殊字符需要转译)，可选，默认值为 "NULL"
+ - sqlHash: sql hash值，用于完全匹配，我们会在`fe.audit.log`打印这个值，可选，这个参数和sql只能二选一，默认值为 "NULL"
+ - partitionNum: 一个扫描节点会扫描的最大partition数量，默认值为0L
+ - tabletNum: 一个扫描节点会扫描的最大扽tablet数量，默认值为0L
Review comment:
Modified.
##########
File path: docs/zh-CN/administrator-guide/block-rule/sql-block.md
##########
@@ -51,15 +59,27 @@ PROPERTIES(
mysql> select * from order_analysis;
ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule
```
+
+```sql
+CREATE SQL_BLOCK_RULE test_rule2 PROPERTIES("partitionNum" = "30", "cardinality"="10000000000","global"="false","enable"="true")
Review comment:
Added explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] DarvenDuanXJ commented on a change in pull request #7403: [feature] sql-block-rule: add partition_num, tablet_num, cardinality in SqlBlockRule to block big/slow sql
Posted by GitBox <gi...@apache.org>.
DarvenDuanXJ commented on a change in pull request #7403:
URL: https://github.com/apache/incubator-doris/pull/7403#discussion_r826951267
##########
File path: fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
##########
@@ -357,8 +359,17 @@ public void execute(TUniqueId queryId) throws Exception {
context.getState().setError(e.getMysqlErrorCode(), e.getMessage());
return;
}
- // limitations: partitionNum, tabletNum, cardinality
+ // limitations: partition_num, tablet_num, cardinality
+ List<ScanNode> scanNodeList = planner.getScanNodes();
+ for (ScanNode scanNode : scanNodeList) {
+ if (scanNode instanceof OlapScanNode) {
+ OlapScanNode olapScanNode = (OlapScanNode) scanNode;
+ Catalog.getCurrentCatalog().getSqlBlockRuleMgr().checkLimitaions(olapScanNode.getSelectedPartitionNum().longValue(),
+ olapScanNode.getSelectedTabletsNum(), olapScanNode.getCardinality(), analyzer.getQualifiedUser());
Review comment:
I think analyzer.getQualifiedUser() can be replaced by context.getQualifiedUser()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7403: [feature] sql-block-rule: add partition_num, tablet_num, cardinality in SqlBlockRule to block big/slow sql
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7403:
URL: https://github.com/apache/incubator-doris/pull/7403#issuecomment-1003836709
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7403: [feature] sql-block-rule: add partition_num, tablet_num, cardinality in SqlBlockRule to block big/slow sql
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #7403:
URL: https://github.com/apache/incubator-doris/pull/7403
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] Henry2SS commented on a change in pull request #7403: [feature] sql-block-rule: add partitionNum, tabletNum, cardinality in SqlBlockRule to block big/slow sql
Posted by GitBox <gi...@apache.org>.
Henry2SS commented on a change in pull request #7403:
URL: https://github.com/apache/incubator-doris/pull/7403#discussion_r775320474
##########
File path: docs/zh-CN/administrator-guide/block-rule/sql-block.md
##########
@@ -51,15 +59,27 @@ PROPERTIES(
mysql> select * from order_analysis;
ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule
```
+
+```sql
+CREATE SQL_BLOCK_RULE test_rule2 PROPERTIES("partitionNum" = "30", "cardinality"="10000000000","global"="false","enable"="true")
Review comment:
Modified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org