You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/07/09 08:09:59 UTC
[GitHub] [incubator-doris] stalary opened a new pull request #6192: ADD: support sql block rule
stalary opened a new pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192
## Proposed changes
Support configuring sql_block_rule by user level to deny the specified SQL
## Types of changes
What types of changes does your code introduce to Doris?
_Put an `x` in the boxes that apply_
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation Update (if none of the other choices apply)
- [ ] Code refactor (Modify the code structure, format the code, etc...)
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._
- [x] I have created an issue on (Fix https://github.com/apache/incubator-doris/issues/6103) and described the bug/feature there in detail
- [ ] Compiling and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] If these changes need document changes, I have updated the document
- [ ] Any dependent changes have been merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xinghuayu007 commented on pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
xinghuayu007 commented on pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#issuecomment-878009710
I think this feature is used to block some abnormal sql, for example , sql caused core. Is it need to limit the number of rules, which will occupied a lot of memory. Is is better to store the hash value of a sql, not the sql self, when a sql is very long?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r671698654
##########
File path: docs/zh-CN/administrator-guide/block-rule/sql-block.md
##########
@@ -0,0 +1,45 @@
+---
+{
+"title": "SQL黑名单",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# SQL黑名单
+
+支持按用户配置SQL黑名单,通过正则匹配的方式拒绝指定SQL
+
+## 具体操作
+
+对SQL规则增删改查
+- 创建SQL阻止规则
+ - user:规则生效的用户,default代表所有用户都生效,如果同时命中指定用户和default的规则,default规则优先
+ - sql:匹配规则(基于正则匹配,特殊字符需要转译)
+ - sqlHash: sql hash值,用于完全匹配,我们会在`fe.audit.log`打印这个值
+ - enable:是否开启阻止规则
+> CREATE SQL_BLOCK_RULE test_rule PROPERTIES("user"="default","sql"="select \\* from test_table","sqlHash":null,"enable"="true")
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
morningman commented on pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#issuecomment-878801096
> I think this feature is used to block some abnormal sql, for example , sql caused core. Is it need to limit the number of rules, which will occupied a lot of memory. Is is better to store the hash value of a sql, not the sql self, when a sql is very long?
Using hash to block problematic SQL may also be a practical requirement. In some cases, we need to quickly block some SQL that affects the cluster through the block rules. There are pros and cons to using regular expressions or hash values:
1. It is more flexible for regular expressions, and can match a series of sql through one rule. But the disadvantage is that the threshold for using regular expressions is high, and wrong expressions may result in matching wrong SQL.
2. The disadvantage of Hash value is that it can only perform complete matching, but it takes up less space and matches more quickly. But any change in SQL will cause a change in its hash value, such as spaces, line breaks, and so on. Therefore, I suggest that we can print the hash value corresponding to sql in `fe.audit.log`, so that users can conveniently use the hash value in the log to quickly add block rule.
In summary, we can implement two block rules, one based on hash and the other based on regular expressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r667319506
##########
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/AlterSqlBlockRuleStmt.java
##########
@@ -0,0 +1,84 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.block.SqlBlockRule;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.UserException;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class AlterSqlBlockRuleStmt extends DdlStmt {
+
+ private final String ruleName;
+
+ private String user;
+
+ private String sql;
+
+ private Boolean enable;
+
+ private final Map<String, String> properties;
+
+ public AlterSqlBlockRuleStmt(String ruleName, Map<String, String> properties) {
+ this.ruleName = ruleName;
+ this.properties = properties;
+ }
+
+ @Override
+ public void analyze(Analyzer analyzer) throws UserException {
+ super.analyze(analyzer);
+ // check properties
+ checkProperties(properties);
+ }
+
+ private void checkProperties(Map<String, String> properties) throws UserException {
+ this.user = properties.get(CreateSqlBlockRuleStmt.USER_PROPERTY);
+ // if not default, need check whether user exist
+ if (StringUtils.isNotEmpty(user) && !SqlBlockRule.DEFAULT_USER.equals(user)) {
+ List<String> allUserIdents = Catalog.getCurrentCatalog().getAuth().getAllUserIdents(false).stream().map(UserIdentity::getUser).collect(Collectors.toList());
+ if (!allUserIdents.contains(user)) {
+ throw new AnalysisException(user + " is not exist");
+ }
+ }
+ this.sql = properties.get(CreateSqlBlockRuleStmt.SQL_PROPERTY);
Review comment:
My design is to allow only the `enable` modification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r667320342
##########
File path: fe/fe-core/src/main/java/org/apache/doris/metric/MetricRepo.java
##########
@@ -298,6 +299,9 @@ public Long getValue() {
"total error rows of routine load");
PALO_METRIC_REGISTER.addPaloMetrics(COUNTER_ROUTINE_LOAD_ERROR_ROWS);
+ COUNTER_HIT_SQL_BLOCK_RULE = new LongCounterMetric("counter_hit_sql_blacklist", MetricUnit.ROWS,
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#issuecomment-894982101
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r666983002
##########
File path: fe/fe-core/src/main/cup/sql_parser.cup
##########
@@ -692,6 +693,14 @@ stmt ::=
{: RESULT = stmt; :}
| uninstall_plugin_stmt : stmt
{: RESULT = stmt; :}
+ | create_sql_block_rule_stmt : stmt
+ {: RESULT = stmt; :}
+ | alter_sql_block_rule_stmt : stmt
+ {: RESULT = stmt; :}
+ | drop_sql_block_rule_stmt : stmt
+ {: RESULT = stmt; :}
+ | show_sql_block_rule_stmt : stmt
+ {: RESULT = stmt; :}
Review comment:
I think you can put these 4 stmts to `create_stmt`, `drop_stmt`, `alter_stmt` and `show_stmt`.
##########
File path: fe/fe-core/src/main/java/org/apache/doris/block/SqlBlockRule.java
##########
@@ -0,0 +1,111 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.block;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.common.io.Text;
+import org.apache.doris.common.io.Writable;
+
+import com.google.common.collect.Lists;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+
+public class SqlBlockRule implements Writable {
+
+ public static final String NAME_TYPE = "SQL BLOCK RULE NAME";
+
+ public static final String DEFAULT_USER = "default";
+
+ // the rule name, cluster unique
+ private String name;
+
+ // default stands for all users
+ private String user;
+
+ private String sql;
+
+ // whether to use the rule
+ private Boolean enable;
+
+ public SqlBlockRule(String name) {
+ this.name = name;
+ }
+
+ public SqlBlockRule(String name, String user, String sql, Boolean enable) {
+ this.name = name;
+ this.user = user;
+ this.sql = sql;
+ this.enable = enable;
+ }
+
+ public static SqlBlockRule fromCreateStmt(CreateSqlBlockRuleStmt stmt) {
+ return new SqlBlockRule(stmt.getRuleName(), stmt.getUser(), stmt.getSql(), stmt.isEnable());
+ }
+
+ public static SqlBlockRule fromAlterStmt(AlterSqlBlockRuleStmt stmt) {
+ return new SqlBlockRule(stmt.getRuleName(), stmt.getUser(), stmt.getSql(), stmt.getEnable());
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public String getUser() {
+ return user;
+ }
+
+ public String getSql() {
+ return sql;
+ }
+
+ public Boolean getEnable() {
+ return enable;
+ }
+
+ public void setUser(String user) {
+ this.user = user;
+ }
+
+ public void setSql(String sql) {
+ this.sql = sql;
+ }
+
+ public void setEnable(Boolean enable) {
+ this.enable = enable;
+ }
+
+ public List<String> getShowInfo() {
+ return Lists.newArrayList(this.name, this.user, this.sql, String.valueOf(this.enable));
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
+ Text.writeString(out, name);
Review comment:
Use GSON to serialize
##########
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/AlterSqlBlockRuleStmt.java
##########
@@ -0,0 +1,84 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.block.SqlBlockRule;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.UserException;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class AlterSqlBlockRuleStmt extends DdlStmt {
+
+ private final String ruleName;
+
+ private String user;
+
+ private String sql;
+
+ private Boolean enable;
+
+ private final Map<String, String> properties;
+
+ public AlterSqlBlockRuleStmt(String ruleName, Map<String, String> properties) {
+ this.ruleName = ruleName;
+ this.properties = properties;
+ }
+
+ @Override
+ public void analyze(Analyzer analyzer) throws UserException {
+ super.analyze(analyzer);
+ // check properties
+ checkProperties(properties);
+ }
+
+ private void checkProperties(Map<String, String> properties) throws UserException {
+ this.user = properties.get(CreateSqlBlockRuleStmt.USER_PROPERTY);
+ // if not default, need check whether user exist
+ if (StringUtils.isNotEmpty(user) && !SqlBlockRule.DEFAULT_USER.equals(user)) {
+ List<String> allUserIdents = Catalog.getCurrentCatalog().getAuth().getAllUserIdents(false).stream().map(UserIdentity::getUser).collect(Collectors.toList());
+ if (!allUserIdents.contains(user)) {
+ throw new AnalysisException(user + " is not exist");
Review comment:
```suggestion
throw new AnalysisException(user + " does not exist");
```
##########
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/CreateSqlBlockRuleStmt.java
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.block.SqlBlockRule;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.FeNameFormat;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.util.Util;
+
+import com.google.common.collect.ImmutableSet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/*
+ Create sqlBlockRule statement
+
+ syntax:
+ CREATE SQL_BLOCK_RULE LOAD NAME PROPERTIES
Review comment:
```suggestion
CREATE SQL_BLOCK_RULE `rule_name` PROPERTIES
```
##########
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/AlterSqlBlockRuleStmt.java
##########
@@ -0,0 +1,84 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.block.SqlBlockRule;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.UserException;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class AlterSqlBlockRuleStmt extends DdlStmt {
+
+ private final String ruleName;
+
+ private String user;
+
+ private String sql;
+
+ private Boolean enable;
+
+ private final Map<String, String> properties;
+
+ public AlterSqlBlockRuleStmt(String ruleName, Map<String, String> properties) {
+ this.ruleName = ruleName;
+ this.properties = properties;
+ }
+
+ @Override
+ public void analyze(Analyzer analyzer) throws UserException {
+ super.analyze(analyzer);
+ // check properties
+ checkProperties(properties);
+ }
+
+ private void checkProperties(Map<String, String> properties) throws UserException {
+ this.user = properties.get(CreateSqlBlockRuleStmt.USER_PROPERTY);
+ // if not default, need check whether user exist
+ if (StringUtils.isNotEmpty(user) && !SqlBlockRule.DEFAULT_USER.equals(user)) {
+ List<String> allUserIdents = Catalog.getCurrentCatalog().getAuth().getAllUserIdents(false).stream().map(UserIdentity::getUser).collect(Collectors.toList());
Review comment:
Why not using `PaloAuth.doesUsernameExist()`?
##########
File path: fe/fe-core/src/main/java/org/apache/doris/block/SqlBlockRuleMgr.java
##########
@@ -0,0 +1,228 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.block;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.analysis.DropSqlBlockRuleStmt;
+import org.apache.doris.analysis.ShowSqlBlockRuleStmt;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+public class SqlBlockRuleMgr implements Writable {
+ private static final Logger LOG = LogManager.getLogger(SqlBlockRuleMgr.class);
+
+ private ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private Map<String, List<SqlBlockRule>> userToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ private Map<String, SqlBlockRule> nameToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ private void writeLock() {
+ lock.writeLock().lock();
+ }
+
+ private void writeUnlock() {
+ lock.writeLock().unlock();
+ }
+
+ public boolean existRule(String name) {
+ return nameToSqlBlockRuleMap.containsKey(name);
+ }
+
+ public List<SqlBlockRule> get(ShowSqlBlockRuleStmt stmt) throws AnalysisException {
+ String ruleName = stmt.getRuleName();
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN");
+ }
+ if (StringUtils.isNotEmpty(ruleName)) {
+ if (nameToSqlBlockRuleMap.containsKey(ruleName)) {
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ return Lists.newArrayList(sqlBlockRule);
+ }
+ return Lists.newArrayList();
+ }
+ return Lists.newArrayList(nameToSqlBlockRuleMap.values());
+ }
+
+ public void createSqlBlockRule(CreateSqlBlockRuleStmt stmt) throws UserException {
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)
+ && !Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(),
+ PrivPredicate.OPERATOR)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN/OPERATOR");
+ }
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromCreateStmt(stmt);
+ create(sqlBlockRule);
+ }
+
+ private void create(SqlBlockRule sqlBlockRule) throws DdlException {
+ writeLock();
+ try {
+ String ruleName = sqlBlockRule.getName();
+ if (existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " already create");
+ }
+ Catalog.getCurrentCatalog().getEditLog().logCreateSqlBlockRule(sqlBlockRule);
Review comment:
call `unprotectedAdd(sqlBlockRule);` before calling `logCreateSqlBlockRule()`.
And same as other 3 rule operations.
##########
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/AlterSqlBlockRuleStmt.java
##########
@@ -0,0 +1,84 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.block.SqlBlockRule;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.UserException;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class AlterSqlBlockRuleStmt extends DdlStmt {
+
+ private final String ruleName;
+
+ private String user;
+
+ private String sql;
+
+ private Boolean enable;
+
+ private final Map<String, String> properties;
+
+ public AlterSqlBlockRuleStmt(String ruleName, Map<String, String> properties) {
+ this.ruleName = ruleName;
+ this.properties = properties;
+ }
+
+ @Override
+ public void analyze(Analyzer analyzer) throws UserException {
+ super.analyze(analyzer);
+ // check properties
+ checkProperties(properties);
+ }
+
+ private void checkProperties(Map<String, String> properties) throws UserException {
+ this.user = properties.get(CreateSqlBlockRuleStmt.USER_PROPERTY);
+ // if not default, need check whether user exist
+ if (StringUtils.isNotEmpty(user) && !SqlBlockRule.DEFAULT_USER.equals(user)) {
+ List<String> allUserIdents = Catalog.getCurrentCatalog().getAuth().getAllUserIdents(false).stream().map(UserIdentity::getUser).collect(Collectors.toList());
+ if (!allUserIdents.contains(user)) {
+ throw new AnalysisException(user + " is not exist");
+ }
+ }
+ this.sql = properties.get(CreateSqlBlockRuleStmt.SQL_PROPERTY);
Review comment:
The `CreateSqlBlockRuleStmt.SQL_PROPERTY` property may not be set? Should check it
##########
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/AlterSqlBlockRuleStmt.java
##########
@@ -0,0 +1,84 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.block.SqlBlockRule;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.UserException;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class AlterSqlBlockRuleStmt extends DdlStmt {
+
+ private final String ruleName;
+
+ private String user;
+
+ private String sql;
+
+ private Boolean enable;
+
+ private final Map<String, String> properties;
+
+ public AlterSqlBlockRuleStmt(String ruleName, Map<String, String> properties) {
+ this.ruleName = ruleName;
+ this.properties = properties;
+ }
+
+ @Override
+ public void analyze(Analyzer analyzer) throws UserException {
+ super.analyze(analyzer);
+ // check properties
+ checkProperties(properties);
Review comment:
privilege check is always in `analyze` phase.
And I think only check `admin` priv is enough
##########
File path: fe/fe-core/src/main/java/org/apache/doris/block/SqlBlockRuleMgr.java
##########
@@ -0,0 +1,228 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.block;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.analysis.DropSqlBlockRuleStmt;
+import org.apache.doris.analysis.ShowSqlBlockRuleStmt;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+public class SqlBlockRuleMgr implements Writable {
+ private static final Logger LOG = LogManager.getLogger(SqlBlockRuleMgr.class);
+
+ private ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private Map<String, List<SqlBlockRule>> userToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ private Map<String, SqlBlockRule> nameToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ private void writeLock() {
+ lock.writeLock().lock();
+ }
+
+ private void writeUnlock() {
+ lock.writeLock().unlock();
+ }
+
+ public boolean existRule(String name) {
+ return nameToSqlBlockRuleMap.containsKey(name);
+ }
+
+ public List<SqlBlockRule> get(ShowSqlBlockRuleStmt stmt) throws AnalysisException {
+ String ruleName = stmt.getRuleName();
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN");
+ }
+ if (StringUtils.isNotEmpty(ruleName)) {
+ if (nameToSqlBlockRuleMap.containsKey(ruleName)) {
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ return Lists.newArrayList(sqlBlockRule);
+ }
+ return Lists.newArrayList();
+ }
+ return Lists.newArrayList(nameToSqlBlockRuleMap.values());
+ }
+
+ public void createSqlBlockRule(CreateSqlBlockRuleStmt stmt) throws UserException {
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)
+ && !Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(),
+ PrivPredicate.OPERATOR)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN/OPERATOR");
+ }
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromCreateStmt(stmt);
+ create(sqlBlockRule);
+ }
+
+ private void create(SqlBlockRule sqlBlockRule) throws DdlException {
+ writeLock();
+ try {
+ String ruleName = sqlBlockRule.getName();
+ if (existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " already create");
+ }
+ Catalog.getCurrentCatalog().getEditLog().logCreateSqlBlockRule(sqlBlockRule);
+ unprotectedAdd(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayCreate(SqlBlockRule sqlBlockRule) {
+ unprotectedAdd(sqlBlockRule);
+ LOG.info("replay create sql block rule: {}", sqlBlockRule);
+ }
+
+ public void alterSqlBlockRule(AlterSqlBlockRuleStmt stmt) throws UserException {
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)
+ && !Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(),
+ PrivPredicate.OPERATOR)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN/OPERATOR");
+ }
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromAlterStmt(stmt);
+ alter(sqlBlockRule);
+ }
+
+ private void alter(SqlBlockRule sqlBlockRule) throws DdlException {
+ writeLock();
+ try {
+ String ruleName = sqlBlockRule.getName();
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ SqlBlockRule originRule = nameToSqlBlockRuleMap.get(ruleName);
+ if (StringUtils.isEmpty(sqlBlockRule.getUser())) {
+ sqlBlockRule.setUser(originRule.getUser());
+ }
+ if (StringUtils.isEmpty(sqlBlockRule.getSql())) {
+ sqlBlockRule.setSql(originRule.getSql());
+ }
+ if (sqlBlockRule.getEnable() == null) {
+ sqlBlockRule.setEnable(originRule.getEnable());
+ }
+ Catalog.getCurrentCatalog().getEditLog().logAlterSqlBlockRule(sqlBlockRule);
+ unprotectedUpdate(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayAlter(SqlBlockRule sqlBlockRule) {
+ unprotectedUpdate(sqlBlockRule);
+ LOG.info("replay alter sql block rule: {}", sqlBlockRule);
+ }
+
+ public void unprotectedUpdate(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ List<SqlBlockRule> sqlBlockRules = userToSqlBlockRuleMap.getOrDefault(sqlBlockRule.getUser(), new ArrayList<>());
+ sqlBlockRules.removeIf(rule -> sqlBlockRule.getName().equals(rule.getName()));
+ sqlBlockRules.add(sqlBlockRule);
+ userToSqlBlockRuleMap.put(sqlBlockRule.getUser(), sqlBlockRules);
+ }
+
+ public void unprotectedAdd(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ List<SqlBlockRule> sqlBlockRules = userToSqlBlockRuleMap.getOrDefault(sqlBlockRule.getUser(), new ArrayList<>());
+ sqlBlockRules.add(sqlBlockRule);
+ userToSqlBlockRuleMap.put(sqlBlockRule.getUser(), sqlBlockRules);
+ }
+
+ public void dropSqlBlockRule(DropSqlBlockRuleStmt stmt) throws UserException {
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)
+ && !Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(),
+ PrivPredicate.OPERATOR)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN/OPERATOR");
+ }
+ List<String> ruleNames = stmt.getRuleNames();
+ drop(ruleNames);
+ }
+
+ public void drop(List<String> ruleNames) throws DdlException {
+ writeLock();
+ try {
+ for (String ruleName : ruleNames) {
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ if (sqlBlockRule == null) {
+ continue;
+ }
+ Catalog.getCurrentCatalog().getEditLog().logDropSqlBlockRule(sqlBlockRule);
+ unprotectedDrop(sqlBlockRule);
+ }
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayDrop(SqlBlockRule sqlBlockRule) {
+ unprotectedDrop(sqlBlockRule);
+ LOG.info("replay drop sql block rule: {}", sqlBlockRule);
+ }
+
+ public void unprotectedDrop(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.remove(sqlBlockRule.getName());
+ List<SqlBlockRule> sqlBlockRules = userToSqlBlockRuleMap.get(sqlBlockRule.getUser());
+ sqlBlockRules.removeIf(rule -> sqlBlockRule.getName().equals(rule.getName()));
+ userToSqlBlockRuleMap.put(sqlBlockRule.getUser(), sqlBlockRules);
+ }
+
+ public Map<String, List<SqlBlockRule>> getUserToSqlBlockRuleMap() {
+ return userToSqlBlockRuleMap;
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
Review comment:
Use GSON serde.
##########
File path: fe/fe-core/src/main/java/org/apache/doris/metric/MetricRepo.java
##########
@@ -298,6 +299,9 @@ public Long getValue() {
"total error rows of routine load");
PALO_METRIC_REGISTER.addPaloMetrics(COUNTER_ROUTINE_LOAD_ERROR_ROWS);
+ COUNTER_HIT_SQL_BLOCK_RULE = new LongCounterMetric("counter_hit_sql_blacklist", MetricUnit.ROWS,
Review comment:
```suggestion
COUNTER_HIT_SQL_BLOCK_RULE = new LongCounterMetric("counter_hit_sql_block_rule", MetricUnit.ROWS,
```
##########
File path: fe/fe-core/src/main/java/org/apache/doris/block/SqlBlockRule.java
##########
@@ -0,0 +1,111 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.block;
Review comment:
The pacakge name can be `org.apache.doris.sqlblock`?
##########
File path: fe/fe-core/src/main/java/org/apache/doris/metric/MetricRepo.java
##########
@@ -298,6 +299,9 @@ public Long getValue() {
"total error rows of routine load");
PALO_METRIC_REGISTER.addPaloMetrics(COUNTER_ROUTINE_LOAD_ERROR_ROWS);
+ COUNTER_HIT_SQL_BLOCK_RULE = new LongCounterMetric("counter_hit_sql_blacklist", MetricUnit.ROWS,
+ "total hit sql blacklist query");
Review comment:
```suggestion
"total hit sql block rule query");
```
##########
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
##########
@@ -4894,6 +4912,10 @@ public RoutineLoadManager getRoutineLoadManager() {
return routineLoadManager;
}
+ public SqlBlockRuleMgr getSqlBlocklistMgr() {
Review comment:
```suggestion
public SqlBlockRuleMgr getSqlBlockRuleMgr() {
```
##########
File path: fe/fe-core/src/test/java/org/apache/doris/qe/StmtExecutorTest.java
##########
@@ -731,5 +733,18 @@ public void testUseFail(@Mocked UseStmt useStmt, @Mocked SqlParser parser) throw
Assert.assertEquals(QueryState.MysqlStateType.ERR, state.getStateType());
}
+
+ @Test(expected = AnalysisException.class)
Review comment:
The unit test is too few...
where is the create/alter/drop/show sql block rule unit test?
##########
File path: fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
##########
@@ -1169,5 +1178,33 @@ private void handleExportStmt() throws Exception {
private List<PrimitiveType> exprToType(List<Expr> exprs) {
return exprs.stream().map(e -> e.getType().getPrimitiveType()).collect(Collectors.toList());
}
+
+ private void matchSql(QueryStmt queryStmt) throws AnalysisException {
+ Map<String, List<SqlBlockRule>> userToSqlBlockRuleMap = Catalog.getCurrentCatalog().getSqlBlocklistMgr().getUserToSqlBlockRuleMap();
+ // match default rule
+ String sql = queryStmt.getOrigStmt().originStmt;
+ List<SqlBlockRule> defaultRules = userToSqlBlockRuleMap.getOrDefault(SqlBlockRule.DEFAULT_USER, new ArrayList<>());
+ for (SqlBlockRule rule : defaultRules) {
+ matchSql(rule, sql);
+ }
+ // match user rule
+ String user = context.getUserIdentity().getUser();
+ List<SqlBlockRule> userRules = userToSqlBlockRuleMap.getOrDefault(user, new ArrayList<>());
+ for (SqlBlockRule rule : userRules) {
+ matchSql(rule, sql);
+ }
+ }
+
+ @VisibleForTesting
+ public static void matchSql(SqlBlockRule rule, String sql) throws AnalysisException {
+ if (rule.getEnable() != null && rule.getEnable()) {
+ String sqlPattern = rule.getSql();
+ Pattern pattern = Pattern.compile(sqlPattern);
Review comment:
the pattern compilation should be done when the rule is created, so that we don't need to compile it every time.
You should be aware that this rule checking logic is on the critical path of query, so performance is top level consideration.
##########
File path: fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
##########
@@ -1169,5 +1178,33 @@ private void handleExportStmt() throws Exception {
private List<PrimitiveType> exprToType(List<Expr> exprs) {
return exprs.stream().map(e -> e.getType().getPrimitiveType()).collect(Collectors.toList());
}
+
+ private void matchSql(QueryStmt queryStmt) throws AnalysisException {
+ Map<String, List<SqlBlockRule>> userToSqlBlockRuleMap = Catalog.getCurrentCatalog().getSqlBlocklistMgr().getUserToSqlBlockRuleMap();
Review comment:
If you get `userToSqlBlockRuleMap` out of the `SqlBlocklistMgr`, then no lock can protect it.
So all check should be done inside the `SqlBlocklistMgr`
##########
File path: fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
##########
@@ -804,6 +805,21 @@ public static void loadJournal(Catalog catalog, JournalEntity journal) {
catalog.getAlterInstance().replayReplaceTable(log);
break;
}
+ case OperationType.OP_CREATE_SQL_BLOCK_RULE: {
Review comment:
I think you missed the `readFields()` method of `JournalEntity.java`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r671286226
##########
File path: fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
##########
@@ -283,6 +283,9 @@ public void execute(TUniqueId queryId) throws Exception {
}
if (parsedStmt instanceof QueryStmt) {
+ QueryStmt queryStmt = (QueryStmt) this.parsedStmt;
+ // match sql
+ matchSql(queryStmt);
Review comment:
I think all kind of stmt can be blocked? Not only select stmt?
And it can be block in `handleQuery()` of `ConnectProcessor`, so that we even don't need to parse it.
##########
File path: docs/zh-CN/administrator-guide/block-rule/sql-block.md
##########
@@ -0,0 +1,45 @@
+---
+{
+"title": "SQL黑名单",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# SQL黑名单
+
+支持按用户配置SQL黑名单,通过正则匹配的方式拒绝指定SQL
+
+## 具体操作
+
+对SQL规则增删改查
+- 创建SQL阻止规则
+ - user:规则生效的用户,default代表所有用户都生效,如果同时命中指定用户和default的规则,default规则优先
+ - sql:匹配规则(基于正则匹配,特殊字符需要转译)
+ - sqlHash: sql hash值,用于完全匹配,我们会在`fe.audit.log`打印这个值
+ - enable:是否开启阻止规则
+> CREATE SQL_BLOCK_RULE test_rule PROPERTIES("user"="default","sql"="select \\* from test_table","sqlHash":null,"enable"="true")
Review comment:
This markdown format seems not right...
For code part, it should be wrapped by ``.
##########
File path: fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
##########
@@ -0,0 +1,239 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.blockrule;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.analysis.DropSqlBlockRuleStmt;
+import org.apache.doris.analysis.ShowSqlBlockRuleStmt;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.metric.MetricRepo;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Pattern;
+
+public class SqlBlockRuleMgr implements Writable {
Review comment:
SQL block rule is a "per user" property, so it is better to put it to the `UserProperty`?
##########
File path: fe/fe-core/src/main/cup/sql_parser.cup
##########
@@ -1827,6 +1837,10 @@ drop_stmt ::=
{:
RESULT = new DropResourceStmt(resourceName);
:}
+ | KW_DROP KW_SQL_BLOCK_RULE ident_list:ruleNames
Review comment:
Can different user have same rule name? If yes, how to handle it? If no, how to check it?
Maybe you should write like: `drop sql_block_rule FROM username ...`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r667320143
##########
File path: fe/fe-core/src/main/java/org/apache/doris/block/SqlBlockRuleMgr.java
##########
@@ -0,0 +1,228 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.block;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.analysis.DropSqlBlockRuleStmt;
+import org.apache.doris.analysis.ShowSqlBlockRuleStmt;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+public class SqlBlockRuleMgr implements Writable {
+ private static final Logger LOG = LogManager.getLogger(SqlBlockRuleMgr.class);
+
+ private ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private Map<String, List<SqlBlockRule>> userToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ private Map<String, SqlBlockRule> nameToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ private void writeLock() {
+ lock.writeLock().lock();
+ }
+
+ private void writeUnlock() {
+ lock.writeLock().unlock();
+ }
+
+ public boolean existRule(String name) {
+ return nameToSqlBlockRuleMap.containsKey(name);
+ }
+
+ public List<SqlBlockRule> get(ShowSqlBlockRuleStmt stmt) throws AnalysisException {
+ String ruleName = stmt.getRuleName();
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN");
+ }
+ if (StringUtils.isNotEmpty(ruleName)) {
+ if (nameToSqlBlockRuleMap.containsKey(ruleName)) {
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ return Lists.newArrayList(sqlBlockRule);
+ }
+ return Lists.newArrayList();
+ }
+ return Lists.newArrayList(nameToSqlBlockRuleMap.values());
+ }
+
+ public void createSqlBlockRule(CreateSqlBlockRuleStmt stmt) throws UserException {
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)
+ && !Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(),
+ PrivPredicate.OPERATOR)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN/OPERATOR");
+ }
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromCreateStmt(stmt);
+ create(sqlBlockRule);
+ }
+
+ private void create(SqlBlockRule sqlBlockRule) throws DdlException {
+ writeLock();
+ try {
+ String ruleName = sqlBlockRule.getName();
+ if (existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " already create");
+ }
+ Catalog.getCurrentCatalog().getEditLog().logCreateSqlBlockRule(sqlBlockRule);
Review comment:
OK, thanks for the remind
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r677884051
##########
File path: fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.blockrule;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.analysis.DropSqlBlockRuleStmt;
+import org.apache.doris.analysis.ShowSqlBlockRuleStmt;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.metric.MetricRepo;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+public class SqlBlockRuleMgr implements Writable {
+ private static final Logger LOG = LogManager.getLogger(SqlBlockRuleMgr.class);
+
+ private ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private Map<String, SqlBlockRule> nameToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ private Map<String, Pattern> sqlPatternMap = Maps.newConcurrentMap();
+
+ private void writeLock() {
+ lock.writeLock().lock();
+ }
+
+ private void writeUnlock() {
+ lock.writeLock().unlock();
+ }
+
+ public boolean existRule(String name) {
+ return nameToSqlBlockRuleMap.containsKey(name);
+ }
+
+ public List<SqlBlockRule> get(ShowSqlBlockRuleStmt stmt) throws AnalysisException {
+ String ruleName = stmt.getRuleName();
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN");
+ }
+ if (StringUtils.isNotEmpty(ruleName)) {
+ if (nameToSqlBlockRuleMap.containsKey(ruleName)) {
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ return Lists.newArrayList(sqlBlockRule);
+ }
+ return Lists.newArrayList();
+ }
+ return Lists.newArrayList(nameToSqlBlockRuleMap.values());
+ }
+
+ public void createSqlBlockRule(CreateSqlBlockRuleStmt stmt) throws UserException {
+ writeLock();
+ try {
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromCreateStmt(stmt);
+ String ruleName = sqlBlockRule.getName();
+ if (existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " already create");
+ }
+ unprotectedAdd(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logCreateSqlBlockRule(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayCreate(SqlBlockRule sqlBlockRule) {
+ unprotectedAdd(sqlBlockRule);
+ LOG.info("replay create sql block rule: {}", sqlBlockRule);
+ }
+
+ public void alterSqlBlockRule(AlterSqlBlockRuleStmt stmt) throws DdlException {
+ writeLock();
+ try {
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromAlterStmt(stmt);
+ String ruleName = sqlBlockRule.getName();
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ SqlBlockRule originRule = nameToSqlBlockRuleMap.get(ruleName);
+ if (StringUtils.isEmpty(sqlBlockRule.getSql())) {
+ sqlBlockRule.setSql(originRule.getSql());
+ }
+ if (StringUtils.isEmpty(sqlBlockRule.getSqlHash())) {
+ sqlBlockRule.setSqlHash(originRule.getSqlHash());
+ }
+ if (sqlBlockRule.getGlobal() == null) {
+ sqlBlockRule.setGlobal(originRule.getGlobal());
+ }
+ if (sqlBlockRule.getEnable() == null) {
+ sqlBlockRule.setEnable(originRule.getEnable());
+ }
+ unprotectedUpdate(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logAlterSqlBlockRule(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayAlter(SqlBlockRule sqlBlockRule) {
+ unprotectedUpdate(sqlBlockRule);
+ LOG.info("replay alter sql block rule: {}", sqlBlockRule);
+ }
+
+ public void unprotectedUpdate(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ }
+
+ public void unprotectedAdd(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ String sql = sqlBlockRule.getSql();
+ if (StringUtils.isNotEmpty(sql)) {
+ sqlPatternMap.put(sql, Pattern.compile(sql));
+ }
+ }
+
+ public void dropSqlBlockRule(DropSqlBlockRuleStmt stmt) throws DdlException {
+ writeLock();
+ try {
+ List<String> ruleNames = stmt.getRuleNames();
+ for (String ruleName : ruleNames) {
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ if (sqlBlockRule == null) {
+ continue;
+ }
+ unprotectedDrop(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logDropSqlBlockRule(sqlBlockRule);
+ }
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayDrop(SqlBlockRule sqlBlockRule) {
+ unprotectedDrop(sqlBlockRule);
+ LOG.info("replay drop sql block rule: {}", sqlBlockRule);
+ }
+
+ public void unprotectedDrop(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.remove(sqlBlockRule.getName());
+ // todo: remove UserProperty
+ }
+
+ public void matchSql(String sql, String user) throws AnalysisException {
+ // match global rule
+ List<SqlBlockRule> globalRules = nameToSqlBlockRuleMap.values().stream().filter(SqlBlockRule::getGlobal).collect(Collectors.toList());
+ for (SqlBlockRule rule : globalRules) {
+ Pattern sqlPattern = sqlPatternMap.get(rule.getSql());
+ matchSql(rule, sql, sqlPattern);
+ }
+ // match user rule
+ String binSqlBlockRules = Catalog.getCurrentCatalog().getAuth().getBindSqlBlockRules(user);
+ if (StringUtils.isNotEmpty(binSqlBlockRules)) {
+ String[] split = binSqlBlockRules.split(",");
+ for (String ruleName : split) {
+ SqlBlockRule rule = nameToSqlBlockRuleMap.get(ruleName);
+ Pattern sqlPattern = sqlPatternMap.get(rule.getSql());
+ matchSql(rule, sql, sqlPattern);
+ }
+ }
+ }
+
+ @VisibleForTesting
+ public static void matchSql(SqlBlockRule rule, String sql, Pattern sqlPattern) throws AnalysisException {
+ if (rule.getEnable() != null && rule.getEnable()) {
+ String sqlHash = rule.getSqlHash();
+ if (sqlHash != null && sqlHash.equals(DigestUtils.md5Hex(sql))) {
+ MetricRepo.COUNTER_HIT_SQL_BLOCK_RULE.increase(1L);
+ throw new AnalysisException("sql match hash sql block rule: " + rule.getName());
+ }
Review comment:
Yes, only one is allowed. I'll change this later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r676148390
##########
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/AlterSqlBlockRuleStmt.java
##########
@@ -0,0 +1,91 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.Map;
+
+public class AlterSqlBlockRuleStmt extends DdlStmt {
+
+ private final String ruleName;
+
+ private String sql;
+
+ private String sqlHash;
+
+ private Boolean global;
+
+ private Boolean enable;
+
+ private final Map<String, String> properties;
+
+ public AlterSqlBlockRuleStmt(String ruleName, Map<String, String> properties) {
+ this.ruleName = ruleName;
+ this.properties = properties;
+ }
+
+ @Override
+ public void analyze(Analyzer analyzer) throws UserException {
+ super.analyze(analyzer);
+ // check auth
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN");
+ }
+ // check properties
+ CreateSqlBlockRuleStmt.checkCommonProperties(properties);
+ setProperties(properties);
+ }
+
+ private void setProperties(Map<String, String> properties) {
+ this.sql = properties.get(CreateSqlBlockRuleStmt.SQL_PROPERTY);
+ this.sqlHash = properties.get(CreateSqlBlockRuleStmt.SQL_HASH_PROPERTY);
+ // allow null, represents no modification
+ String globalStr = properties.get(CreateSqlBlockRuleStmt.GLOBAL_PROPERTY);
+ this.global = StringUtils.isNotEmpty(globalStr) ? Boolean.parseBoolean(globalStr) : null;
+ String enableStr = properties.get(CreateSqlBlockRuleStmt.ENABLE_PROPERTY);
+ this.enable = StringUtils.isNotEmpty(enableStr) ? Boolean.parseBoolean(enableStr) : null;
+ }
+
+ public String getRuleName() {
+ return ruleName;
+ }
+
+ public String getSql() {
+ return sql;
+ }
+
+ public Boolean getGlobal() {
+ return global;
+ }
+
+ public Boolean getEnable() {
+ return enable;
+ }
+
+ public String getSqlHash() {
+ return sqlHash;
+ }
Review comment:
Override `toSql()` for all statements
##########
File path: fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
##########
@@ -172,6 +176,13 @@ private void handleQuery() {
ctx.getState().setError("Unsupported character set(UTF-8)");
return;
}
+ try {
+ Catalog.getCurrentCatalog().getSqlBlockRuleMgr().matchSql(originStmt, ctx.getQualifiedUser());
+ } catch (AnalysisException e) {
+ LOG.error(e.getMessage());
Review comment:
use warn log
##########
File path: fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.blockrule;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.analysis.DropSqlBlockRuleStmt;
+import org.apache.doris.analysis.ShowSqlBlockRuleStmt;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.metric.MetricRepo;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+public class SqlBlockRuleMgr implements Writable {
+ private static final Logger LOG = LogManager.getLogger(SqlBlockRuleMgr.class);
+
+ private ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private Map<String, SqlBlockRule> nameToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ private Map<String, Pattern> sqlPatternMap = Maps.newConcurrentMap();
+
+ private void writeLock() {
+ lock.writeLock().lock();
+ }
+
+ private void writeUnlock() {
+ lock.writeLock().unlock();
+ }
+
+ public boolean existRule(String name) {
+ return nameToSqlBlockRuleMap.containsKey(name);
+ }
+
+ public List<SqlBlockRule> get(ShowSqlBlockRuleStmt stmt) throws AnalysisException {
+ String ruleName = stmt.getRuleName();
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN");
+ }
+ if (StringUtils.isNotEmpty(ruleName)) {
+ if (nameToSqlBlockRuleMap.containsKey(ruleName)) {
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ return Lists.newArrayList(sqlBlockRule);
+ }
+ return Lists.newArrayList();
+ }
+ return Lists.newArrayList(nameToSqlBlockRuleMap.values());
+ }
+
+ public void createSqlBlockRule(CreateSqlBlockRuleStmt stmt) throws UserException {
+ writeLock();
+ try {
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromCreateStmt(stmt);
+ String ruleName = sqlBlockRule.getName();
+ if (existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " already create");
+ }
+ unprotectedAdd(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logCreateSqlBlockRule(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayCreate(SqlBlockRule sqlBlockRule) {
+ unprotectedAdd(sqlBlockRule);
+ LOG.info("replay create sql block rule: {}", sqlBlockRule);
+ }
+
+ public void alterSqlBlockRule(AlterSqlBlockRuleStmt stmt) throws DdlException {
+ writeLock();
+ try {
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromAlterStmt(stmt);
+ String ruleName = sqlBlockRule.getName();
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ SqlBlockRule originRule = nameToSqlBlockRuleMap.get(ruleName);
+ if (StringUtils.isEmpty(sqlBlockRule.getSql())) {
+ sqlBlockRule.setSql(originRule.getSql());
+ }
+ if (StringUtils.isEmpty(sqlBlockRule.getSqlHash())) {
+ sqlBlockRule.setSqlHash(originRule.getSqlHash());
+ }
+ if (sqlBlockRule.getGlobal() == null) {
+ sqlBlockRule.setGlobal(originRule.getGlobal());
+ }
+ if (sqlBlockRule.getEnable() == null) {
+ sqlBlockRule.setEnable(originRule.getEnable());
+ }
+ unprotectedUpdate(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logAlterSqlBlockRule(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayAlter(SqlBlockRule sqlBlockRule) {
+ unprotectedUpdate(sqlBlockRule);
+ LOG.info("replay alter sql block rule: {}", sqlBlockRule);
+ }
+
+ public void unprotectedUpdate(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ }
+
+ public void unprotectedAdd(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ String sql = sqlBlockRule.getSql();
+ if (StringUtils.isNotEmpty(sql)) {
+ sqlPatternMap.put(sql, Pattern.compile(sql));
+ }
+ }
+
+ public void dropSqlBlockRule(DropSqlBlockRuleStmt stmt) throws DdlException {
+ writeLock();
+ try {
+ List<String> ruleNames = stmt.getRuleNames();
+ for (String ruleName : ruleNames) {
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ if (sqlBlockRule == null) {
+ continue;
+ }
+ unprotectedDrop(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logDropSqlBlockRule(sqlBlockRule);
+ }
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayDrop(SqlBlockRule sqlBlockRule) {
+ unprotectedDrop(sqlBlockRule);
+ LOG.info("replay drop sql block rule: {}", sqlBlockRule);
+ }
+
+ public void unprotectedDrop(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.remove(sqlBlockRule.getName());
+ // todo: remove UserProperty
+ }
+
+ public void matchSql(String sql, String user) throws AnalysisException {
+ // match global rule
+ List<SqlBlockRule> globalRules = nameToSqlBlockRuleMap.values().stream().filter(SqlBlockRule::getGlobal).collect(Collectors.toList());
+ for (SqlBlockRule rule : globalRules) {
+ Pattern sqlPattern = sqlPatternMap.get(rule.getSql());
+ matchSql(rule, sql, sqlPattern);
+ }
+ // match user rule
+ String binSqlBlockRules = Catalog.getCurrentCatalog().getAuth().getBindSqlBlockRules(user);
+ if (StringUtils.isNotEmpty(binSqlBlockRules)) {
+ String[] split = binSqlBlockRules.split(",");
+ for (String ruleName : split) {
+ SqlBlockRule rule = nameToSqlBlockRuleMap.get(ruleName);
+ Pattern sqlPattern = sqlPatternMap.get(rule.getSql());
+ matchSql(rule, sql, sqlPattern);
+ }
+ }
+ }
+
+ @VisibleForTesting
+ public static void matchSql(SqlBlockRule rule, String sql, Pattern sqlPattern) throws AnalysisException {
+ if (rule.getEnable() != null && rule.getEnable()) {
+ String sqlHash = rule.getSqlHash();
+ if (sqlHash != null && sqlHash.equals(DigestUtils.md5Hex(sql))) {
+ MetricRepo.COUNTER_HIT_SQL_BLOCK_RULE.increase(1L);
+ throw new AnalysisException("sql match hash sql block rule: " + rule.getName());
+ }
+ if (sqlPattern != null && sqlPattern.matcher(sql).find()) {
+ MetricRepo.COUNTER_HIT_SQL_BLOCK_RULE.increase(1L);
+ throw new AnalysisException("sql match regex sql block rule: " + rule.getName());
+ }
+ }
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
+ out.writeInt(nameToSqlBlockRuleMap.size());
Review comment:
Use GSON
##########
File path: fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.blockrule;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.analysis.DropSqlBlockRuleStmt;
+import org.apache.doris.analysis.ShowSqlBlockRuleStmt;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.metric.MetricRepo;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+public class SqlBlockRuleMgr implements Writable {
+ private static final Logger LOG = LogManager.getLogger(SqlBlockRuleMgr.class);
+
+ private ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private Map<String, SqlBlockRule> nameToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ private Map<String, Pattern> sqlPatternMap = Maps.newConcurrentMap();
+
+ private void writeLock() {
+ lock.writeLock().lock();
+ }
+
+ private void writeUnlock() {
+ lock.writeLock().unlock();
+ }
+
+ public boolean existRule(String name) {
+ return nameToSqlBlockRuleMap.containsKey(name);
+ }
+
+ public List<SqlBlockRule> get(ShowSqlBlockRuleStmt stmt) throws AnalysisException {
+ String ruleName = stmt.getRuleName();
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN");
+ }
+ if (StringUtils.isNotEmpty(ruleName)) {
+ if (nameToSqlBlockRuleMap.containsKey(ruleName)) {
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ return Lists.newArrayList(sqlBlockRule);
+ }
+ return Lists.newArrayList();
+ }
+ return Lists.newArrayList(nameToSqlBlockRuleMap.values());
+ }
+
+ public void createSqlBlockRule(CreateSqlBlockRuleStmt stmt) throws UserException {
+ writeLock();
+ try {
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromCreateStmt(stmt);
+ String ruleName = sqlBlockRule.getName();
+ if (existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " already create");
+ }
+ unprotectedAdd(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logCreateSqlBlockRule(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayCreate(SqlBlockRule sqlBlockRule) {
+ unprotectedAdd(sqlBlockRule);
+ LOG.info("replay create sql block rule: {}", sqlBlockRule);
+ }
+
+ public void alterSqlBlockRule(AlterSqlBlockRuleStmt stmt) throws DdlException {
+ writeLock();
+ try {
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromAlterStmt(stmt);
+ String ruleName = sqlBlockRule.getName();
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ SqlBlockRule originRule = nameToSqlBlockRuleMap.get(ruleName);
+ if (StringUtils.isEmpty(sqlBlockRule.getSql())) {
+ sqlBlockRule.setSql(originRule.getSql());
+ }
+ if (StringUtils.isEmpty(sqlBlockRule.getSqlHash())) {
+ sqlBlockRule.setSqlHash(originRule.getSqlHash());
+ }
+ if (sqlBlockRule.getGlobal() == null) {
+ sqlBlockRule.setGlobal(originRule.getGlobal());
+ }
+ if (sqlBlockRule.getEnable() == null) {
+ sqlBlockRule.setEnable(originRule.getEnable());
+ }
+ unprotectedUpdate(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logAlterSqlBlockRule(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayAlter(SqlBlockRule sqlBlockRule) {
+ unprotectedUpdate(sqlBlockRule);
+ LOG.info("replay alter sql block rule: {}", sqlBlockRule);
+ }
+
+ public void unprotectedUpdate(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ }
+
+ public void unprotectedAdd(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ String sql = sqlBlockRule.getSql();
+ if (StringUtils.isNotEmpty(sql)) {
+ sqlPatternMap.put(sql, Pattern.compile(sql));
+ }
+ }
+
+ public void dropSqlBlockRule(DropSqlBlockRuleStmt stmt) throws DdlException {
+ writeLock();
+ try {
+ List<String> ruleNames = stmt.getRuleNames();
+ for (String ruleName : ruleNames) {
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ if (sqlBlockRule == null) {
+ continue;
+ }
Review comment:
better to check if all rules which need to be dropped exists, and then drop them together with only one edit log, to make the operation atomically.
And for drop operation, we don't need to write whole rule, just rule name is enough.
##########
File path: fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRule.java
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.blockrule;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.common.io.Text;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.persist.gson.GsonUtils;
+
+import com.google.common.collect.Lists;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+
+public class SqlBlockRule implements Writable {
+
+ public static final String NAME_TYPE = "SQL BLOCK RULE NAME";
+
+ public static final String DEFAULT_USER = "default";
+
+ // the rule name, cluster unique
+ private String name;
Review comment:
Need to add annotation `@SerializedName` to all fields need to be serialized.
##########
File path: fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.blockrule;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.analysis.DropSqlBlockRuleStmt;
+import org.apache.doris.analysis.ShowSqlBlockRuleStmt;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.metric.MetricRepo;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+public class SqlBlockRuleMgr implements Writable {
+ private static final Logger LOG = LogManager.getLogger(SqlBlockRuleMgr.class);
+
+ private ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private Map<String, SqlBlockRule> nameToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ private Map<String, Pattern> sqlPatternMap = Maps.newConcurrentMap();
+
+ private void writeLock() {
+ lock.writeLock().lock();
+ }
+
+ private void writeUnlock() {
+ lock.writeLock().unlock();
+ }
+
+ public boolean existRule(String name) {
+ return nameToSqlBlockRuleMap.containsKey(name);
+ }
+
+ public List<SqlBlockRule> get(ShowSqlBlockRuleStmt stmt) throws AnalysisException {
+ String ruleName = stmt.getRuleName();
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)) {
Review comment:
Privilege should be checked at analysis phase.
##########
File path: fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.blockrule;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.analysis.DropSqlBlockRuleStmt;
+import org.apache.doris.analysis.ShowSqlBlockRuleStmt;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.metric.MetricRepo;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+public class SqlBlockRuleMgr implements Writable {
+ private static final Logger LOG = LogManager.getLogger(SqlBlockRuleMgr.class);
+
+ private ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private Map<String, SqlBlockRule> nameToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ private Map<String, Pattern> sqlPatternMap = Maps.newConcurrentMap();
+
+ private void writeLock() {
+ lock.writeLock().lock();
+ }
+
+ private void writeUnlock() {
+ lock.writeLock().unlock();
+ }
+
+ public boolean existRule(String name) {
+ return nameToSqlBlockRuleMap.containsKey(name);
+ }
+
+ public List<SqlBlockRule> get(ShowSqlBlockRuleStmt stmt) throws AnalysisException {
Review comment:
This not a good method name...
##########
File path: fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.blockrule;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.analysis.DropSqlBlockRuleStmt;
+import org.apache.doris.analysis.ShowSqlBlockRuleStmt;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.metric.MetricRepo;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+public class SqlBlockRuleMgr implements Writable {
+ private static final Logger LOG = LogManager.getLogger(SqlBlockRuleMgr.class);
+
+ private ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private Map<String, SqlBlockRule> nameToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ private Map<String, Pattern> sqlPatternMap = Maps.newConcurrentMap();
+
+ private void writeLock() {
+ lock.writeLock().lock();
+ }
+
+ private void writeUnlock() {
+ lock.writeLock().unlock();
+ }
+
+ public boolean existRule(String name) {
+ return nameToSqlBlockRuleMap.containsKey(name);
+ }
+
+ public List<SqlBlockRule> get(ShowSqlBlockRuleStmt stmt) throws AnalysisException {
+ String ruleName = stmt.getRuleName();
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN");
+ }
+ if (StringUtils.isNotEmpty(ruleName)) {
+ if (nameToSqlBlockRuleMap.containsKey(ruleName)) {
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ return Lists.newArrayList(sqlBlockRule);
+ }
+ return Lists.newArrayList();
+ }
+ return Lists.newArrayList(nameToSqlBlockRuleMap.values());
+ }
+
+ public void createSqlBlockRule(CreateSqlBlockRuleStmt stmt) throws UserException {
+ writeLock();
+ try {
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromCreateStmt(stmt);
+ String ruleName = sqlBlockRule.getName();
+ if (existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " already create");
+ }
+ unprotectedAdd(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logCreateSqlBlockRule(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayCreate(SqlBlockRule sqlBlockRule) {
+ unprotectedAdd(sqlBlockRule);
+ LOG.info("replay create sql block rule: {}", sqlBlockRule);
+ }
+
+ public void alterSqlBlockRule(AlterSqlBlockRuleStmt stmt) throws DdlException {
+ writeLock();
+ try {
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromAlterStmt(stmt);
+ String ruleName = sqlBlockRule.getName();
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ SqlBlockRule originRule = nameToSqlBlockRuleMap.get(ruleName);
+ if (StringUtils.isEmpty(sqlBlockRule.getSql())) {
+ sqlBlockRule.setSql(originRule.getSql());
+ }
+ if (StringUtils.isEmpty(sqlBlockRule.getSqlHash())) {
+ sqlBlockRule.setSqlHash(originRule.getSqlHash());
+ }
+ if (sqlBlockRule.getGlobal() == null) {
+ sqlBlockRule.setGlobal(originRule.getGlobal());
+ }
+ if (sqlBlockRule.getEnable() == null) {
+ sqlBlockRule.setEnable(originRule.getEnable());
+ }
+ unprotectedUpdate(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logAlterSqlBlockRule(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayAlter(SqlBlockRule sqlBlockRule) {
+ unprotectedUpdate(sqlBlockRule);
+ LOG.info("replay alter sql block rule: {}", sqlBlockRule);
+ }
+
+ public void unprotectedUpdate(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ }
+
+ public void unprotectedAdd(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ String sql = sqlBlockRule.getSql();
+ if (StringUtils.isNotEmpty(sql)) {
+ sqlPatternMap.put(sql, Pattern.compile(sql));
+ }
+ }
+
+ public void dropSqlBlockRule(DropSqlBlockRuleStmt stmt) throws DdlException {
+ writeLock();
+ try {
+ List<String> ruleNames = stmt.getRuleNames();
+ for (String ruleName : ruleNames) {
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ if (sqlBlockRule == null) {
+ continue;
+ }
+ unprotectedDrop(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logDropSqlBlockRule(sqlBlockRule);
+ }
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayDrop(SqlBlockRule sqlBlockRule) {
+ unprotectedDrop(sqlBlockRule);
+ LOG.info("replay drop sql block rule: {}", sqlBlockRule);
+ }
+
+ public void unprotectedDrop(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.remove(sqlBlockRule.getName());
+ // todo: remove UserProperty
+ }
+
+ public void matchSql(String sql, String user) throws AnalysisException {
+ // match global rule
+ List<SqlBlockRule> globalRules = nameToSqlBlockRuleMap.values().stream().filter(SqlBlockRule::getGlobal).collect(Collectors.toList());
+ for (SqlBlockRule rule : globalRules) {
+ Pattern sqlPattern = sqlPatternMap.get(rule.getSql());
+ matchSql(rule, sql, sqlPattern);
+ }
+ // match user rule
+ String binSqlBlockRules = Catalog.getCurrentCatalog().getAuth().getBindSqlBlockRules(user);
+ if (StringUtils.isNotEmpty(binSqlBlockRules)) {
+ String[] split = binSqlBlockRules.split(",");
+ for (String ruleName : split) {
+ SqlBlockRule rule = nameToSqlBlockRuleMap.get(ruleName);
+ Pattern sqlPattern = sqlPatternMap.get(rule.getSql());
+ matchSql(rule, sql, sqlPattern);
+ }
+ }
+ }
+
+ @VisibleForTesting
+ public static void matchSql(SqlBlockRule rule, String sql, Pattern sqlPattern) throws AnalysisException {
+ if (rule.getEnable() != null && rule.getEnable()) {
Review comment:
rule's enable properties should not be null, it user don't set it, fill it with default value.
##########
File path: fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
##########
@@ -143,13 +145,15 @@ private void auditAfterExec(String origStmt, StatementBase parsedStmt, Data.PQue
ctx.getAuditEventBuilder().setFeIp(FrontendOptions.getLocalHostAddress());
- // We put origin query stmt at the end of audit log, for parsing the log more convenient.
+ String sql;
if (!ctx.getState().isQuery() && (parsedStmt != null && parsedStmt.needAuditEncryption())) {
- ctx.getAuditEventBuilder().setStmt(parsedStmt.toSql());
+ sql = parsedStmt.toSql();
} else {
- ctx.getAuditEventBuilder().setStmt(origStmt);
+ sql = origStmt;
}
-
+ ctx.getAuditEventBuilder().setSqlHash(DigestUtils.md5Hex(sql));
Review comment:
calc md5 of SQL once and save it in connect ctx, so that we can reuse it.
##########
File path: docs/zh-CN/administrator-guide/block-rule/sql-block.md
##########
@@ -0,0 +1,59 @@
+---
+{
+"title": "SQL黑名单",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# SQL黑名单
+
+支持按用户配置SQL黑名单,通过正则匹配的方式拒绝指定SQL
+
+## 规则
+
+对SQL规则增删改查
+- 创建SQL阻止规则
+ - sql:匹配规则(基于正则匹配,特殊字符需要转译),可选
+ - sqlHash: sql hash值,用于完全匹配,我们会在`fe.audit.log`打印这个值,可选
+ - global:是否全局(所有用户)生效,默认为false
+ - enable:是否开启阻止规则,默认为true
+```
+CREATE SQL_BLOCK_RULE test_rule PROPERTIES("sql"="select \\* from test_table","sqlHash":null,"enable"="true")
+```
+- 查看已配置的SQL阻止规则,不指定规则名则为查看所有规则
+```
+SHOW SQL_BLOCK_RULE [FOR RULE_NAME]
+```
+- 修改SQL阻止规则,允许对sql/global/enable等每一项进行修改
+```
+ALTER SQL_BLOCK_RULE test_rule PROPERTIES("sql"="select \\* from test_table","enable"="true")
+```
+- 删除SQL阻止规则,支持多规则,以`,`隔开
+```
+DROP SQL_BLOCK_RULE test_rule1,test_rule2
+```
+
+## 用户规则绑定
+如果配置global=false,则需要配置指定用户的规则绑定,多个规则使用`,`分隔
+```
+SET PROPERTY FOR 'jack' 'bind_sql_block_rules' = 'test_rule1,test_rule2'
Review comment:
```suggestion
SET PROPERTY FOR 'jack' 'sql_block_rules' = 'test_rule1,test_rule2'
```
##########
File path: fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.blockrule;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.analysis.DropSqlBlockRuleStmt;
+import org.apache.doris.analysis.ShowSqlBlockRuleStmt;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.metric.MetricRepo;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+public class SqlBlockRuleMgr implements Writable {
+ private static final Logger LOG = LogManager.getLogger(SqlBlockRuleMgr.class);
+
+ private ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private Map<String, SqlBlockRule> nameToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ private Map<String, Pattern> sqlPatternMap = Maps.newConcurrentMap();
+
+ private void writeLock() {
+ lock.writeLock().lock();
+ }
+
+ private void writeUnlock() {
+ lock.writeLock().unlock();
+ }
+
+ public boolean existRule(String name) {
+ return nameToSqlBlockRuleMap.containsKey(name);
+ }
+
+ public List<SqlBlockRule> get(ShowSqlBlockRuleStmt stmt) throws AnalysisException {
+ String ruleName = stmt.getRuleName();
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN");
+ }
+ if (StringUtils.isNotEmpty(ruleName)) {
+ if (nameToSqlBlockRuleMap.containsKey(ruleName)) {
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ return Lists.newArrayList(sqlBlockRule);
+ }
+ return Lists.newArrayList();
+ }
+ return Lists.newArrayList(nameToSqlBlockRuleMap.values());
+ }
+
+ public void createSqlBlockRule(CreateSqlBlockRuleStmt stmt) throws UserException {
+ writeLock();
+ try {
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromCreateStmt(stmt);
+ String ruleName = sqlBlockRule.getName();
+ if (existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " already create");
+ }
+ unprotectedAdd(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logCreateSqlBlockRule(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayCreate(SqlBlockRule sqlBlockRule) {
+ unprotectedAdd(sqlBlockRule);
+ LOG.info("replay create sql block rule: {}", sqlBlockRule);
+ }
+
+ public void alterSqlBlockRule(AlterSqlBlockRuleStmt stmt) throws DdlException {
+ writeLock();
+ try {
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromAlterStmt(stmt);
+ String ruleName = sqlBlockRule.getName();
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ SqlBlockRule originRule = nameToSqlBlockRuleMap.get(ruleName);
+ if (StringUtils.isEmpty(sqlBlockRule.getSql())) {
+ sqlBlockRule.setSql(originRule.getSql());
+ }
+ if (StringUtils.isEmpty(sqlBlockRule.getSqlHash())) {
+ sqlBlockRule.setSqlHash(originRule.getSqlHash());
+ }
+ if (sqlBlockRule.getGlobal() == null) {
+ sqlBlockRule.setGlobal(originRule.getGlobal());
+ }
+ if (sqlBlockRule.getEnable() == null) {
+ sqlBlockRule.setEnable(originRule.getEnable());
+ }
+ unprotectedUpdate(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logAlterSqlBlockRule(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayAlter(SqlBlockRule sqlBlockRule) {
+ unprotectedUpdate(sqlBlockRule);
+ LOG.info("replay alter sql block rule: {}", sqlBlockRule);
+ }
+
+ public void unprotectedUpdate(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ }
+
+ public void unprotectedAdd(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ String sql = sqlBlockRule.getSql();
+ if (StringUtils.isNotEmpty(sql)) {
+ sqlPatternMap.put(sql, Pattern.compile(sql));
+ }
+ }
+
+ public void dropSqlBlockRule(DropSqlBlockRuleStmt stmt) throws DdlException {
+ writeLock();
+ try {
+ List<String> ruleNames = stmt.getRuleNames();
+ for (String ruleName : ruleNames) {
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ if (sqlBlockRule == null) {
+ continue;
+ }
+ unprotectedDrop(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logDropSqlBlockRule(sqlBlockRule);
+ }
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayDrop(SqlBlockRule sqlBlockRule) {
+ unprotectedDrop(sqlBlockRule);
+ LOG.info("replay drop sql block rule: {}", sqlBlockRule);
+ }
+
+ public void unprotectedDrop(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.remove(sqlBlockRule.getName());
+ // todo: remove UserProperty
+ }
+
+ public void matchSql(String sql, String user) throws AnalysisException {
+ // match global rule
+ List<SqlBlockRule> globalRules = nameToSqlBlockRuleMap.values().stream().filter(SqlBlockRule::getGlobal).collect(Collectors.toList());
+ for (SqlBlockRule rule : globalRules) {
+ Pattern sqlPattern = sqlPatternMap.get(rule.getSql());
+ matchSql(rule, sql, sqlPattern);
+ }
+ // match user rule
+ String binSqlBlockRules = Catalog.getCurrentCatalog().getAuth().getBindSqlBlockRules(user);
+ if (StringUtils.isNotEmpty(binSqlBlockRules)) {
+ String[] split = binSqlBlockRules.split(",");
+ for (String ruleName : split) {
+ SqlBlockRule rule = nameToSqlBlockRuleMap.get(ruleName);
+ Pattern sqlPattern = sqlPatternMap.get(rule.getSql());
+ matchSql(rule, sql, sqlPattern);
+ }
+ }
+ }
+
+ @VisibleForTesting
+ public static void matchSql(SqlBlockRule rule, String sql, Pattern sqlPattern) throws AnalysisException {
+ if (rule.getEnable() != null && rule.getEnable()) {
+ String sqlHash = rule.getSqlHash();
+ if (sqlHash != null && sqlHash.equals(DigestUtils.md5Hex(sql))) {
+ MetricRepo.COUNTER_HIT_SQL_BLOCK_RULE.increase(1L);
+ throw new AnalysisException("sql match hash sql block rule: " + rule.getName());
+ }
Review comment:
Do we allow to set sql hash and sql regex in one rule at same time?
I don't think so. So it needs to be checked that only one kind of property can be set.
##########
File path: fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.blockrule;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.analysis.DropSqlBlockRuleStmt;
+import org.apache.doris.analysis.ShowSqlBlockRuleStmt;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.metric.MetricRepo;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+public class SqlBlockRuleMgr implements Writable {
+ private static final Logger LOG = LogManager.getLogger(SqlBlockRuleMgr.class);
+
+ private ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private Map<String, SqlBlockRule> nameToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ private Map<String, Pattern> sqlPatternMap = Maps.newConcurrentMap();
+
+ private void writeLock() {
+ lock.writeLock().lock();
+ }
+
+ private void writeUnlock() {
+ lock.writeLock().unlock();
+ }
+
+ public boolean existRule(String name) {
+ return nameToSqlBlockRuleMap.containsKey(name);
+ }
+
+ public List<SqlBlockRule> get(ShowSqlBlockRuleStmt stmt) throws AnalysisException {
+ String ruleName = stmt.getRuleName();
+ if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)) {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN");
+ }
+ if (StringUtils.isNotEmpty(ruleName)) {
+ if (nameToSqlBlockRuleMap.containsKey(ruleName)) {
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ return Lists.newArrayList(sqlBlockRule);
+ }
+ return Lists.newArrayList();
+ }
+ return Lists.newArrayList(nameToSqlBlockRuleMap.values());
+ }
+
+ public void createSqlBlockRule(CreateSqlBlockRuleStmt stmt) throws UserException {
+ writeLock();
+ try {
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromCreateStmt(stmt);
+ String ruleName = sqlBlockRule.getName();
+ if (existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " already create");
+ }
+ unprotectedAdd(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logCreateSqlBlockRule(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayCreate(SqlBlockRule sqlBlockRule) {
+ unprotectedAdd(sqlBlockRule);
+ LOG.info("replay create sql block rule: {}", sqlBlockRule);
+ }
+
+ public void alterSqlBlockRule(AlterSqlBlockRuleStmt stmt) throws DdlException {
+ writeLock();
+ try {
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromAlterStmt(stmt);
+ String ruleName = sqlBlockRule.getName();
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ SqlBlockRule originRule = nameToSqlBlockRuleMap.get(ruleName);
+ if (StringUtils.isEmpty(sqlBlockRule.getSql())) {
+ sqlBlockRule.setSql(originRule.getSql());
+ }
+ if (StringUtils.isEmpty(sqlBlockRule.getSqlHash())) {
+ sqlBlockRule.setSqlHash(originRule.getSqlHash());
+ }
+ if (sqlBlockRule.getGlobal() == null) {
+ sqlBlockRule.setGlobal(originRule.getGlobal());
+ }
+ if (sqlBlockRule.getEnable() == null) {
+ sqlBlockRule.setEnable(originRule.getEnable());
+ }
+ unprotectedUpdate(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logAlterSqlBlockRule(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayAlter(SqlBlockRule sqlBlockRule) {
+ unprotectedUpdate(sqlBlockRule);
+ LOG.info("replay alter sql block rule: {}", sqlBlockRule);
+ }
+
+ public void unprotectedUpdate(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ }
+
+ public void unprotectedAdd(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ String sql = sqlBlockRule.getSql();
+ if (StringUtils.isNotEmpty(sql)) {
+ sqlPatternMap.put(sql, Pattern.compile(sql));
+ }
+ }
+
+ public void dropSqlBlockRule(DropSqlBlockRuleStmt stmt) throws DdlException {
+ writeLock();
+ try {
+ List<String> ruleNames = stmt.getRuleNames();
+ for (String ruleName : ruleNames) {
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ if (sqlBlockRule == null) {
+ continue;
+ }
+ unprotectedDrop(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logDropSqlBlockRule(sqlBlockRule);
+ }
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayDrop(SqlBlockRule sqlBlockRule) {
+ unprotectedDrop(sqlBlockRule);
+ LOG.info("replay drop sql block rule: {}", sqlBlockRule);
+ }
+
+ public void unprotectedDrop(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.remove(sqlBlockRule.getName());
+ // todo: remove UserProperty
+ }
+
+ public void matchSql(String sql, String user) throws AnalysisException {
+ // match global rule
+ List<SqlBlockRule> globalRules = nameToSqlBlockRuleMap.values().stream().filter(SqlBlockRule::getGlobal).collect(Collectors.toList());
+ for (SqlBlockRule rule : globalRules) {
+ Pattern sqlPattern = sqlPatternMap.get(rule.getSql());
+ matchSql(rule, sql, sqlPattern);
+ }
+ // match user rule
+ String binSqlBlockRules = Catalog.getCurrentCatalog().getAuth().getBindSqlBlockRules(user);
+ if (StringUtils.isNotEmpty(binSqlBlockRules)) {
+ String[] split = binSqlBlockRules.split(",");
+ for (String ruleName : split) {
+ SqlBlockRule rule = nameToSqlBlockRuleMap.get(ruleName);
+ Pattern sqlPattern = sqlPatternMap.get(rule.getSql());
+ matchSql(rule, sql, sqlPattern);
+ }
+ }
+ }
+
+ @VisibleForTesting
+ public static void matchSql(SqlBlockRule rule, String sql, Pattern sqlPattern) throws AnalysisException {
+ if (rule.getEnable() != null && rule.getEnable()) {
+ String sqlHash = rule.getSqlHash();
+ if (sqlHash != null && sqlHash.equals(DigestUtils.md5Hex(sql))) {
Review comment:
the md5 of a sql should only be calculated once for one query.
And since we will record the sql hash in audit log, I think it should be calculated outside and pass md5 of sql to here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r667318757
##########
File path: fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
##########
@@ -1169,5 +1178,33 @@ private void handleExportStmt() throws Exception {
private List<PrimitiveType> exprToType(List<Expr> exprs) {
return exprs.stream().map(e -> e.getType().getPrimitiveType()).collect(Collectors.toList());
}
+
+ private void matchSql(QueryStmt queryStmt) throws AnalysisException {
+ Map<String, List<SqlBlockRule>> userToSqlBlockRuleMap = Catalog.getCurrentCatalog().getSqlBlocklistMgr().getUserToSqlBlockRuleMap();
+ // match default rule
+ String sql = queryStmt.getOrigStmt().originStmt;
+ List<SqlBlockRule> defaultRules = userToSqlBlockRuleMap.getOrDefault(SqlBlockRule.DEFAULT_USER, new ArrayList<>());
+ for (SqlBlockRule rule : defaultRules) {
+ matchSql(rule, sql);
+ }
+ // match user rule
+ String user = context.getUserIdentity().getUser();
+ List<SqlBlockRule> userRules = userToSqlBlockRuleMap.getOrDefault(user, new ArrayList<>());
+ for (SqlBlockRule rule : userRules) {
+ matchSql(rule, sql);
+ }
+ }
+
+ @VisibleForTesting
+ public static void matchSql(SqlBlockRule rule, String sql) throws AnalysisException {
+ if (rule.getEnable() != null && rule.getEnable()) {
+ String sqlPattern = rule.getSql();
+ Pattern pattern = Pattern.compile(sqlPattern);
Review comment:
Yes, there is a performance problem. Let me fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r667319554
##########
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/CreateSqlBlockRuleStmt.java
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.block.SqlBlockRule;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.FeNameFormat;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.util.Util;
+
+import com.google.common.collect.ImmutableSet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/*
+ Create sqlBlockRule statement
+
+ syntax:
+ CREATE SQL_BLOCK_RULE LOAD NAME PROPERTIES
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r668439080
##########
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/AlterSqlBlockRuleStmt.java
##########
@@ -0,0 +1,84 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.block.SqlBlockRule;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.UserException;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class AlterSqlBlockRuleStmt extends DdlStmt {
+
+ private final String ruleName;
+
+ private String user;
+
+ private String sql;
+
+ private Boolean enable;
+
+ private final Map<String, String> properties;
+
+ public AlterSqlBlockRuleStmt(String ruleName, Map<String, String> properties) {
+ this.ruleName = ruleName;
+ this.properties = properties;
+ }
+
+ @Override
+ public void analyze(Analyzer analyzer) throws UserException {
+ super.analyze(analyzer);
+ // check properties
+ checkProperties(properties);
Review comment:
AlterRoutineLoadStmt is a little bit specially because it need table info, which not contains in alter stmt.
But for this stmt, we can check ADMIN priv here.
Of cause `checkProperties` is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r667318831
##########
File path: fe/fe-core/src/test/java/org/apache/doris/qe/StmtExecutorTest.java
##########
@@ -731,5 +733,18 @@ public void testUseFail(@Mocked UseStmt useStmt, @Mocked SqlParser parser) throw
Assert.assertEquals(QueryState.MysqlStateType.ERR, state.getStateType());
}
+
+ @Test(expected = AnalysisException.class)
Review comment:
I will submit it later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r667317736
##########
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/AlterSqlBlockRuleStmt.java
##########
@@ -0,0 +1,84 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.block.SqlBlockRule;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.UserException;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class AlterSqlBlockRuleStmt extends DdlStmt {
+
+ private final String ruleName;
+
+ private String user;
+
+ private String sql;
+
+ private Boolean enable;
+
+ private final Map<String, String> properties;
+
+ public AlterSqlBlockRuleStmt(String ruleName, Map<String, String> properties) {
+ this.ruleName = ruleName;
+ this.properties = properties;
+ }
+
+ @Override
+ public void analyze(Analyzer analyzer) throws UserException {
+ super.analyze(analyzer);
+ // check properties
+ checkProperties(properties);
+ }
+
+ private void checkProperties(Map<String, String> properties) throws UserException {
+ this.user = properties.get(CreateSqlBlockRuleStmt.USER_PROPERTY);
+ // if not default, need check whether user exist
+ if (StringUtils.isNotEmpty(user) && !SqlBlockRule.DEFAULT_USER.equals(user)) {
+ List<String> allUserIdents = Catalog.getCurrentCatalog().getAuth().getAllUserIdents(false).stream().map(UserIdentity::getUser).collect(Collectors.toList());
Review comment:
I didn't find this method before, so I'll change it
##########
File path: fe/fe-core/src/main/cup/sql_parser.cup
##########
@@ -692,6 +693,14 @@ stmt ::=
{: RESULT = stmt; :}
| uninstall_plugin_stmt : stmt
{: RESULT = stmt; :}
+ | create_sql_block_rule_stmt : stmt
+ {: RESULT = stmt; :}
+ | alter_sql_block_rule_stmt : stmt
+ {: RESULT = stmt; :}
+ | drop_sql_block_rule_stmt : stmt
+ {: RESULT = stmt; :}
+ | show_sql_block_rule_stmt : stmt
+ {: RESULT = stmt; :}
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r667320243
##########
File path: fe/fe-core/src/main/java/org/apache/doris/metric/MetricRepo.java
##########
@@ -298,6 +299,9 @@ public Long getValue() {
"total error rows of routine load");
PALO_METRIC_REGISTER.addPaloMetrics(COUNTER_ROUTINE_LOAD_ERROR_ROWS);
+ COUNTER_HIT_SQL_BLOCK_RULE = new LongCounterMetric("counter_hit_sql_blacklist", MetricUnit.ROWS,
+ "total hit sql blacklist query");
Review comment:
Sorry, I forgot to change this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r677887444
##########
File path: fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
##########
@@ -143,13 +145,15 @@ private void auditAfterExec(String origStmt, StatementBase parsedStmt, Data.PQue
ctx.getAuditEventBuilder().setFeIp(FrontendOptions.getLocalHostAddress());
- // We put origin query stmt at the end of audit log, for parsing the log more convenient.
+ String sql;
if (!ctx.getState().isQuery() && (parsedStmt != null && parsedStmt.needAuditEncryption())) {
- ctx.getAuditEventBuilder().setStmt(parsedStmt.toSql());
+ sql = parsedStmt.toSql();
} else {
- ctx.getAuditEventBuilder().setStmt(origStmt);
+ sql = origStmt;
}
-
+ ctx.getAuditEventBuilder().setSqlHash(DigestUtils.md5Hex(sql));
Review comment:
Ok, let me think about this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r684780556
##########
File path: fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
##########
@@ -0,0 +1,222 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.blockrule;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.analysis.DropSqlBlockRuleStmt;
+import org.apache.doris.analysis.ShowSqlBlockRuleStmt;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.io.Text;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.metric.MetricRepo;
+import org.apache.doris.persist.gson.GsonUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.gson.annotations.SerializedName;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+public class SqlBlockRuleMgr implements Writable {
+ private static final Logger LOG = LogManager.getLogger(SqlBlockRuleMgr.class);
+
+ private ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ @SerializedName(value = "nameToSqlBlockRuleMap")
+ private Map<String, SqlBlockRule> nameToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ @SerializedName(value = "sqlPatternMap")
+ private Map<String, Pattern> sqlPatternMap = Maps.newConcurrentMap();
Review comment:
Are you sure the `Pattern` can be serialized by Json?
And I think the `sqlPatternMap` is useless, why not just save the pattern in each `SqlBlockRule`?
##########
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
##########
@@ -1882,6 +1885,14 @@ public long loadSmallFiles(DataInputStream in, long checksum) throws IOException
return checksum;
}
+ public long loadSqlBlockRule(DataInputStream in, long checksum) throws IOException {
+ if (Catalog.getCurrentCatalogJournalVersion() >= FeMetaVersion.VERSION_103) {
+ sqlBlockRuleMgr.readFields(in);
Review comment:
```suggestion
sqlBlockRuleMgr = SqlBlockRuleMgr.read(in);
```
##########
File path: fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java
##########
@@ -0,0 +1,222 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.blockrule;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.analysis.DropSqlBlockRuleStmt;
+import org.apache.doris.analysis.ShowSqlBlockRuleStmt;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.io.Text;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.metric.MetricRepo;
+import org.apache.doris.persist.gson.GsonUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.gson.annotations.SerializedName;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+public class SqlBlockRuleMgr implements Writable {
+ private static final Logger LOG = LogManager.getLogger(SqlBlockRuleMgr.class);
+
+ private ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ @SerializedName(value = "nameToSqlBlockRuleMap")
+ private Map<String, SqlBlockRule> nameToSqlBlockRuleMap = Maps.newConcurrentMap();
+
+ @SerializedName(value = "sqlPatternMap")
+ private Map<String, Pattern> sqlPatternMap = Maps.newConcurrentMap();
+
+ private void writeLock() {
+ lock.writeLock().lock();
+ }
+
+ private void writeUnlock() {
+ lock.writeLock().unlock();
+ }
+
+ public boolean existRule(String name) {
+ return nameToSqlBlockRuleMap.containsKey(name);
+ }
+
+ public List<SqlBlockRule> getSqlBlockRule(ShowSqlBlockRuleStmt stmt) throws AnalysisException {
+ String ruleName = stmt.getRuleName();
+ if (StringUtils.isNotEmpty(ruleName)) {
+ if (nameToSqlBlockRuleMap.containsKey(ruleName)) {
+ SqlBlockRule sqlBlockRule = nameToSqlBlockRuleMap.get(ruleName);
+ return Lists.newArrayList(sqlBlockRule);
+ }
+ return Lists.newArrayList();
+ }
+ return Lists.newArrayList(nameToSqlBlockRuleMap.values());
+ }
+
+ public void createSqlBlockRule(CreateSqlBlockRuleStmt stmt) throws UserException {
+ writeLock();
+ try {
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromCreateStmt(stmt);
+ String ruleName = sqlBlockRule.getName();
+ if (existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " already create");
+ }
+ unprotectedAdd(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logCreateSqlBlockRule(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayCreate(SqlBlockRule sqlBlockRule) {
+ unprotectedAdd(sqlBlockRule);
+ LOG.info("replay create sql block rule: {}", sqlBlockRule);
+ }
+
+ public void alterSqlBlockRule(AlterSqlBlockRuleStmt stmt) throws DdlException {
+ writeLock();
+ try {
+ SqlBlockRule sqlBlockRule = SqlBlockRule.fromAlterStmt(stmt);
+ String ruleName = sqlBlockRule.getName();
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ SqlBlockRule originRule = nameToSqlBlockRuleMap.get(ruleName);
+ if (StringUtils.isEmpty(sqlBlockRule.getSql())) {
+ sqlBlockRule.setSql(originRule.getSql());
+ }
+ if (StringUtils.isEmpty(sqlBlockRule.getSqlHash())) {
+ sqlBlockRule.setSqlHash(originRule.getSqlHash());
+ }
+ if (sqlBlockRule.getGlobal() == null) {
+ sqlBlockRule.setGlobal(originRule.getGlobal());
+ }
+ if (sqlBlockRule.getEnable() == null) {
+ sqlBlockRule.setEnable(originRule.getEnable());
+ }
+ unprotectedUpdate(sqlBlockRule);
+ Catalog.getCurrentCatalog().getEditLog().logAlterSqlBlockRule(sqlBlockRule);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayAlter(SqlBlockRule sqlBlockRule) {
+ unprotectedUpdate(sqlBlockRule);
+ LOG.info("replay alter sql block rule: {}", sqlBlockRule);
+ }
+
+ public void unprotectedUpdate(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ }
+
+ public void unprotectedAdd(SqlBlockRule sqlBlockRule) {
+ nameToSqlBlockRuleMap.put(sqlBlockRule.getName(), sqlBlockRule);
+ String sql = sqlBlockRule.getSql();
+ if (StringUtils.isNotEmpty(sql)) {
+ sqlPatternMap.put(sql, Pattern.compile(sql));
+ }
+ }
+
+ public void dropSqlBlockRule(DropSqlBlockRuleStmt stmt) throws DdlException {
+ writeLock();
+ try {
+ List<String> ruleNames = stmt.getRuleNames();
+ for (String ruleName : ruleNames) {
+ if (!existRule(ruleName)) {
+ throw new DdlException("the sql block rule " + ruleName + " not exist");
+ }
+ }
+ unprotectedDrop(ruleNames);
+ Catalog.getCurrentCatalog().getEditLog().logDropSqlBlockRule(ruleNames);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void replayDrop(List<String> ruleNames) {
+ unprotectedDrop(ruleNames);
+ LOG.info("replay drop sql block ruleNames: {}", ruleNames);
+ }
+
+ public void unprotectedDrop(List<String> ruleNames) {
+ ruleNames.forEach(name -> nameToSqlBlockRuleMap.remove(name));
+ }
+
+ public void matchSql(String originSql, String sqlHash, String user) throws AnalysisException {
+ // match global rule
+ List<SqlBlockRule> globalRules = nameToSqlBlockRuleMap.values().stream().filter(SqlBlockRule::getGlobal).collect(Collectors.toList());
+ for (SqlBlockRule rule : globalRules) {
+ matchSql(rule, originSql, sqlHash);
+ }
+ // match user rule
+ String bindSqlBlockRules = Catalog.getCurrentCatalog().getAuth().getSqlBlockRules(user);
+ if (StringUtils.isNotEmpty(bindSqlBlockRules)) {
+ String[] split = bindSqlBlockRules.split(",");
Review comment:
This `split()` should not be done every time we call `matchSql`.
It should be prepared already.
##########
File path: fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserProperty.java
##########
@@ -89,12 +90,25 @@
*/
private WhiteList whiteList = new WhiteList();
+ // the binding of sql_block_rule name, multiple are separated by ','
+ private String sqlBlockRules = null;
Review comment:
This can be put into `commonProperties`
##########
File path: fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserProperty.java
##########
@@ -89,12 +90,25 @@
*/
private WhiteList whiteList = new WhiteList();
+ // the binding of sql_block_rule name, multiple are separated by ','
+ private String sqlBlockRules = null;
+
+ @Deprecated
Review comment:
Why adding these deprecated fields back?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r667319845
##########
File path: fe/fe-core/src/main/java/org/apache/doris/block/SqlBlockRule.java
##########
@@ -0,0 +1,111 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.block;
+
+import org.apache.doris.analysis.AlterSqlBlockRuleStmt;
+import org.apache.doris.analysis.CreateSqlBlockRuleStmt;
+import org.apache.doris.common.io.Text;
+import org.apache.doris.common.io.Writable;
+
+import com.google.common.collect.Lists;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+
+public class SqlBlockRule implements Writable {
+
+ public static final String NAME_TYPE = "SQL BLOCK RULE NAME";
+
+ public static final String DEFAULT_USER = "default";
+
+ // the rule name, cluster unique
+ private String name;
+
+ // default stands for all users
+ private String user;
+
+ private String sql;
+
+ // whether to use the rule
+ private Boolean enable;
+
+ public SqlBlockRule(String name) {
+ this.name = name;
+ }
+
+ public SqlBlockRule(String name, String user, String sql, Boolean enable) {
+ this.name = name;
+ this.user = user;
+ this.sql = sql;
+ this.enable = enable;
+ }
+
+ public static SqlBlockRule fromCreateStmt(CreateSqlBlockRuleStmt stmt) {
+ return new SqlBlockRule(stmt.getRuleName(), stmt.getUser(), stmt.getSql(), stmt.isEnable());
+ }
+
+ public static SqlBlockRule fromAlterStmt(AlterSqlBlockRuleStmt stmt) {
+ return new SqlBlockRule(stmt.getRuleName(), stmt.getUser(), stmt.getSql(), stmt.getEnable());
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public String getUser() {
+ return user;
+ }
+
+ public String getSql() {
+ return sql;
+ }
+
+ public Boolean getEnable() {
+ return enable;
+ }
+
+ public void setUser(String user) {
+ this.user = user;
+ }
+
+ public void setSql(String sql) {
+ this.sql = sql;
+ }
+
+ public void setEnable(Boolean enable) {
+ this.enable = enable;
+ }
+
+ public List<String> getShowInfo() {
+ return Lists.newArrayList(this.name, this.user, this.sql, String.valueOf(this.enable));
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
+ Text.writeString(out, name);
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#issuecomment-878011931
> I think this feature is used to block some abnormal sql, for example , sql caused core. Is it need to limit the number of rules, which will occupied a lot of memory. Is is better to store the hash value of a sql, not the sql self, when a sql is very long?
At present, in my scenario, some special user are not allowed to join, and some large tables are not allowed to query the full table, really do not need too many rules
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r667319925
##########
File path: fe/fe-core/src/main/java/org/apache/doris/block/SqlBlockRule.java
##########
@@ -0,0 +1,111 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.block;
Review comment:
I will alter it to blockrule,In the future I will support ip Block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r667319313
##########
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/AlterSqlBlockRuleStmt.java
##########
@@ -0,0 +1,84 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.block.SqlBlockRule;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.UserException;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class AlterSqlBlockRuleStmt extends DdlStmt {
+
+ private final String ruleName;
+
+ private String user;
+
+ private String sql;
+
+ private Boolean enable;
+
+ private final Map<String, String> properties;
+
+ public AlterSqlBlockRuleStmt(String ruleName, Map<String, String> properties) {
+ this.ruleName = ruleName;
+ this.properties = properties;
+ }
+
+ @Override
+ public void analyze(Analyzer analyzer) throws UserException {
+ super.analyze(analyzer);
+ // check properties
+ checkProperties(properties);
Review comment:
`checkProperties` whether needed?I referred to `AlterRoutineLoadStmt`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r671699208
##########
File path: fe/fe-core/src/main/cup/sql_parser.cup
##########
@@ -1827,6 +1837,10 @@ drop_stmt ::=
{:
RESULT = new DropResourceStmt(resourceName);
:}
+ | KW_DROP KW_SQL_BLOCK_RULE ident_list:ruleNames
Review comment:
The current design is that rule names are not allowed to be repeated, so I think it's easier to use? Maybe we can support configuring multiple users
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#issuecomment-879099590
> > I think this feature is used to block some abnormal sql, for example , sql caused core. Is it need to limit the number of rules, which will occupied a lot of memory. Is is better to store the hash value of a sql, not the sql self, when a sql is very long?
>
> Using hash to block problematic SQL may also be a practical requirement. In some cases, we need to quickly block some SQL that affects the cluster through the block rules. There are pros and cons to using regular expressions or hash values:
>
> 1. It is more flexible for regular expressions, and can match a series of sql through one rule. But the disadvantage is that the threshold for using regular expressions is high, and wrong expressions may result in matching wrong SQL.
> 2. The disadvantage of Hash value is that it can only perform complete matching, but it takes up less space and matches more quickly. But any change in SQL will cause a change in its hash value, such as spaces, line breaks, and so on. Therefore, I suggest that we can print the hash value corresponding to sql in `fe.audit.log`, so that users can conveniently use the hash value in the log to quickly add block rule.
>
> In summary, we can implement two block rules, one based on hash and the other based on regular expressions.
Okay, I'm going to support both matches
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r667320399
##########
File path: fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
##########
@@ -804,6 +805,21 @@ public static void loadJournal(Catalog catalog, JournalEntity journal) {
catalog.getAlterInstance().replayReplaceTable(log);
break;
}
+ case OperationType.OP_CREATE_SQL_BLOCK_RULE: {
Review comment:
Yes, I forgot this when I migrated the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r684881534
##########
File path: fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserProperty.java
##########
@@ -121,6 +126,17 @@ public long getMaxQueryInstances() {
return commonProperties.getMaxQueryInstances();// maxQueryInstances;
}
+ public String[] getSqlBlockRules() {
+ if (this.sqlBlockRulesSplit.length != 0) {
+ return this.sqlBlockRulesSplit;
+ }
+ String sqlBlockRules = commonProperties.getSqlBlockRules();
Review comment:
This method is only protected by `readLock`, but you do write operation here.
Why not just save the splitted rules in `commonProperties`?
##########
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
##########
@@ -1896,6 +1899,14 @@ public long loadSmallFiles(DataInputStream in, long checksum) throws IOException
return checksum;
}
+ public long loadSqlBlockRule(DataInputStream in, long checksum) throws IOException {
+ if (Catalog.getCurrentCatalogJournalVersion() >= FeMetaVersion.VERSION_103) {
Review comment:
```suggestion
if (Catalog.getCurrentCatalogJournalVersion() >= FeMetaVersion.VERSION_104) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a change in pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
stalary commented on a change in pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#discussion_r684786424
##########
File path: fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserProperty.java
##########
@@ -89,12 +90,25 @@
*/
private WhiteList whiteList = new WhiteList();
+ // the binding of sql_block_rule name, multiple are separated by ','
+ private String sqlBlockRules = null;
+
+ @Deprecated
Review comment:
The problem with rebase,I will change it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6192: ADD: support sql block rule
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #6192:
URL: https://github.com/apache/incubator-doris/pull/6192#issuecomment-897698268
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org