You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zddr (via GitHub)" <gi...@apache.org> on 2023/08/08 14:15:22 UTC

[GitHub] [doris] zddr opened a new pull request, #22754: [improvement](auth)select priv for Nereids

zddr opened a new pull request, #22754:
URL: https://github.com/apache/doris/pull/22754

   ## Proposed changes
   
   Issue Number: close #xxx
   
   add a rewriter in "Final rewrite and check" to check col auth
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] zddr commented on pull request #22754: [opt](Nereids) support column permission control

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #22754:
URL: https://github.com/apache/doris/pull/22754#issuecomment-1676159157

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] zddr commented on pull request #22754: [opt](Nereids) support column permission control

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #22754:
URL: https://github.com/apache/doris/pull/22754#issuecomment-1676200192

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] zddr commented on pull request #22754: [opt](Nereids) support column permission control

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #22754:
URL: https://github.com/apache/doris/pull/22754#issuecomment-1674852900

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] hello-stephen commented on pull request #22754: [opt](Nereids) support column permission control

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22754:
URL: https://github.com/apache/doris/pull/22754#issuecomment-1682234336

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.35 seconds
    stream load tsv:          537 seconds loaded 74807831229 Bytes, about 132 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
    insert into select:          29.0 seconds inserted 10000000 Rows, about 344K ops/s
    storage size: 17162155280 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] morningman commented on a diff in pull request #22754: [opt](Nereids) support column permission control

Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #22754:
URL: https://github.com/apache/doris/pull/22754#discussion_r1288830651


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AuthChecker.java:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.catalog.DatabaseIf;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.common.UserException;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.algebra.CatalogRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+
+public class AuthChecker implements RewriteRuleFactory {
+
+    @Override
+    public List<Rule> buildRules() {
+        return ImmutableList.of(logicalProject(logicalRelation()).then(project -> dealRelation(project))
+                        .toRule(RuleType.RELATION_AUTHENTICATION),
+                logicalProject(logicalFilter(logicalRelation())).then(project -> dealFilter(project))
+                        .toRule(RuleType.RELATION_AUTHENTICATION));
+    }
+
+    public static Plan dealFilter(LogicalProject<LogicalFilter<LogicalRelation>> project) {
+        Plan plan = project.child(0);
+        if (!(plan instanceof LogicalFilter)) {
+            return project;
+        }
+        LogicalFilter filter = (LogicalFilter) plan;
+        Plan relation = filter.child(0);
+        if (!(relation instanceof CatalogRelation)) {
+            return project;
+        }
+        Set<String> cols = Sets.newHashSet();
+        // get cols from filter
+        getCols(filter.getConjuncts(), cols);
+        // get cols from project
+        getCols(project.getProjects(), cols);
+        checkAuth((CatalogRelation) relation, cols);
+        return project;
+    }
+
+    public static LogicalProject dealRelation(
+            LogicalProject<LogicalRelation> project) {
+        Plan relation = project.child(0);
+        if (!(relation instanceof CatalogRelation)) {
+            return project;
+        }
+        Set<String> cols = Sets.newHashSet();
+        // get cols from project
+        getCols(project.getProjects(), cols);
+        checkAuth((CatalogRelation) relation, cols);
+        return project;
+    }
+
+    public static void checkAuth(CatalogRelation catalogRelation, Set<String> cols) {
+        TableIf table = catalogRelation.getTable();
+        if (table == null) {
+            return;
+        }
+        DatabaseIf database = catalogRelation.getDatabase();
+        if (database == null) {
+            return;
+        }
+        CatalogIf catalog = database.getCatalog();
+        if (catalog == null) {
+            return;
+        }
+        if (cols.size() == 0) {
+            return;
+        }
+        try {
+            Env.getCurrentEnv().getAccessManager()
+                    .checkColumnsPriv(ConnectContext.get().getCurrentUserIdentity(), catalog.getName(),
+                            database.getFullName(), table.getName(), cols, PrivPredicate.SELECT);

Review Comment:
   Can we check other privs like LOAD?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] zddr commented on pull request #22754: [opt](Nereids) support column permission control

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #22754:
URL: https://github.com/apache/doris/pull/22754#issuecomment-1677238862

   run feut


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] hello-stephen commented on pull request #22754: [opt](Nereids) support column permission control

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22754:
URL: https://github.com/apache/doris/pull/22754#issuecomment-1676169033

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 47.45 seconds
    stream load tsv:          506 seconds loaded 74807831229 Bytes, about 140 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
    insert into select:          29.3 seconds inserted 10000000 Rows, about 341K ops/s
    storage size: 17162412279 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] zddr commented on pull request #22754: [opt](Nereids) support column permission control

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #22754:
URL: https://github.com/apache/doris/pull/22754#issuecomment-1677178404

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] zddr closed pull request #22754: [opt](Nereids) support column permission control

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr closed pull request #22754: [opt](Nereids) support column permission control
URL: https://github.com/apache/doris/pull/22754


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] zddr commented on pull request #22754: [opt](Nereids) support column permission control

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #22754:
URL: https://github.com/apache/doris/pull/22754#issuecomment-1678323274

   run p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] zddr commented on pull request #22754: [opt](Nereids) support column permission control

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #22754:
URL: https://github.com/apache/doris/pull/22754#issuecomment-1676961653

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] zddr commented on pull request #22754: [opt](Nereids) support column permission control

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #22754:
URL: https://github.com/apache/doris/pull/22754#issuecomment-1687358348

   由于视图问题,暂时不加列权限


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] zddr commented on pull request #22754: [improvement](auth)select priv for Nereids

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #22754:
URL: https://github.com/apache/doris/pull/22754#issuecomment-1669710386

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] zddr commented on pull request #22754: [opt](Nereids) support column permission control

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #22754:
URL: https://github.com/apache/doris/pull/22754#issuecomment-1678323168

   run feut


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] zddr commented on pull request #22754: [opt](Nereids) support column permission control

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #22754:
URL: https://github.com/apache/doris/pull/22754#issuecomment-1682167619

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] hello-stephen commented on pull request #22754: [opt](Nereids) support column permission control

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22754:
URL: https://github.com/apache/doris/pull/22754#issuecomment-1676208291

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.77 seconds
    stream load tsv:          511 seconds loaded 74807831229 Bytes, about 139 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          29.4 seconds inserted 10000000 Rows, about 340K ops/s
    storage size: 17162409436 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] zddr commented on a diff in pull request #22754: [opt](Nereids) support column permission control

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on code in PR #22754:
URL: https://github.com/apache/doris/pull/22754#discussion_r1291367822


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AuthChecker.java:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.catalog.DatabaseIf;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.common.UserException;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.algebra.CatalogRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+
+public class AuthChecker implements RewriteRuleFactory {
+
+    @Override
+    public List<Rule> buildRules() {
+        return ImmutableList.of(logicalProject(logicalRelation()).then(project -> dealRelation(project))
+                        .toRule(RuleType.RELATION_AUTHENTICATION),
+                logicalProject(logicalFilter(logicalRelation())).then(project -> dealFilter(project))
+                        .toRule(RuleType.RELATION_AUTHENTICATION));
+    }
+
+    public static Plan dealFilter(LogicalProject<LogicalFilter<LogicalRelation>> project) {
+        Plan plan = project.child(0);
+        if (!(plan instanceof LogicalFilter)) {
+            return project;
+        }
+        LogicalFilter filter = (LogicalFilter) plan;
+        Plan relation = filter.child(0);
+        if (!(relation instanceof CatalogRelation)) {
+            return project;
+        }
+        Set<String> cols = Sets.newHashSet();
+        // get cols from filter
+        getCols(filter.getConjuncts(), cols);
+        // get cols from project
+        getCols(project.getProjects(), cols);
+        checkAuth((CatalogRelation) relation, cols);
+        return project;
+    }
+
+    public static LogicalProject dealRelation(
+            LogicalProject<LogicalRelation> project) {
+        Plan relation = project.child(0);
+        if (!(relation instanceof CatalogRelation)) {
+            return project;
+        }
+        Set<String> cols = Sets.newHashSet();
+        // get cols from project
+        getCols(project.getProjects(), cols);
+        checkAuth((CatalogRelation) relation, cols);
+        return project;
+    }
+
+    public static void checkAuth(CatalogRelation catalogRelation, Set<String> cols) {
+        TableIf table = catalogRelation.getTable();
+        if (table == null) {
+            return;
+        }
+        DatabaseIf database = catalogRelation.getDatabase();
+        if (database == null) {
+            return;
+        }
+        CatalogIf catalog = database.getCatalog();
+        if (catalog == null) {
+            return;
+        }
+        if (cols.size() == 0) {
+            return;
+        }
+        try {
+            Env.getCurrentEnv().getAccessManager()
+                    .checkColumnsPriv(ConnectContext.get().getCurrentUserIdentity(), catalog.getName(),
+                            database.getFullName(), table.getName(), cols, PrivPredicate.SELECT);

Review Comment:
   not this method



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] zddr commented on pull request #22754: [opt](Nereids) support column permission control

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #22754:
URL: https://github.com/apache/doris/pull/22754#issuecomment-1674831159

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org