You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/06/18 14:15:33 UTC
[GitHub] [doris] AshinGau opened a new pull request, #10246: [feature-wip](multi-catalog) add CatalogPrivTable to support unified authority management of datalake
AshinGau opened a new pull request, #10246:
URL: https://github.com/apache/doris/pull/10246
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
Support unified authority management of datalake.
Supported:
1. Change FeMetaVersion to 111, compatible with upgrade from 110.
2. Add catalog level privileges, and degrade global level privileges to catalog level if FeMetaVersion < 111.
3. Support 'show all grants', 'show roles' statement.
4. Previous version of SQL syntax.
Todo:
1. three-segement format catalog.database.table in SQL syntax.
2. User document for the unified authority management of datalake.
3. LDAP services to provide authentication.
## Checklist(Required)
1. Does it affect the original behavior: (No)
2. Has unit tests been added: (Yes)
3. Has document been added or modified: (No)
4. Does it need to update dependencies: (No)
5. Are there any changes that cannot be rolled back: (No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #10246: [feature-wip](multi-catalog) add CatalogPrivTable to support unified authority management of datalake
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #10246:
URL: https://github.com/apache/doris/pull/10246#discussion_r901452877
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/TablePattern.java:
##########
@@ -110,34 +160,27 @@ public boolean equals(Object obj) {
return false;
}
TablePattern other = (TablePattern) obj;
- return db.equals(other.getQualifiedDb()) && tbl.equals(other.getTbl());
+ return ctl.equals(other.getQualifiedCtl()) && db.equals(other.getQualifiedDb()) && tbl.equals(other.getTbl());
}
@Override
public int hashCode() {
int result = 17;
+ result = 31 * result + ctl.hashCode();
result = 31 * result + db.hashCode();
result = 31 * result + tbl.hashCode();
return result;
}
@Override
public String toString() {
- StringBuilder sb = new StringBuilder();
- sb.append(db).append(".").append(tbl);
- return sb.toString();
+ return String.format("%s.%s.%s", ctl, db, tbl);
}
@Override
public void write(DataOutput out) throws IOException {
Preconditions.checkState(isAnalyzed);
- Text.writeString(out, db);
- Text.writeString(out, tbl);
- }
-
- public void readFields(DataInput in) throws IOException {
- db = Text.readString(in);
- tbl = Text.readString(in);
- isAnalyzed = true;
+ String json = GsonUtils.GSON.toJson(this);
Review Comment:
Need to add version check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10246: [feature-wip](multi-catalog) add CatalogPrivTable to support unified authority management of datalake
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #10246:
URL: https://github.com/apache/doris/pull/10246#issuecomment-1160223331
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #10246: [feature-wip](multi-catalog) add CatalogPrivTable to support unified authority management of datalake
Posted by GitBox <gi...@apache.org>.
morningman merged PR #10246:
URL: https://github.com/apache/doris/pull/10246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10246: [feature-wip](multi-catalog) add CatalogPrivTable to support unified authority management of datalake
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #10246:
URL: https://github.com/apache/doris/pull/10246#issuecomment-1160223272
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] AshinGau commented on a diff in pull request #10246: [feature-wip](multi-catalog) add CatalogPrivTable to support unified authority management of datalake
Posted by GitBox <gi...@apache.org>.
AshinGau commented on code in PR #10246:
URL: https://github.com/apache/doris/pull/10246#discussion_r900931944
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/InternalDataSource.java:
##########
@@ -198,7 +198,7 @@
* There is only one internal data source in a cluster. And its id is always 0.
*/
public class InternalDataSource implements DataSourceIf {
- public static final String INTERNAL_DS_NAME = "__internal";
Review Comment:
Need #10213 to be merged first. Underscores will case authentication exception. I will remove this after #10213 is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] AshinGau commented on a diff in pull request #10246: [feature-wip](multi-catalog) add CatalogPrivTable to support unified authority management of datalake
Posted by GitBox <gi...@apache.org>.
AshinGau commented on code in PR #10246:
URL: https://github.com/apache/doris/pull/10246#discussion_r902360054
##########
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/CatalogPrivEntry.java:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.mysql.privilege;
+
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.CaseSensibility;
+import org.apache.doris.common.FeMetaVersion;
+import org.apache.doris.common.PatternMatcher;
+import org.apache.doris.common.io.Text;
+import org.apache.doris.datasource.InternalDataSource;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+
+public class CatalogPrivEntry extends PrivEntry {
+ protected static final String ANY_CTL = "*";
+
+ protected PatternMatcher ctlPattern;
+ protected String origCtl;
+ protected boolean isAnyCtl;
+
+ protected CatalogPrivEntry() {
+ }
+
+ protected CatalogPrivEntry(PatternMatcher userPattern, String user,
+ PatternMatcher hostPattern, String origHost,
+ PatternMatcher ctlPattern, String origCtl,
+ boolean isDomain, PrivBitSet privSet) {
+ super(hostPattern, origHost, userPattern, user, isDomain, privSet);
+ this.ctlPattern = ctlPattern;
+ this.origCtl = origCtl;
+ if (origCtl.equals(ANY_CTL)) {
+ isAnyCtl = true;
+ }
+ }
+
+ public static CatalogPrivEntry create(String user, String host, String ctl, boolean isDomain, PrivBitSet privs)
+ throws AnalysisException {
+ PatternMatcher hostPattern = PatternMatcher.createMysqlPattern(host, CaseSensibility.HOST.getCaseSensibility());
+
+ PatternMatcher ctlPattern = createCtlPatternMatcher(ctl);
+
+ PatternMatcher userPattern = PatternMatcher.createMysqlPattern(user, CaseSensibility.USER.getCaseSensibility());
+
+ if (privs.containsNodePriv() || privs.containsResourcePriv()) {
+ throw new AnalysisException("Datasource privilege can not contains node or resource privileges: " + privs);
+ }
+
+ return new CatalogPrivEntry(userPattern, user, hostPattern, host, ctlPattern, ctl, isDomain, privs);
+ }
+
+ private static PatternMatcher createCtlPatternMatcher(String ctl) throws AnalysisException {
+ boolean dsCaseSensibility = CaseSensibility.CATALOG.getCaseSensibility();
+ return PatternMatcher.createMysqlPattern(ctl.equals(ANY_CTL) ? "%" : ctl, dsCaseSensibility);
+ }
+
+ public PatternMatcher getCtlPattern() {
+ return ctlPattern;
+ }
+
+ public String getOrigCtl() {
+ return origCtl;
+ }
+
+ public boolean isAnyCtl() {
+ return isAnyCtl;
+ }
+
+ @Override
+ public int compareTo(PrivEntry other) {
+ if (!(other instanceof CatalogPrivEntry)) {
+ throw new ClassCastException("cannot cast " + other.getClass().toString() + " to " + this.getClass());
+ }
+
+ CatalogPrivEntry otherEntry = (CatalogPrivEntry) other;
+ return compareAssist(origUser, otherEntry.origUser,
+ origHost, otherEntry.origHost,
+ origCtl, otherEntry.origCtl);
+ }
+
+ @Override
+ public boolean keyMatch(PrivEntry other) {
+ if (!(other instanceof CatalogPrivEntry)) {
+ return false;
+ }
+
+ CatalogPrivEntry otherEntry = (CatalogPrivEntry) other;
+ return origUser.equals(otherEntry.origUser) && origHost.equals(otherEntry.origHost)
+ && origCtl.equals(otherEntry.origCtl) && isDomain == otherEntry.isDomain;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("catalog privilege. user: %s, host: %s, ctl: %s, priv: %s, set by resolver: %b",
+ origUser, origHost, origCtl, privSet.toString(), isSetByDomainResolver);
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
+ if (!isClassNameWrote) {
+ String className = CatalogPrivEntry.class.getCanonicalName();
+ Text.writeString(out, className);
+ isClassNameWrote = true;
+ }
+ super.write(out);
+ Text.writeString(out, origCtl);
+ isClassNameWrote = false;
+ }
+
+ public void readFields(DataInput in) throws IOException {
+ super.readFields(in);
+
+ if (Catalog.getCurrentCatalogJournalVersion() >= FeMetaVersion.VERSION_111) {
Review Comment:
Should revert. DbPrivEntry extends CatalogPrivEntry, so DbPrivEntry.readFields will call CatalogPrivEntry.readFields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #10246: [feature-wip](multi-catalog) add CatalogPrivTable to support unified authority management of datalake
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #10246:
URL: https://github.com/apache/doris/pull/10246#discussion_r901044061
##########
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/CatalogPrivEntry.java:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.mysql.privilege;
+
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.CaseSensibility;
+import org.apache.doris.common.FeMetaVersion;
+import org.apache.doris.common.PatternMatcher;
+import org.apache.doris.common.io.Text;
+import org.apache.doris.datasource.InternalDataSource;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+
+public class CatalogPrivEntry extends PrivEntry {
+ protected static final String ANY_CTL = "*";
+
+ protected PatternMatcher ctlPattern;
+ protected String origCtl;
+ protected boolean isAnyCtl;
+
+ protected CatalogPrivEntry() {
+ }
+
+ protected CatalogPrivEntry(PatternMatcher userPattern, String user,
+ PatternMatcher hostPattern, String origHost,
+ PatternMatcher ctlPattern, String origCtl,
+ boolean isDomain, PrivBitSet privSet) {
+ super(hostPattern, origHost, userPattern, user, isDomain, privSet);
+ this.ctlPattern = ctlPattern;
+ this.origCtl = origCtl;
+ if (origCtl.equals(ANY_CTL)) {
+ isAnyCtl = true;
+ }
+ }
+
+ public static CatalogPrivEntry create(String user, String host, String ctl, boolean isDomain, PrivBitSet privs)
+ throws AnalysisException {
+ PatternMatcher hostPattern = PatternMatcher.createMysqlPattern(host, CaseSensibility.HOST.getCaseSensibility());
+
+ PatternMatcher ctlPattern = createCtlPatternMatcher(ctl);
+
+ PatternMatcher userPattern = PatternMatcher.createMysqlPattern(user, CaseSensibility.USER.getCaseSensibility());
+
+ if (privs.containsNodePriv() || privs.containsResourcePriv()) {
+ throw new AnalysisException("Datasource privilege can not contains node or resource privileges: " + privs);
+ }
+
+ return new CatalogPrivEntry(userPattern, user, hostPattern, host, ctlPattern, ctl, isDomain, privs);
+ }
+
+ private static PatternMatcher createCtlPatternMatcher(String ctl) throws AnalysisException {
+ boolean dsCaseSensibility = CaseSensibility.CATALOG.getCaseSensibility();
+ return PatternMatcher.createMysqlPattern(ctl.equals(ANY_CTL) ? "%" : ctl, dsCaseSensibility);
+ }
+
+ public PatternMatcher getCtlPattern() {
+ return ctlPattern;
+ }
+
+ public String getOrigCtl() {
+ return origCtl;
+ }
+
+ public boolean isAnyCtl() {
+ return isAnyCtl;
+ }
+
+ @Override
+ public int compareTo(PrivEntry other) {
+ if (!(other instanceof CatalogPrivEntry)) {
+ throw new ClassCastException("cannot cast " + other.getClass().toString() + " to " + this.getClass());
+ }
+
+ CatalogPrivEntry otherEntry = (CatalogPrivEntry) other;
+ return compareAssist(origUser, otherEntry.origUser,
+ origHost, otherEntry.origHost,
+ origCtl, otherEntry.origCtl);
+ }
+
+ @Override
+ public boolean keyMatch(PrivEntry other) {
+ if (!(other instanceof CatalogPrivEntry)) {
+ return false;
+ }
+
+ CatalogPrivEntry otherEntry = (CatalogPrivEntry) other;
+ return origUser.equals(otherEntry.origUser) && origHost.equals(otherEntry.origHost)
+ && origCtl.equals(otherEntry.origCtl) && isDomain == otherEntry.isDomain;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("catalog privilege. user: %s, host: %s, ctl: %s, priv: %s, set by resolver: %b",
+ origUser, origHost, origCtl, privSet.toString(), isSetByDomainResolver);
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
+ if (!isClassNameWrote) {
+ String className = CatalogPrivEntry.class.getCanonicalName();
+ Text.writeString(out, className);
+ isClassNameWrote = true;
+ }
+ super.write(out);
+ Text.writeString(out, origCtl);
+ isClassNameWrote = false;
+ }
+
+ public void readFields(DataInput in) throws IOException {
+ super.readFields(in);
+
+ if (Catalog.getCurrentCatalogJournalVersion() >= FeMetaVersion.VERSION_111) {
Review Comment:
This is a new cluster, no need to check meta version.
Because every newly created instance of this class must have `origCtl` field
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/TableName.java:
##########
@@ -136,17 +153,22 @@ public String toSql() {
@Override
public void write(DataOutput out) throws IOException {
+ Text.writeString(out, ctl);
Review Comment:
Better to change it to GSON method this time
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/TablePattern.java:
##########
@@ -70,23 +87,39 @@ public String getTbl() {
public PrivLevel getPrivLevel() {
Preconditions.checkState(isAnalyzed);
- if (db.equals("*")) {
+ if (ctl.equals("*")) {
return PrivLevel.GLOBAL;
- } else if (!tbl.equals("*")) {
+ } else if (db.equals("*")) {
+ return PrivLevel.CATALOG;
+ } else if (tbl.equals("*")) {
+ return PrivLevel.DATABASE;
+ } else {
return PrivLevel.TABLE;
+ }
+ }
+
+ public void analyze(Analyzer analyzer) throws AnalysisException {
+ if (ctl == null) {
+ analyze(analyzer.getClusterName(), analyzer.getDefaultCatalog());
} else {
- return PrivLevel.DATABASE;
+ analyze(analyzer.getClusterName());
}
}
- public void analyze(String clusterName) throws AnalysisException {
+ public void analyze(String clusterName, String catalogName) throws AnalysisException {
Review Comment:
```suggestion
private void analyze(String catalogName, String clusterName) throws AnalysisException {
```
##########
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PrivEntry.java:
##########
@@ -253,4 +254,15 @@ public void readFields(DataInput in) throws IOException {
public int compareTo(PrivEntry o) {
throw new NotImplementedException();
}
+
+ protected static int compareAssist(String... strings) {
Review Comment:
Add comment to explain this method
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/TablePattern.java:
##########
@@ -110,32 +147,37 @@ public boolean equals(Object obj) {
return false;
}
TablePattern other = (TablePattern) obj;
- return db.equals(other.getQualifiedDb()) && tbl.equals(other.getTbl());
+ return ctl.equals(other.getQualifiedCtl()) && db.equals(other.getQualifiedDb()) && tbl.equals(other.getTbl());
}
@Override
public int hashCode() {
int result = 17;
+ result = 31 * result + ctl.hashCode();
result = 31 * result + db.hashCode();
result = 31 * result + tbl.hashCode();
return result;
}
@Override
public String toString() {
- StringBuilder sb = new StringBuilder();
- sb.append(db).append(".").append(tbl);
- return sb.toString();
+ return String.format("%s.%s.%s", ctl, db, tbl);
}
@Override
public void write(DataOutput out) throws IOException {
Preconditions.checkState(isAnalyzed);
+ Text.writeString(out, ctl);
Review Comment:
Change this to GSON
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #10246: [feature-wip](multi-catalog) add CatalogPrivTable to support unified authority management of datalake
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #10246:
URL: https://github.com/apache/doris/pull/10246#discussion_r901452877
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/TablePattern.java:
##########
@@ -110,34 +160,27 @@ public boolean equals(Object obj) {
return false;
}
TablePattern other = (TablePattern) obj;
- return db.equals(other.getQualifiedDb()) && tbl.equals(other.getTbl());
+ return ctl.equals(other.getQualifiedCtl()) && db.equals(other.getQualifiedDb()) && tbl.equals(other.getTbl());
}
@Override
public int hashCode() {
int result = 17;
+ result = 31 * result + ctl.hashCode();
result = 31 * result + db.hashCode();
result = 31 * result + tbl.hashCode();
return result;
}
@Override
public String toString() {
- StringBuilder sb = new StringBuilder();
- sb.append(db).append(".").append(tbl);
- return sb.toString();
+ return String.format("%s.%s.%s", ctl, db, tbl);
}
@Override
public void write(DataOutput out) throws IOException {
Preconditions.checkState(isAnalyzed);
- Text.writeString(out, db);
- Text.writeString(out, tbl);
- }
-
- public void readFields(DataInput in) throws IOException {
- db = Text.readString(in);
- tbl = Text.readString(in);
- isAnalyzed = true;
+ String json = GsonUtils.GSON.toJson(this);
Review Comment:
Need to add version check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org