You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ya...@apache.org on 2022/12/07 10:44:28 UTC
[incubator-kyuubi] branch master updated: [KYUUBI #3904] [Spark] New Authz Plan Serde Layer
This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 2540f44a8 [KYUUBI #3904] [Spark] New Authz Plan Serde Layer
2540f44a8 is described below
commit 2540f44a8769a538a34a3256fb66f4346d6de79e
Author: Kent Yao <ya...@apache.org>
AuthorDate: Wed Dec 7 18:44:15 2022 +0800
[KYUUBI #3904] [Spark] New Authz Plan Serde Layer
### _Why are the changes needed?_
This PR redesigned the authorization part of the spark authz module with a New Authz Plan Serde Layer.
#### Motivation
- add a general layer to describe a command, so that we can add a new command or users can add a third-party command easily according to the specification.
- get rid of the spark version check. The built-in spark commands frequently vary from version to version, which makes us hard to maintain at compile& runtime phase, and the third-party commands are hard to check by spark versions.
#### Data structure

Overall, we introduce 2 general basic data structures:
- CommandSpec: used to describe a command
- `classname` as key for the read-side to get the spec by a particular command
- pre-defined operation type
- descriptors
- Descriptor: used to describe an object, such as table, db, query,
- `fieldName`: the object to get
- `fieldExtractor`: the method to get the object; use SPI to load
- sub-descriptors: such as columns in a table
- etc.
#### SPI
- Extractor: implementations for `fieldExtractor`
- key: the name of the extractor for the read-side to get itself
- func: converting the field value to specific and general objects
#### Code Path
- Write code path
- automatically generated default json configuration files
- custom json configuration files for thrid-party commands
- Read code path
- Load json as maps
- RuleAuthorization -> PrivilegeBuilder.build -> get command desc from maps -> build privileges with the retrieved desc.
#### TODOs
- Add back the ArcticCommand
- Add delta command
- Add ways for loading custom json configuration files
- Add hudi commands
- etc
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #3904 from yaooqinn/na.
Closes #3904
efafcbaa2 [Kent Yao] sort
7d2b3e41c [Kent Yao] ci
379e93343 [Kent Yao] style
b52ab418a [Kent Yao] comments
9a24be64d [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
8926f0446 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
b45453a3e [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
f56148ea2 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
e47749d96 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
c32feef63 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
49dbb6842 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
2bced47b6 [Kent Yao] Merge branch 'master' into na
453541bd1 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer
Authored-by: Kent Yao <ya...@apache.org>
Signed-off-by: Kent Yao <ya...@apache.org>
---
extensions/spark/kyuubi-spark-authz/pom.xml | 5 +
...bi.plugin.spark.authz.serde.ActionTypeExtractor | 18 +
...kyuubi.plugin.spark.authz.serde.ColumnExtractor | 26 +
...uubi.plugin.spark.authz.serde.DatabaseExtractor | 23 +
...uubi.plugin.spark.authz.serde.FunctionExtractor | 20 +
....plugin.spark.authz.serde.FunctionTypeExtractor | 20 +
....kyuubi.plugin.spark.authz.serde.QueryExtractor | 19 +
....kyuubi.plugin.spark.authz.serde.TableExtractor | 23 +
...ubi.plugin.spark.authz.serde.TableTypeExtractor | 19 +
.../src/main/resources/database_command_spec.json | 129 +++
.../src/main/resources/function_command_spec.json | 75 ++
.../src/main/resources/table_command_spec.json | 1174 ++++++++++++++++++++
.../plugin/spark/authz/IcebergCommands.scala | 37 +-
.../kyuubi/plugin/spark/authz/OperationType.scala | 83 --
.../plugin/spark/authz/PrivilegesBuilder.scala | 533 ++-------
.../ranger/RuleApplyRowFilterAndDataMasking.scala | 5 +-
.../spark/authz/ranger/RuleAuthorization.scala | 3 +-
.../plugin/spark/authz/serde/CommandSpec.scala | 74 ++
.../plugin/spark/authz/serde/Descriptor.scala | 252 +++++
.../plugin/spark/authz/serde/Extractor.scala | 29 +
.../kyuubi/plugin/spark/authz/serde/Function.scala | 28 +
.../kyuubi/plugin/spark/authz/serde/Table.scala | 34 +
.../spark/authz/serde/actionTypeExtractors.scala | 42 +
.../spark/authz/serde/columnExtractors.scala | 89 ++
.../spark/authz/serde/databaseExtractors.scala | 89 ++
.../spark/authz/serde/functionExtractors.scala | 57 +
.../spark/authz/serde/functionTypeExtractors.scala | 81 ++
.../kyuubi/plugin/spark/authz/serde/package.scala | 55 +
.../plugin/spark/authz/serde/queryExtractors.scala | 48 +
.../plugin/spark/authz/serde/tableExtractors.scala | 141 +++
.../spark/authz/serde/tableTypeExtractors.scala | 70 ++
.../kyuubi/plugin/spark/authz/v2Commands.scala | 397 -------
.../IcebergCatalogPrivilegesBuilderSuite.scala | 13 +-
.../spark/authz/PrivilegesBuilderSuite.scala | 597 +++++-----
.../spark/authz/V2CommandsPrivilegesSuite.scala | 137 +--
.../plugin/spark/authz/gen/DatabaseCommands.scala | 128 +++
.../plugin/spark/authz/gen/FunctionCommands.scala | 81 ++
.../plugin/spark/authz/gen/IcebergCommands.scala | 55 +
.../spark/authz/gen/JsonSpecFileGenerator.scala | 50 +
.../plugin/spark/authz/gen/TableCommands.scala | 600 ++++++++++
.../authz/ranger/RangerSparkExtensionSuite.scala | 3 +-
41 files changed, 3958 insertions(+), 1404 deletions(-)
diff --git a/extensions/spark/kyuubi-spark-authz/pom.xml b/extensions/spark/kyuubi-spark-authz/pom.xml
index 494704aec..2d5c6f793 100644
--- a/extensions/spark/kyuubi-spark-authz/pom.xml
+++ b/extensions/spark/kyuubi-spark-authz/pom.xml
@@ -251,6 +251,11 @@
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.module</groupId>
+ <artifactId>jackson-module-scala_${scala.binary.version}</artifactId>
+ </dependency>
+
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-common_${scala.binary.version}</artifactId>
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.ActionTypeExtractor b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.ActionTypeExtractor
new file mode 100644
index 000000000..e3a8b2d2d
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.ActionTypeExtractor
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.OverwriteOrInsertActionTypeExtractor
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.ColumnExtractor b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.ColumnExtractor
new file mode 100644
index 000000000..ed76c15d1
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.ColumnExtractor
@@ -0,0 +1,26 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.PartitionColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.PartitionLocsSeqColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.PartitionOptionColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.PartitionSeqColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringSeqColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringSeqLastColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringSeqOptionColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StructFieldSeqColumnExtractor
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.DatabaseExtractor b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.DatabaseExtractor
new file mode 100644
index 000000000..2a269ee50
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.DatabaseExtractor
@@ -0,0 +1,23 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.ResolvedDBObjectNameDatabaseExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.ResolvedNamespaceDatabaseExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringDatabaseExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringOptionDatabaseExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringSeqDatabaseExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringSeqOptionDatabaseExtractor
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.FunctionExtractor b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.FunctionExtractor
new file mode 100644
index 000000000..4686bb033
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.FunctionExtractor
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.ExpressionInfoFunctionExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.FunctionIdentifierFunctionExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringFunctionExtractor
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.FunctionTypeExtractor b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.FunctionTypeExtractor
new file mode 100644
index 000000000..475f47afc
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.FunctionTypeExtractor
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.ExpressionInfoFunctionTypeExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.FunctionIdentifierFunctionTypeExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.TempMarkerFunctionTypeExtractor
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.QueryExtractor b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.QueryExtractor
new file mode 100644
index 000000000..c659114f9
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.QueryExtractor
@@ -0,0 +1,19 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.LogicalPlanOptionQueryExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.LogicalPlanQueryExtractor
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.TableExtractor b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.TableExtractor
new file mode 100644
index 000000000..94dcc5027
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.TableExtractor
@@ -0,0 +1,23 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.CatalogTableTableExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.DataSourceV2RelationTableExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.IdentifierTableExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.LogicalRelationTableExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.ResolvedTableTableExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.TableIdentifierTableExtractor
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.TableTypeExtractor b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.TableTypeExtractor
new file mode 100644
index 000000000..251a31758
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.TableTypeExtractor
@@ -0,0 +1,19 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.TableIdentifierTableTypeExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.ViewTypeTableTypeExtractor
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/resources/database_command_spec.json b/extensions/spark/kyuubi-spark-authz/src/main/resources/database_command_spec.json
new file mode 100644
index 000000000..46ed732a0
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/resources/database_command_spec.json
@@ -0,0 +1,129 @@
+[ {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.CommentOnNamespace",
+ "databaseDescs" : [ {
+ "fieldName" : "child",
+ "fieldExtractor" : "ResolvedNamespaceDatabaseExtractor",
+ "isInput" : false
+ } ],
+ "opType" : "ALTERDATABASE"
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.CreateNamespace",
+ "databaseDescs" : [ {
+ "fieldName" : "name",
+ "fieldExtractor" : "ResolvedDBObjectNameDatabaseExtractor",
+ "isInput" : false
+ }, {
+ "fieldName" : "namespace",
+ "fieldExtractor" : "StringSeqDatabaseExtractor",
+ "isInput" : false
+ } ],
+ "opType" : "CREATEDATABASE"
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.DescribeNamespace",
+ "databaseDescs" : [ {
+ "fieldName" : "namespace",
+ "fieldExtractor" : "ResolvedNamespaceDatabaseExtractor",
+ "isInput" : true
+ } ],
+ "opType" : "DESCDATABASE"
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.DropNamespace",
+ "databaseDescs" : [ {
+ "fieldName" : "namespace",
+ "fieldExtractor" : "ResolvedNamespaceDatabaseExtractor",
+ "isInput" : false
+ } ],
+ "opType" : "DROPDATABASE"
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.SetCatalogAndNamespace",
+ "databaseDescs" : [ {
+ "fieldName" : "child",
+ "fieldExtractor" : "ResolvedDBObjectNameDatabaseExtractor",
+ "isInput" : true
+ }, {
+ "fieldName" : "namespace",
+ "fieldExtractor" : "StringSeqOptionDatabaseExtractor",
+ "isInput" : true
+ } ],
+ "opType" : "SWITCHDATABASE"
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.SetNamespaceLocation",
+ "databaseDescs" : [ {
+ "fieldName" : "namespace",
+ "fieldExtractor" : "ResolvedNamespaceDatabaseExtractor",
+ "isInput" : false
+ } ],
+ "opType" : "ALTERDATABASE_LOCATION"
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.SetNamespaceProperties",
+ "databaseDescs" : [ {
+ "fieldName" : "namespace",
+ "fieldExtractor" : "ResolvedNamespaceDatabaseExtractor",
+ "isInput" : false
+ } ],
+ "opType" : "ALTERDATABASE"
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AlterDatabasePropertiesCommand",
+ "databaseDescs" : [ {
+ "fieldName" : "databaseName",
+ "fieldExtractor" : "StringDatabaseExtractor",
+ "isInput" : false
+ } ],
+ "opType" : "ALTERDATABASE"
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AlterDatabaseSetLocationCommand",
+ "databaseDescs" : [ {
+ "fieldName" : "databaseName",
+ "fieldExtractor" : "StringDatabaseExtractor",
+ "isInput" : false
+ } ],
+ "opType" : "ALTERDATABASE_LOCATION"
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AnalyzeTablesCommand",
+ "databaseDescs" : [ {
+ "fieldName" : "databaseName",
+ "fieldExtractor" : "StringOptionDatabaseExtractor",
+ "isInput" : true
+ } ],
+ "opType" : "ANALYZE_TABLE"
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.CreateDatabaseCommand",
+ "databaseDescs" : [ {
+ "fieldName" : "databaseName",
+ "fieldExtractor" : "StringDatabaseExtractor",
+ "isInput" : false
+ } ],
+ "opType" : "CREATEDATABASE"
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.DescribeDatabaseCommand",
+ "databaseDescs" : [ {
+ "fieldName" : "databaseName",
+ "fieldExtractor" : "StringDatabaseExtractor",
+ "isInput" : true
+ } ],
+ "opType" : "DESCDATABASE"
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.DropDatabaseCommand",
+ "databaseDescs" : [ {
+ "fieldName" : "databaseName",
+ "fieldExtractor" : "StringDatabaseExtractor",
+ "isInput" : false
+ } ],
+ "opType" : "DROPDATABASE"
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.SetDatabaseCommand",
+ "databaseDescs" : [ {
+ "fieldName" : "databaseName",
+ "fieldExtractor" : "StringDatabaseExtractor",
+ "isInput" : true
+ } ],
+ "opType" : "SWITCHDATABASE"
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.SetNamespaceCommand",
+ "databaseDescs" : [ {
+ "fieldName" : "namespace",
+ "fieldExtractor" : "StringSeqDatabaseExtractor",
+ "isInput" : true
+ } ],
+ "opType" : "SWITCHDATABASE"
+} ]
\ No newline at end of file
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/resources/function_command_spec.json b/extensions/spark/kyuubi-spark-authz/src/main/resources/function_command_spec.json
new file mode 100644
index 000000000..2211cd9a6
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/resources/function_command_spec.json
@@ -0,0 +1,75 @@
+[ {
+ "classname" : "org.apache.spark.sql.execution.command.CreateFunctionCommand",
+ "functionDescs" : [ {
+ "fieldName" : "functionName",
+ "fieldExtractor" : "StringFunctionExtractor",
+ "databaseDesc" : {
+ "fieldName" : "databaseName",
+ "fieldExtractor" : "StringOptionDatabaseExtractor",
+ "isInput" : false
+ },
+ "functionTypeDesc" : {
+ "fieldName" : "isTemp",
+ "fieldExtractor" : "TempMarkerFunctionTypeExtractor",
+ "skipTypes" : [ "TEMP" ]
+ },
+ "isInput" : false
+ } ],
+ "opType" : "CREATEFUNCTION"
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.DescribeFunctionCommand",
+ "functionDescs" : [ {
+ "fieldName" : "info",
+ "fieldExtractor" : "ExpressionInfoFunctionExtractor",
+ "databaseDesc" : null,
+ "functionTypeDesc" : {
+ "fieldName" : "info",
+ "fieldExtractor" : "ExpressionInfoFunctionTypeExtractor",
+ "skipTypes" : [ "TEMP", "SYSTEM" ]
+ },
+ "isInput" : true
+ }, {
+ "fieldName" : "functionName",
+ "fieldExtractor" : "FunctionIdentifierFunctionExtractor",
+ "databaseDesc" : null,
+ "functionTypeDesc" : {
+ "fieldName" : "functionName",
+ "fieldExtractor" : "FunctionIdentifierFunctionTypeExtractor",
+ "skipTypes" : [ "TEMP", "SYSTEM" ]
+ },
+ "isInput" : true
+ } ],
+ "opType" : "DESCFUNCTION"
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.DropFunctionCommand",
+ "functionDescs" : [ {
+ "fieldName" : "functionName",
+ "fieldExtractor" : "StringFunctionExtractor",
+ "databaseDesc" : {
+ "fieldName" : "databaseName",
+ "fieldExtractor" : "StringOptionDatabaseExtractor",
+ "isInput" : false
+ },
+ "functionTypeDesc" : {
+ "fieldName" : "isTemp",
+ "fieldExtractor" : "TempMarkerFunctionTypeExtractor",
+ "skipTypes" : [ "TEMP" ]
+ },
+ "isInput" : false
+ } ],
+ "opType" : "DROPFUNCTION"
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.RefreshFunctionCommand",
+ "functionDescs" : [ {
+ "fieldName" : "functionName",
+ "fieldExtractor" : "StringFunctionExtractor",
+ "databaseDesc" : {
+ "fieldName" : "databaseName",
+ "fieldExtractor" : "StringOptionDatabaseExtractor",
+ "isInput" : false
+ },
+ "functionTypeDesc" : null,
+ "isInput" : false
+ } ],
+ "opType" : "RELOADFUNCTION"
+} ]
\ No newline at end of file
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/resources/table_command_spec.json b/extensions/spark/kyuubi-spark-authz/src/main/resources/table_command_spec.json
new file mode 100644
index 000000000..ce582efa8
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/resources/table_command_spec.json
@@ -0,0 +1,1174 @@
+[ {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.AddColumns",
+ "tableDescs" : [ {
+ "fieldName" : "child",
+ "fieldExtractor" : "ResolvedTableTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_ADDCOLS",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.AddPartitions",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "DataSourceV2RelationTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_ADDPARTS",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.AlterColumn",
+ "tableDescs" : [ {
+ "fieldName" : "child",
+ "fieldExtractor" : "ResolvedTableTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_ADDCOLS",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.AlterTable",
+ "tableDescs" : [ {
+ "fieldName" : "ident",
+ "fieldExtractor" : "IdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_PROPERTIES",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.AppendData",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "DataSourceV2RelationTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : {
+ "fieldName" : null,
+ "fieldExtractor" : null,
+ "actionType" : "INSERT"
+ },
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "QUERY",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.CacheTable",
+ "tableDescs" : [ ],
+ "opType" : "CREATEVIEW",
+ "queryDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.CacheTableAsSelect",
+ "tableDescs" : [ ],
+ "opType" : "CREATEVIEW",
+ "queryDescs" : [ {
+ "fieldName" : "plan",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.CommentOnTable",
+ "tableDescs" : [ {
+ "fieldName" : "child",
+ "fieldExtractor" : "ResolvedTableTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_PROPERTIES",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.CreateTable",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "IdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "CREATETABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.CreateTableAsSelect",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "IdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "CREATETABLE_AS_SELECT",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.CreateV2Table",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "IdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "CREATETABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.DeleteFromTable",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "DataSourceV2RelationTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : {
+ "fieldName" : null,
+ "fieldExtractor" : null,
+ "actionType" : "UPDATE"
+ },
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "QUERY",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.DropColumns",
+ "tableDescs" : [ {
+ "fieldName" : "child",
+ "fieldExtractor" : "ResolvedTableTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_ADDCOLS",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.DropPartitions",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "DataSourceV2RelationTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_DROPPARTS",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.DropTable",
+ "tableDescs" : [ {
+ "fieldName" : "child",
+ "fieldExtractor" : "ResolvedTableTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ }, {
+ "fieldName" : "ident",
+ "fieldExtractor" : "IdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "DROPTABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.MergeIntoTable",
+ "tableDescs" : [ {
+ "fieldName" : "targetTable",
+ "fieldExtractor" : "DataSourceV2RelationTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : {
+ "fieldName" : null,
+ "fieldExtractor" : null,
+ "actionType" : "UPDATE"
+ },
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "QUERY",
+ "queryDescs" : [ {
+ "fieldName" : "sourceTable",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.OverwriteByExpression",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "DataSourceV2RelationTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : {
+ "fieldName" : null,
+ "fieldExtractor" : null,
+ "actionType" : "INSERT_OVERWRITE"
+ },
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "QUERY",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.OverwritePartitionsDynamic",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "DataSourceV2RelationTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : {
+ "fieldName" : null,
+ "fieldExtractor" : null,
+ "actionType" : "INSERT_OVERWRITE"
+ },
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "QUERY",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.RefreshTable",
+ "tableDescs" : [ {
+ "fieldName" : "child",
+ "fieldExtractor" : "ResolvedTableTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "QUERY",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.RenameColumn",
+ "tableDescs" : [ {
+ "fieldName" : "child",
+ "fieldExtractor" : "ResolvedTableTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_RENAMECOL",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.RenamePartitions",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "DataSourceV2RelationTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_RENAMEPART",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.RepairTable",
+ "tableDescs" : [ {
+ "fieldName" : "child",
+ "fieldExtractor" : "ResolvedTableTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "MSCK",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.ReplaceColumns",
+ "tableDescs" : [ {
+ "fieldName" : "child",
+ "fieldExtractor" : "ResolvedTableTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_REPLACECOLS",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.ReplaceTable",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "IdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "CREATETABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.ReplaceTableAsSelect",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "IdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "CREATETABLE_AS_SELECT",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.ShowCreateTable",
+ "tableDescs" : [ {
+ "fieldName" : "child",
+ "fieldExtractor" : "ResolvedTableTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "SHOW_CREATETABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.ShowTableProperties",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "ResolvedTableTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "SHOW_TBLPROPERTIES",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.TruncatePartition",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "DataSourceV2RelationTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_DROPPARTS",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.TruncateTable",
+ "tableDescs" : [ {
+ "fieldName" : "child",
+ "fieldExtractor" : "ResolvedTableTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "TRUNCATETABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.UpdateTable",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "DataSourceV2RelationTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : {
+ "fieldName" : null,
+ "fieldExtractor" : null,
+ "actionType" : "UPDATE"
+ },
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "QUERY",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AlterTableAddColumnsCommand",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "colsToAdd",
+ "fieldExtractor" : "StructFieldSeqColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_ADDCOLS",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AlterTableAddPartitionCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "partitionSpecsAndLocs",
+ "fieldExtractor" : "PartitionLocsSeqColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_ADDPARTS",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AlterTableChangeColumnCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "columnName",
+ "fieldExtractor" : "StringColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_REPLACECOLS",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AlterTableDropPartitionCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "specs",
+ "fieldExtractor" : "PartitionSeqColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_DROPPARTS",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AlterTableRecoverPartitionsCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "MSCK",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AlterTableRenameCommand",
+ "tableDescs" : [ {
+ "fieldName" : "oldName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : {
+ "fieldName" : null,
+ "fieldExtractor" : null,
+ "actionType" : "DELETE"
+ },
+ "tableTypeDesc" : {
+ "fieldName" : "oldName",
+ "fieldExtractor" : "TableIdentifierTableTypeExtractor",
+ "skipTypes" : [ "TEMP_VIEW" ]
+ },
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ }, {
+ "fieldName" : "newName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : {
+ "fieldName" : "oldName",
+ "fieldExtractor" : "TableIdentifierTableTypeExtractor",
+ "skipTypes" : [ "TEMP_VIEW" ]
+ },
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_RENAME",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AlterTableRenamePartitionCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "oldPartition",
+ "fieldExtractor" : "PartitionColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_RENAMEPART",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AlterTableSerDePropertiesCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "partSpec",
+ "fieldExtractor" : "PartitionOptionColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_SERDEPROPERTIES",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AlterTableSetLocationCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "partitionSpec",
+ "fieldExtractor" : "PartitionOptionColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_LOCATION",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AlterTableSetPropertiesCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_PROPERTIES",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AlterTableUnsetPropertiesCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERTABLE_PROPERTIES",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AlterViewAsCommand",
+ "tableDescs" : [ {
+ "fieldName" : "name",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : {
+ "fieldName" : "name",
+ "fieldExtractor" : "TableIdentifierTableTypeExtractor",
+ "skipTypes" : [ "TEMP_VIEW" ]
+ },
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ALTERVIEW_AS",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AnalyzeColumnCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableIdent",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "columnNames",
+ "fieldExtractor" : "StringSeqColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ }, {
+ "fieldName" : "tableIdent",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "columnNames",
+ "fieldExtractor" : "StringSeqOptionColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ANALYZE_TABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AnalyzePartitionCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableIdent",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "partitionSpec",
+ "fieldExtractor" : "PartitionColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ANALYZE_TABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AnalyzeTableCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableIdent",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ANALYZE_TABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.AnalyzeTablesCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableIdent",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "ANALYZE_TABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.CacheTableCommand",
+ "tableDescs" : [ ],
+ "opType" : "CREATEVIEW",
+ "queryDescs" : [ {
+ "fieldName" : "plan",
+ "fieldExtractor" : "LogicalPlanOptionQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "CatalogTableTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "CREATETABLE_AS_SELECT",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.CreateDataSourceTableCommand",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "CatalogTableTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "CREATETABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.CreateTableCommand",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "CatalogTableTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "CREATETABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.CreateTableLikeCommand",
+ "tableDescs" : [ {
+ "fieldName" : "targetTable",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : true
+ }, {
+ "fieldName" : "sourceTable",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : true
+ } ],
+ "opType" : "CREATETABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.CreateViewCommand",
+ "tableDescs" : [ {
+ "fieldName" : "name",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : {
+ "fieldName" : "viewType",
+ "fieldExtractor" : "ViewTypeTableTypeExtractor",
+ "skipTypes" : [ "TEMP_VIEW", "GLOBAL_TEMP_VIEW" ]
+ },
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "CREATEVIEW",
+ "queryDescs" : [ {
+ "fieldName" : "plan",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ }, {
+ "fieldName" : "child",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.DescribeColumnCommand",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "colNameParts",
+ "fieldExtractor" : "StringSeqLastColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "DESCTABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.DescribeTableCommand",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "partitionSpec",
+ "fieldExtractor" : "PartitionColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : true
+ } ],
+ "opType" : "DESCTABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.DropTableCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "TableIdentifierTableTypeExtractor",
+ "skipTypes" : [ "TEMP_VIEW" ]
+ },
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "DROPTABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.InsertIntoDataSourceDirCommand",
+ "tableDescs" : [ ],
+ "opType" : "QUERY",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.LoadDataCommand",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "partition",
+ "fieldExtractor" : "PartitionOptionColumnExtractor"
+ },
+ "actionTypeDesc" : {
+ "fieldName" : "isOverwrite",
+ "fieldExtractor" : "OverwriteOrInsertActionTypeExtractor",
+ "actionType" : null
+ },
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "LOAD",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.RefreshTableCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableIdent",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "QUERY",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.RepairTableCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "MSCK",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.ShowColumnsCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "SHOWCOLUMNS",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.ShowCreateTableAsSerdeCommand",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "SHOW_CREATETABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.ShowCreateTableCommand",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "SHOW_CREATETABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.ShowPartitionsCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "spec",
+ "fieldExtractor" : "PartitionOptionColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "SHOWPARTITIONS",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.ShowTablePropertiesCommand",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "SHOW_TBLPROPERTIES",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.command.TruncateTableCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableName",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "partitionSpec",
+ "fieldExtractor" : "PartitionOptionColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "TRUNCATETABLE",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.datasources.CreateTempViewUsing",
+ "tableDescs" : [ ],
+ "opType" : "CREATEVIEW",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.datasources.InsertIntoDataSourceCommand",
+ "tableDescs" : [ {
+ "fieldName" : "logicalRelation",
+ "fieldExtractor" : "LogicalRelationTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : {
+ "fieldName" : "overwrite",
+ "fieldExtractor" : "OverwriteOrInsertActionTypeExtractor",
+ "actionType" : null
+ },
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "QUERY",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand",
+ "tableDescs" : [ ],
+ "opType" : "QUERY",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.datasources.InsertIntoHiveDirCommand",
+ "tableDescs" : [ ],
+ "opType" : "QUERY",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.datasources.RefreshTable",
+ "tableDescs" : [ {
+ "fieldName" : "tableIdent",
+ "fieldExtractor" : "TableIdentifierTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : true,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "QUERY",
+ "queryDescs" : [ ]
+}, {
+ "classname" : "org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand",
+ "tableDescs" : [ ],
+ "opType" : "QUERY",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableDesc",
+ "fieldExtractor" : "CatalogTableTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "outputColumnNames",
+ "fieldExtractor" : "StringSeqColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "CREATETABLE_AS_SELECT",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.hive.execution.InsertIntoHiveTable",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "CatalogTableTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "outputColumnNames",
+ "fieldExtractor" : "StringSeqColumnExtractor"
+ },
+ "actionTypeDesc" : {
+ "fieldName" : "overwrite",
+ "fieldExtractor" : "OverwriteOrInsertActionTypeExtractor",
+ "actionType" : null
+ },
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "QUERY",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.hive.execution.OptimizedCreateHiveTableAsSelectCommand",
+ "tableDescs" : [ {
+ "fieldName" : "tableDesc",
+ "fieldExtractor" : "CatalogTableTableExtractor",
+ "columnDesc" : {
+ "fieldName" : "outputColumnNames",
+ "fieldExtractor" : "StringSeqColumnExtractor"
+ },
+ "actionTypeDesc" : null,
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "CREATETABLE_AS_SELECT",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.DeleteFromIcebergTable",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "DataSourceV2RelationTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : {
+ "fieldName" : null,
+ "fieldExtractor" : null,
+ "actionType" : "UPDATE"
+ },
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "QUERY",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable",
+ "tableDescs" : [ {
+ "fieldName" : "targetTable",
+ "fieldExtractor" : "DataSourceV2RelationTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : {
+ "fieldName" : null,
+ "fieldExtractor" : null,
+ "actionType" : "UPDATE"
+ },
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "QUERY",
+ "queryDescs" : [ {
+ "fieldName" : "sourceTable",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+}, {
+ "classname" : "org.apache.spark.sql.catalyst.plans.logical.UpdateIcebergTable",
+ "tableDescs" : [ {
+ "fieldName" : "table",
+ "fieldExtractor" : "DataSourceV2RelationTableExtractor",
+ "columnDesc" : null,
+ "actionTypeDesc" : {
+ "fieldName" : null,
+ "fieldExtractor" : null,
+ "actionType" : "UPDATE"
+ },
+ "tableTypeDesc" : null,
+ "isInput" : false,
+ "setCurrentDatabaseIfMissing" : false
+ } ],
+ "opType" : "QUERY",
+ "queryDescs" : [ {
+ "fieldName" : "query",
+ "fieldExtractor" : "LogicalPlanQueryExtractor"
+ } ]
+} ]
\ No newline at end of file
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/IcebergCommands.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/IcebergCommands.scala
index b2b69be12..c61a1a4e3 100644
--- a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/IcebergCommands.scala
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/IcebergCommands.scala
@@ -17,14 +17,11 @@
package org.apache.kyuubi.plugin.spark.authz
-import scala.collection.mutable.ArrayBuffer
-
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.kyuubi.plugin.spark.authz.OperationType._
import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
-import org.apache.kyuubi.plugin.spark.authz.v2Commands.CommandType.CommandType
/**
* Building privilege objects
@@ -69,9 +66,6 @@ object IcebergCommands extends Enumeration {
* @param operationType OperationType for converting accessType
* @param leastVer minimum Spark version required
* @param mostVer maximum Spark version supported
- * @param commandTypes Seq of [[CommandType]] hinting privilege building
- * @param buildInput input [[PrivilegeObject]] for privilege check
- * @param buildOutput output [[PrivilegeObject]] for privilege check
* @param outputActionType [[PrivilegeObjectActionType]] for output [[PrivilegeObject]]
* @param resolveOutputTableOwner Whether to resolve table owner for output [[PrivilegeObject]]
*/
@@ -79,50 +73,23 @@ object IcebergCommands extends Enumeration {
operationType: OperationType = QUERY,
leastVer: Option[String] = None,
mostVer: Option[String] = None,
- commandTypes: Seq[CommandType] = Seq.empty,
- buildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[CommandType]) => Unit =
- v2Commands.defaultBuildInput,
- buildOutput: (
- LogicalPlan,
- ArrayBuffer[PrivilegeObject],
- Seq[CommandType],
- PrivilegeObjectActionType,
- Boolean) => Unit = v2Commands.defaultBuildOutput,
outputActionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER,
resolveOutputTableOwner: Boolean = true)
- extends super.Val {
-
- def buildPrivileges(
- plan: LogicalPlan,
- inputObjs: ArrayBuffer[PrivilegeObject],
- outputObjs: ArrayBuffer[PrivilegeObject]): Unit = {
- this.buildInput(plan, inputObjs, commandTypes)
- this.buildOutput(
- plan,
- outputObjs,
- commandTypes,
- outputActionType,
- resolveOutputTableOwner)
- }
- }
+ extends super.Val
// dml commands
val DeleteFromIcebergTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
operationType = ALTERTABLE_DROPPARTS,
leastVer = Some("3.2"),
- commandTypes = Seq(v2Commands.CommandType.HasTableAsIdentifierOption),
outputActionType = PrivilegeObjectActionType.UPDATE)
val UpdateIcebergTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
operationType = ALTERTABLE_ADDPARTS,
leastVer = Some("3.2"),
- commandTypes = Seq(v2Commands.CommandType.HasTableAsIdentifierOption),
outputActionType = PrivilegeObjectActionType.UPDATE)
val UnresolvedMergeIntoIcebergTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder()
- val MergeIntoIcebergTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- buildInput = v2Commands.MergeIntoTable.buildInput,
- buildOutput = v2Commands.MergeIntoTable.buildOutput)
+ val MergeIntoIcebergTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder()
}
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/OperationType.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/OperationType.scala
index 2da4637bc..046ab3e2a 100644
--- a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/OperationType.scala
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/OperationType.scala
@@ -29,87 +29,4 @@ object OperationType extends Enumeration {
DROPDATABASE, DROPFUNCTION, DROPTABLE, DROPVIEW, EXPLAIN, LOAD, MSCK, QUERY, RELOADFUNCTION,
SHOWCONF, SHOW_CREATETABLE, SHOWCOLUMNS, SHOWDATABASES, SHOWFUNCTIONS, SHOWPARTITIONS,
SHOWTABLES, SHOW_TBLPROPERTIES, SWITCHDATABASE, TRUNCATETABLE = Value
-
- /**
- * Mapping Spark plan's nodeName to operation type
- * @param clzName nodeName
- * @return
- */
- def apply(clzName: String): OperationType = {
- clzName match {
- case v2Cmd if v2Commands.accept(v2Cmd) =>
- v2Commands.withName(v2Cmd).operationType
-
- case "AddArchivesCommand" => EXPLAIN
- case "AddFilesCommand" => EXPLAIN
- case "AddJarsCommand" => EXPLAIN
- case "AlterDatabasePropertiesCommand" |
- "SetNamespaceProperties" => ALTERDATABASE
- case "AlterDatabaseSetLocationCommand" |
- "SetNamespaceLocation" => ALTERDATABASE_LOCATION
- case "AlterTableAddColumnsCommand" |
- "AlterHoodieTableAddColumnsCommand" => ALTERTABLE_ADDCOLS
- case "AlterTableAddPartitionCommand" => ALTERTABLE_ADDPARTS
- case "AlterTableChangeColumnCommand" => ALTERTABLE_REPLACECOLS
- case "AlterTableDropPartitionCommand" => ALTERTABLE_DROPPARTS
- case "AlterTableRenameCommand" => ALTERTABLE_RENAME
- case "AlterTableRecoverPartitionsCommand" |
- "RepairTableCommand" => MSCK
- case "AlterTableRenamePartitionCommand" => ALTERTABLE_RENAMEPART
- case "AlterTableSerDePropertiesCommand" => ALTERTABLE_SERDEPROPERTIES
- case "AlterTableSetLocationCommand" => ALTERTABLE_LOCATION
- case "AlterTableSetPropertiesCommand" |
- "AlterTableUnsetPropertiesCommand" => ALTERTABLE_PROPERTIES
- case ava if ava.contains("AlterViewAs") => ALTERVIEW_AS
- case ac if ac.startsWith("Analyze") => ANALYZE_TABLE
- case "CreateDatabaseCommand" => CREATEDATABASE
- case "CreateFunctionCommand" | "CreateFunction" => CREATEFUNCTION
- case "CreateDataSourceTableAsSelectCommand" |
- "CreateHiveTableAsSelectCommand" |
- "OptimizedCreateHiveTableAsSelectCommand" => CREATETABLE_AS_SELECT
- case "CreateTableCommand" |
- "CreateDataSourceTableCommand" |
- "CreateTableLikeCommand" => CREATETABLE
- case "CreateViewCommand" |
- "CacheTableCommand" |
- "CreateTempViewUsing" => CREATEVIEW
- case "DescribeDatabaseCommand" | "DescribeNamespace" => DESCDATABASE
- case "DescribeFunctionCommand" => DESCFUNCTION
- case "DescribeColumnCommand" | "DescribeTableCommand" => DESCTABLE
- case "DropDatabaseCommand" | "DropNamespace" => DROPDATABASE
- case "DropFunctionCommand" | "DropFunction" => DROPFUNCTION
- case "DropTableCommand" => DROPTABLE
- case "ExplainCommand" => EXPLAIN
- case "InsertIntoDataSourceCommand" |
- "InsertIntoDataSourceDirCommand" |
- "InsertIntoHiveTable" |
- "InsertIntoHiveDirCommand" |
- "SaveIntoDataSourceCommand" => QUERY
- case "LoadDataCommand" => LOAD
- case "SetCommand" => SHOWCONF
- case "RefreshFunctionCommand" | "RefreshFunction" => RELOADFUNCTION
- case "RefreshTableCommand" | "RefreshTable" => QUERY
- case "SetCatalogCommand" |
- "SetCatalogAndNamespace" |
- "SetNamespaceCommand" |
- "SetDatabaseCommand" => SWITCHDATABASE
- case "ShowCatalogsCommand" |
- "ShowCurrentNamespaceCommand" => SHOWDATABASES
- case "ShowTablesCommand" |
- "ShowViewsCommand" |
- "ShowTables" => SHOWTABLES
- case "ShowColumnsCommand" => SHOWCOLUMNS
- case "ShowCreateTableAsSerdeCommand" |
- "ShowCreateTableCommand" |
- "ShowCreateTable" => SHOW_CREATETABLE
- case "ShowFunctionsCommand" => SHOWFUNCTIONS
- case "ShowPartitionsCommand" => SHOWPARTITIONS
- case "ShowTablePropertiesCommand" |
- "ShowTableProperties" => SHOW_TBLPROPERTIES
- case "TruncateTableCommand" => TRUNCATETABLE
- case "UncacheTableCommand" => DROPVIEW
- case _ => QUERY
- }
- }
-
}
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala
index 4c1f82163..c9950b68d 100644
--- a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala
@@ -20,24 +20,24 @@ package org.apache.kyuubi.plugin.spark.authz
import scala.collection.mutable.ArrayBuffer
import org.apache.spark.sql.SparkSession
-import org.apache.spark.sql.catalyst.{FunctionIdentifier, TableIdentifier}
-import org.apache.spark.sql.catalyst.analysis.{NoSuchTableException, PersistedView, ViewType}
+import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.catalog.CatalogTable
-import org.apache.spark.sql.catalyst.catalog.CatalogTypes.TablePartitionSpec
-import org.apache.spark.sql.catalyst.expressions.{Expression, ExpressionInfo, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.{Expression, NamedExpression}
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.connector.catalog.Identifier
-import org.apache.spark.sql.execution.datasources.LogicalRelation
-import org.apache.spark.sql.types.StructField
+import org.slf4j.LoggerFactory
+import org.apache.kyuubi.plugin.spark.authz.OperationType.OperationType
import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType._
import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType._
+import org.apache.kyuubi.plugin.spark.authz.serde._
import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
import org.apache.kyuubi.plugin.spark.authz.util.PermanentViewMarker
-import org.apache.kyuubi.plugin.spark.authz.v2Commands.v2TablePrivileges
object PrivilegesBuilder {
+ final private val LOG = LoggerFactory.getLogger(getClass)
+
def databasePrivileges(db: String): PrivilegeObject = {
PrivilegeObject(DATABASE, PrivilegeObjectActionType.OTHER, db, db)
}
@@ -50,10 +50,26 @@ object PrivilegesBuilder {
PrivilegeObject(TABLE_OR_VIEW, actionType, table.database.orNull, table.table, columns, owner)
}
+ private def v2TablePrivileges(
+ table: Identifier,
+ columns: Seq[String] = Nil,
+ owner: Option[String] = None,
+ actionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER): PrivilegeObject = {
+ PrivilegeObject(
+ TABLE_OR_VIEW,
+ actionType,
+ quote(table.namespace()),
+ table.name(),
+ columns,
+ owner)
+ }
private def functionPrivileges(
- db: String,
- functionName: String): PrivilegeObject = {
- PrivilegeObject(FUNCTION, PrivilegeObjectActionType.OTHER, db, functionName)
+ function: Function): PrivilegeObject = {
+ PrivilegeObject(
+ FUNCTION,
+ PrivilegeObjectActionType.OTHER,
+ function.database.orNull,
+ function.functionName)
}
private def collectLeaves(expr: Expression): Seq[NamedExpression] = {
@@ -71,29 +87,6 @@ object PrivilegesBuilder {
}
}
- private def isTempView(
- tableIdent: TableIdentifier,
- spark: SparkSession): Boolean = {
- val parts = tableIdent.database match {
- case Some(db) =>
- Seq(db, tableIdent.table)
- case _ =>
- Seq(tableIdent.table)
- }
-
- spark.sessionState.catalog.isTempView(parts)
- }
-
- private def isPersistentFunction(
- functionIdent: FunctionIdentifier,
- spark: SparkSession): Boolean = {
- val (database, funcName) = functionIdent.database match {
- case Some(_) => (functionIdent.database, functionIdent.funcName)
- case _ => (Some(spark.catalog.currentDatabase), functionIdent.funcName)
- }
- spark.sessionState.catalog.isPersistentFunction(FunctionIdentifier(funcName, database))
- }
-
/**
* Build PrivilegeObjects from Spark LogicalPlan
*
@@ -201,416 +194,96 @@ object PrivilegesBuilder {
plan: LogicalPlan,
inputObjs: ArrayBuffer[PrivilegeObject],
outputObjs: ArrayBuffer[PrivilegeObject],
- spark: SparkSession): Unit = {
-
- def getPlanField[T](field: String): T = {
- getFieldVal[T](plan, field)
- }
-
- def getTableName: TableIdentifier = {
- getPlanField[TableIdentifier]("tableName")
- }
-
- def getTableIdent: TableIdentifier = {
- getPlanField[TableIdentifier]("tableIdent")
- }
-
- def getMultipartIdentifier: TableIdentifier = {
- val multipartIdentifier = getPlanField[Seq[String]]("multipartIdentifier")
- assert(multipartIdentifier.nonEmpty)
- val table = multipartIdentifier.last
- val db = Some(quote(multipartIdentifier.init))
- TableIdentifier(table, db)
- }
-
- def getQuery: LogicalPlan = {
- getPlanField[LogicalPlan]("query")
- }
-
- def tablePrivilegesWithOwner(
- tableId: TableIdentifier,
- columns: Seq[String] = Nil,
- actionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER)
- : PrivilegeObject = {
- val owner =
- try {
- val table = spark.sessionState.catalog.getTableMetadata(tableId)
- extractTableOwner(table)
- } catch {
- case _: NoSuchTableException =>
- None
+ spark: SparkSession): OperationType = {
+
+ def getTablePriv(tableDesc: TableDesc): Seq[PrivilegeObject] = {
+ try {
+ val maybeTable = tableDesc.extract(plan, spark)
+ maybeTable match {
+ case Some(table) =>
+ // TODO Use strings instead of ti for general purpose.
+ var identifier = TableIdentifier(table.table, table.database)
+ if (tableDesc.setCurrentDatabaseIfMissing) {
+ identifier = setCurrentDBIfNecessary(identifier, spark)
+ }
+ if (tableDesc.tableTypeDesc.exists(_.skip(plan))) {
+ Nil
+ } else {
+ val actionType = tableDesc.actionTypeDesc.map(_.extract(plan)).getOrElse(OTHER)
+ val columnNames = tableDesc.columnDesc.map(_.extract(plan)).getOrElse(Nil)
+ Seq(tablePrivileges(identifier, columnNames, table.owner, actionType))
+ }
+ case None => Nil
}
- tablePrivileges(tableId, columns, owner, actionType)
+ } catch {
+ case e: Exception =>
+ LOG.warn(tableDesc.error(plan, e))
+ Nil
+ }
}
- plan.nodeName match {
- case v2Cmd if v2Commands.accept(v2Cmd) =>
- v2Commands.withName(v2Cmd).buildPrivileges(plan, inputObjs, outputObjs)
-
- case icebergCmd if IcebergCommands.accept(icebergCmd) =>
- IcebergCommands.withName(icebergCmd).buildPrivileges(plan, inputObjs, outputObjs)
-
- case "AlterDatabasePropertiesCommand" |
- "AlterDatabaseSetLocationCommand" |
- "CreateDatabaseCommand" |
- "DropDatabaseCommand" =>
- val database = getPlanField[String]("databaseName")
- outputObjs += databasePrivileges(database)
-
- case "AlterTableAddColumnsCommand" =>
- val table = getPlanField[TableIdentifier]("table")
- val cols = getPlanField[Seq[StructField]]("colsToAdd").map(_.name)
- outputObjs += tablePrivilegesWithOwner(table, cols)
-
- case "AlterTableAddPartitionCommand" =>
- val table = getTableName
- val cols = getPlanField[Seq[(TablePartitionSpec, Option[String])]]("partitionSpecsAndLocs")
- .flatMap(_._1.keySet).distinct
- outputObjs += tablePrivilegesWithOwner(table, cols)
-
- case "AlterTableChangeColumnCommand" =>
- val table = getTableName
- val cols = getPlanField[String]("columnName") :: Nil
- outputObjs += tablePrivilegesWithOwner(table, cols)
-
- case "AlterTableDropPartitionCommand" =>
- val table = getTableName
- val cols = getPlanField[Seq[TablePartitionSpec]]("specs").flatMap(_.keySet).distinct
- outputObjs += tablePrivilegesWithOwner(table, cols)
-
- case "AlterTableRenameCommand" =>
- val oldTable = getPlanField[TableIdentifier]("oldName")
- val newTable = getPlanField[TableIdentifier]("newName")
- if (!isTempView(oldTable, spark)) {
- outputObjs += tablePrivilegesWithOwner(
- oldTable,
- actionType = PrivilegeObjectActionType.DELETE)
- outputObjs += tablePrivileges(newTable)
- }
-
- // this is for spark 3.1 or below
- case "AlterTableRecoverPartitionsCommand" =>
- val table = getTableName
- outputObjs += tablePrivilegesWithOwner(table)
-
- case "AlterTableRenamePartitionCommand" =>
- val table = getTableName
- val cols = getPlanField[TablePartitionSpec]("oldPartition").keySet.toSeq
- outputObjs += tablePrivilegesWithOwner(table, cols)
-
- case "AlterTableSerDePropertiesCommand" =>
- val table = getTableName
- val cols = getPlanField[Option[TablePartitionSpec]]("partSpec")
- .toSeq.flatMap(_.keySet)
- outputObjs += tablePrivilegesWithOwner(table, cols)
-
- case "AlterTableSetLocationCommand" =>
- val table = getTableName
- val cols = getPlanField[Option[TablePartitionSpec]]("partitionSpec")
- .toSeq.flatMap(_.keySet)
- outputObjs += tablePrivilegesWithOwner(table, cols)
-
- case "AlterTableSetPropertiesCommand" |
- "AlterTableUnsetPropertiesCommand" =>
- val table = getTableName
- outputObjs += tablePrivilegesWithOwner(table)
-
- case "AlterViewAsCommand" =>
- val view = getPlanField[TableIdentifier]("name")
- if (!isTempView(view, spark)) {
- outputObjs += tablePrivilegesWithOwner(view)
- }
- buildQuery(getQuery, inputObjs)
-
- case "AlterViewAs" =>
-
- case "AnalyzeColumnCommand" =>
- val table = getTableIdent
- val cols =
- if (isSparkVersionAtLeast("3.0")) {
- getPlanField[Option[Seq[String]]]("columnNames").getOrElse(Nil)
- } else {
- getPlanField[Seq[String]]("columnNames")
+ plan.getClass.getName match {
+ case classname if DB_COMMAND_SPECS.contains(classname) =>
+ val desc = DB_COMMAND_SPECS(classname)
+ desc.databaseDescs.foreach { databaseDesc =>
+ try {
+ val database = databaseDesc.extract(plan)
+ if (databaseDesc.isInput) {
+ inputObjs += databasePrivileges(database)
+ } else {
+ outputObjs += databasePrivileges(database)
+ }
+ } catch {
+ case e: Exception =>
+ LOG.warn(databaseDesc.error(plan, e))
}
- inputObjs += tablePrivilegesWithOwner(table, cols)
-
- case "AnalyzePartitionCommand" =>
- val table = getTableIdent
- val cols = getPlanField[Map[String, Option[String]]]("partitionSpec")
- .keySet.toSeq
- inputObjs += tablePrivilegesWithOwner(table, cols)
-
- case "AnalyzeTableCommand" |
- "RefreshTableCommand" |
- "RefreshTable" =>
- inputObjs += tablePrivilegesWithOwner(getTableIdent)
-
- case "AnalyzeTablesCommand" =>
- val db = getPlanField[Option[String]]("databaseName")
- if (db.nonEmpty) {
- inputObjs += databasePrivileges(db.get)
}
+ desc.operationType
- case "CacheTableCommand" =>
- getPlanField[Option[LogicalPlan]]("plan").foreach(buildQuery(_, inputObjs))
-
- case "CreateViewCommand" =>
- if (getPlanField[ViewType]("viewType") == PersistedView) {
- val view = getPlanField[TableIdentifier]("name")
- outputObjs += tablePrivileges(view)
- }
- val query =
- if (isSparkVersionAtMost("3.1")) {
- getPlanField[LogicalPlan]("child")
+ case classname if TABLE_COMMAND_SPECS.contains(classname) =>
+ val spec = TABLE_COMMAND_SPECS(classname)
+ spec.tableDescs.foreach { td =>
+ if (td.isInput) {
+ inputObjs ++= getTablePriv(td)
} else {
- getPlanField[LogicalPlan]("plan")
+ outputObjs ++= getTablePriv(td)
}
- buildQuery(query, inputObjs)
-
- case "CreateView" => // revisit this after spark has view catalog
- case "CreateDataSourceTableCommand" | "CreateTableCommand" =>
- val table = getPlanField[CatalogTable]("table").identifier
- // fixme: do we need to add columns to check?
- outputObjs += tablePrivileges(table)
-
- case "CreateDataSourceTableAsSelectCommand" =>
- val table = getPlanField[CatalogTable]("table").identifier
- outputObjs += tablePrivileges(table)
- buildQuery(getQuery, inputObjs)
-
- case "CreateHiveTableAsSelectCommand" |
- "OptimizedCreateHiveTableAsSelectCommand" =>
- val table = getPlanField[CatalogTable]("tableDesc").identifier
- val cols = getPlanField[Seq[String]]("outputColumnNames")
- outputObjs += tablePrivileges(table, cols)
- buildQuery(getQuery, inputObjs)
-
- case "CreateFunctionCommand" |
- "DropFunctionCommand" =>
- val isTemp = getPlanField[Boolean]("isTemp")
- if (!isTemp) {
- val db = getPlanField[Option[String]]("databaseName")
- val functionName = getPlanField[String]("functionName")
- outputObjs += functionPrivileges(db.orNull, functionName)
}
-
- case "RefreshFunctionCommand" =>
- val db = getPlanField[Option[String]]("databaseName")
- val functionName = getPlanField[String]("functionName")
- outputObjs += functionPrivileges(db.orNull, functionName)
-
- case "CreateFunction" | "DropFunction" | "RefreshFunction" =>
- val child = getPlanField("child")
- val database = getFieldVal[Seq[String]](child, "nameParts")
- inputObjs += databasePrivileges(quote(database))
-
- case "CreateTableLikeCommand" =>
- val target = setCurrentDBIfNecessary(getPlanField[TableIdentifier]("targetTable"), spark)
- val source = setCurrentDBIfNecessary(getPlanField[TableIdentifier]("sourceTable"), spark)
- inputObjs += tablePrivilegesWithOwner(source)
- outputObjs += tablePrivileges(target)
-
- case "CreateTempViewUsing" =>
-
- case "DescribeColumnCommand" =>
- val table = getPlanField[TableIdentifier]("table")
- val cols = getPlanField[Seq[String]]("colNameParts").takeRight(1)
- inputObjs += tablePrivilegesWithOwner(table, cols)
-
- case "DescribeTableCommand" =>
- val table = setCurrentDBIfNecessary(getPlanField[TableIdentifier]("table"), spark)
- inputObjs += tablePrivilegesWithOwner(table)
-
- case "DescribeDatabaseCommand" | "SetDatabaseCommand" =>
- val database = getPlanField[String]("databaseName")
- inputObjs += databasePrivileges(database)
-
- case "DescribeNamespace" =>
- val child = getPlanField[Any]("namespace")
- val database = getFieldVal[Seq[String]](child, "namespace")
- inputObjs += databasePrivileges(quote(database))
-
- case "DescribeFunctionCommand" =>
- val (db: Option[String], funName: String) =
- if (isSparkVersionAtLeast("3.3")) {
- val info = getPlanField[ExpressionInfo]("info")
- (Option(info.getDb), info.getName)
- } else {
- val funcIdent = getPlanField[FunctionIdentifier]("functionName")
- (funcIdent.database, funcIdent.funcName)
+ spec.queryDescs.foreach { qd =>
+ try {
+ buildQuery(qd.extract(plan), inputObjs)
+ } catch {
+ case e: Exception =>
+ LOG.warn(qd.error(plan, e))
}
- val isPersistentFun = isPersistentFunction(FunctionIdentifier(funName, db), spark)
- if (isPersistentFun) {
- inputObjs += functionPrivileges(db.orNull, funName)
}
-
- case "DropTableCommand" =>
- if (!isTempView(getPlanField[TableIdentifier]("tableName"), spark)) {
- outputObjs += tablePrivilegesWithOwner(getTableName)
- }
-
- case "ExplainCommand" =>
-
- case "ExternalCommandExecutor" =>
-
- case "InsertIntoDataSourceCommand" =>
- val logicalRelation = getPlanField[LogicalRelation]("logicalRelation")
- logicalRelation.catalogTable.foreach { t =>
- val overwrite = getPlanField[Boolean]("overwrite")
- val actionType = if (overwrite) INSERT_OVERWRITE else INSERT
- outputObjs += tablePrivileges(
- t.identifier,
- owner = extractTableOwner(t),
- actionType = actionType)
- }
- buildQuery(getQuery, inputObjs)
-
- case "InsertIntoDataSourceDirCommand" |
- "SaveIntoDataSourceCommand" |
- "InsertIntoHadoopFsRelationCommand" |
- "InsertIntoHiveDirCommand" =>
- // TODO: Should get the table via datasource options?
- buildQuery(getQuery, inputObjs)
-
- case "InsertIntoHiveTable" =>
- val table = getPlanField[CatalogTable]("table")
- val overwrite = getPlanField[Boolean]("overwrite")
- val actionType = if (overwrite) INSERT_OVERWRITE else INSERT
- val owner = extractTableOwner(table)
- outputObjs += tablePrivileges(table.identifier, owner = owner, actionType = actionType)
- buildQuery(getQuery, inputObjs)
-
- case "LoadDataCommand" =>
- val table = getPlanField[TableIdentifier]("table")
- val overwrite = getPlanField[Boolean]("isOverwrite")
- val actionType = if (overwrite) INSERT_OVERWRITE else INSERT
- val cols = getPlanField[Option[TablePartitionSpec]]("partition")
- .map(_.keySet).getOrElse(Nil)
- outputObjs += tablePrivilegesWithOwner(table, cols.toSeq, actionType = actionType)
-
- case "RepairTableCommand" =>
- val enableAddPartitions = getPlanField[Boolean]("enableAddPartitions")
- if (enableAddPartitions) {
- outputObjs += tablePrivilegesWithOwner(getTableName, actionType = INSERT)
- } else if (getPlanField[Boolean]("enableDropPartitions")) {
- outputObjs += tablePrivilegesWithOwner(getTableName, actionType = DELETE)
- } else {
- inputObjs += tablePrivilegesWithOwner(getTableName)
- }
-
- case "SetCatalogAndNamespace" =>
- if (isSparkVersionAtLeast("3.3")) {
- val child = getPlanField[Any]("child")
- val nameParts = getFieldVal[Seq[String]](child, "nameParts")
- inputObjs += databasePrivileges(quote(nameParts))
-
- } else {
- getPlanField[Option[String]]("catalogName").foreach { catalog =>
- // fixme do we really need to skip spark_catalog?
- if (catalog != "spark_catalog") {
- inputObjs += databasePrivileges(catalog)
+ spec.operationType
+
+ case classname if FUNCTION_COMMAND_SPECS.contains(classname) =>
+ val spec = FUNCTION_COMMAND_SPECS(classname)
+ spec.functionDescs.foreach { fd =>
+ try {
+ val function = fd.extract(plan)
+ if (!fd.functionTypeDesc.exists(_.skip(plan, spark))) {
+ if (fd.isInput) {
+ inputObjs += functionPrivileges(function)
+ } else {
+ outputObjs += functionPrivileges(function)
+ }
}
- }
- getPlanField[Option[Seq[String]]]("namespace").foreach { nameParts =>
- inputObjs += databasePrivileges(quote(nameParts))
+ } catch {
+ case e: Exception =>
+ LOG.warn(fd.error(plan, e))
}
}
+ spec.operationType
- case "SetCatalogCommand" =>
- inputObjs += databasePrivileges(getPlanField[String]("catalogName"))
-
- case "SetNamespaceCommand" =>
- val namespace = quote(getPlanField[Seq[String]]("namespace"))
- inputObjs += databasePrivileges(namespace)
-
- case "TruncateTableCommand" =>
- val table = getTableName
- val cols = getPlanField[Option[TablePartitionSpec]]("partitionSpec")
- .map(_.keySet).getOrElse(Nil)
- outputObjs += tablePrivilegesWithOwner(table, cols.toSeq)
-
- case "ShowColumnsCommand" =>
- inputObjs += tablePrivilegesWithOwner(getTableName)
-
- case "ShowCreateTableCommand" |
- "ShowCreateTableAsSerdeCommand" |
- "ShowTablePropertiesCommand" =>
- val table = getPlanField[TableIdentifier]("table")
- inputObjs += tablePrivilegesWithOwner(table)
-
- case "ShowTableProperties" =>
- val resolvedTable = getPlanField[Any]("table")
- val identifier = getFieldVal[AnyRef](resolvedTable, "identifier")
- val namespace = invoke(identifier, "namespace").asInstanceOf[Array[String]]
- val table = invoke(identifier, "name").asInstanceOf[String]
- val owner = getTableOwnerFromV2Plan(resolvedTable, identifier.asInstanceOf[Identifier])
- inputObjs += PrivilegeObject(
- TABLE_OR_VIEW,
- PrivilegeObjectActionType.OTHER,
- quote(namespace),
- table,
- Nil,
- owner)
-
- case "ShowCreateTable" =>
- val resolvedTable = getPlanField[Any]("child")
- val identifier = getFieldVal[AnyRef](resolvedTable, "identifier")
- val namespace = invoke(identifier, "namespace").asInstanceOf[Array[String]]
- val table = invoke(identifier, "name").asInstanceOf[String]
- val owner = getTableOwnerFromV2Plan(resolvedTable, identifier.asInstanceOf[Identifier])
- inputObjs += PrivilegeObject(
- TABLE_OR_VIEW,
- PrivilegeObjectActionType.OTHER,
- quote(namespace),
- table,
- Nil,
- owner)
-
- case "ShowFunctionsCommand" =>
-
- case "ShowPartitionsCommand" =>
- val cols = getPlanField[Option[TablePartitionSpec]]("spec")
- .map(_.keySet.toSeq).getOrElse(Nil)
- inputObjs += tablePrivilegesWithOwner(getTableName, cols)
-
- case "SetNamespaceProperties" | "SetNamespaceLocation" =>
- val resolvedNamespace = getPlanField[Any]("namespace")
- val databases = getFieldVal[Seq[String]](resolvedNamespace, "namespace")
- outputObjs += databasePrivileges(quote(databases))
-
- case "ReplaceArcticData" =>
- val relation = getPlanField[Any]("table")
- val identifier = getFieldVal[Option[Identifier]](relation, "identifier")
- val namespace = invoke(identifier.get, "namespace").asInstanceOf[Array[String]]
- val table = invoke(identifier.get, "name").asInstanceOf[String]
- val owner = getTableOwnerFromV2Plan(relation, identifier.get)
- outputObjs += PrivilegeObject(
- TABLE_OR_VIEW,
- PrivilegeObjectActionType.UPDATE,
- quote(namespace),
- table,
- Nil,
- owner)
-
- case _ =>
- // AddArchivesCommand
- // AddFileCommand
- // AddJarCommand
- // ClearCacheCommand
- // DescribeQueryCommand
- // ExplainCommand
- // ListArchivesCommand
- // ListFilesCommand
- // ListJarsCommand
- // RefreshResource
- // ResetCommand
- // SetCommand
- // ShowDatabasesCommand
- // StreamingExplainCommand
- // UncacheTableCommand
+ case _ => OperationType.QUERY
}
}
+ type PrivilegesAndOpType = (Seq[PrivilegeObject], Seq[PrivilegeObject], OperationType)
+
/**
* Build input and output privilege objects from a Spark's LogicalPlan
*
@@ -623,15 +296,17 @@ object PrivilegesBuilder {
*/
def build(
plan: LogicalPlan,
- spark: SparkSession): (Seq[PrivilegeObject], Seq[PrivilegeObject]) = {
+ spark: SparkSession): PrivilegesAndOpType = {
val inputObjs = new ArrayBuffer[PrivilegeObject]
val outputObjs = new ArrayBuffer[PrivilegeObject]
- plan match {
+ val opType = plan match {
// RunnableCommand
case cmd: Command => buildCommand(cmd, inputObjs, outputObjs, spark)
// Queries
- case _ => buildQuery(plan, inputObjs)
+ case _ =>
+ buildQuery(plan, inputObjs)
+ OperationType.QUERY
}
- (inputObjs, outputObjs)
+ (inputObjs, outputObjs, opType)
}
}
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RuleApplyRowFilterAndDataMasking.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RuleApplyRowFilterAndDataMasking.scala
index a30e2e4ff..ed6910f78 100644
--- a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RuleApplyRowFilterAndDataMasking.scala
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RuleApplyRowFilterAndDataMasking.scala
@@ -24,7 +24,8 @@ import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, Project
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.connector.catalog.Identifier
-import org.apache.kyuubi.plugin.spark.authz.{IcebergCommands, ObjectType, OperationType}
+import org.apache.kyuubi.plugin.spark.authz.{IcebergCommands, ObjectType}
+import org.apache.kyuubi.plugin.spark.authz.serde._
import org.apache.kyuubi.plugin.spark.authz.util.{PermanentViewMarker, RowFilterAndDataMaskingMarker}
import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
@@ -82,7 +83,7 @@ class RuleApplyRowFilterAndDataMasking(spark: SparkSession) extends Rule[Logical
identifier: TableIdentifier,
spark: SparkSession): LogicalPlan = {
val ugi = getAuthzUgi(spark.sparkContext)
- val opType = OperationType(plan.nodeName)
+ val opType = operationType(plan)
val parse = spark.sessionState.sqlParser.parseExpression _
val are = AccessResource(ObjectType.TABLE, identifier.database.orNull, identifier.table, null)
val art = AccessRequest(are, ugi, opType, AccessType.SELECT)
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RuleAuthorization.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RuleAuthorization.scala
index 2c946adbc..54ab33860 100644
--- a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RuleAuthorization.scala
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RuleAuthorization.scala
@@ -47,8 +47,7 @@ object RuleAuthorization {
def checkPrivileges(spark: SparkSession, plan: LogicalPlan): Unit = {
val auditHandler = new SparkRangerAuditHandler
val ugi = getAuthzUgi(spark.sparkContext)
- val opType = OperationType(plan.nodeName)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, opType) = PrivilegesBuilder.build(plan, spark)
val requests = new ArrayBuffer[AccessRequest]()
if (inputs.isEmpty && opType == OperationType.SHOWDATABASES) {
val resource = AccessResource(DATABASE, null)
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/CommandSpec.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/CommandSpec.scala
new file mode 100644
index 000000000..d4d065c80
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/CommandSpec.scala
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.serde
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType
+import org.apache.kyuubi.plugin.spark.authz.OperationType.OperationType
+
+/**
+ * A command specification contains
+ * - different [[Descriptor]]s for specific implementations. It's a list to cover:
+ * - A command may have multiple object to describe, such as create table A like B
+ * - An object descriptor may vary through spark versions, it wins at least once if one of
+ * the descriptors matches
+ * - the classname of a command which this spec point to
+ * - the [[OperationType]] of this command which finally maps to an access privilege
+ */
+trait CommandSpec {
+ def classname: String
+ def opType: String
+ final def operationType: OperationType = OperationType.withName(opType)
+}
+
+/**
+ * A specification describe a database command
+ *
+ * @param classname the database command classname
+ * @param databaseDescs a list of database descriptors
+ * @param opType operation type, e.g. CREATEDATABASE
+ */
+case class DatabaseCommandSpec(
+ classname: String,
+ databaseDescs: Seq[DatabaseDesc],
+ opType: String = "QUERY") extends CommandSpec {}
+
+/**
+ * A specification describe a function command
+ *
+ * @param classname the database command classname
+ * @param functionDescs a list of function descriptors
+ * @param opType operation type, e.g. DROPFUNCTION
+ */
+case class FunctionCommandSpec(
+ classname: String,
+ functionDescs: Seq[FunctionDesc],
+ opType: String) extends CommandSpec
+
+/**
+ * A specification describe a table command
+ *
+ * @param classname the database command classname
+ * @param tableDescs a list of table descriptors
+ * @param opType operation type, e.g. DROPFUNCTION
+ * @param queryDescs the query descriptors a table command may have
+ */
+case class TableCommandSpec(
+ classname: String,
+ tableDescs: Seq[TableDesc],
+ opType: String = "QUERY",
+ queryDescs: Seq[QueryDesc] = Nil) extends CommandSpec
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/Descriptor.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/Descriptor.scala
new file mode 100644
index 000000000..325d8e918
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/Descriptor.scala
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.serde
+
+import org.apache.spark.SPARK_VERSION
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.serde.ActionTypeExtractor.actionTypeExtractors
+import org.apache.kyuubi.plugin.spark.authz.serde.ColumnExtractor.columnExtractors
+import org.apache.kyuubi.plugin.spark.authz.serde.DatabaseExtractor.dbExtractors
+import org.apache.kyuubi.plugin.spark.authz.serde.FunctionExtractor.functionExtractors
+import org.apache.kyuubi.plugin.spark.authz.serde.FunctionType.FunctionType
+import org.apache.kyuubi.plugin.spark.authz.serde.FunctionTypeExtractor.functionTypeExtractors
+import org.apache.kyuubi.plugin.spark.authz.serde.QueryExtractor.queryExtractors
+import org.apache.kyuubi.plugin.spark.authz.serde.TableExtractor.tableExtractors
+import org.apache.kyuubi.plugin.spark.authz.serde.TableType.TableType
+import org.apache.kyuubi.plugin.spark.authz.serde.TableTypeExtractor.tableTypeExtractors
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+/**
+ * A database object(such as database, table, function) descriptor describes its name and getter
+ * in/from another object(such as a spark sql command).
+ */
+sealed trait Descriptor {
+
+ /**
+ * Describes the field name, such as a table field name in the spark logical plan.
+ *
+ * @return the database object field name
+ */
+ def fieldName: String
+
+ /**
+ * The key that points the [[Extractor]] to for the field, which extracts key information
+ * we want to represents a table, column list, etc.
+ */
+ def fieldExtractor: String
+
+ /**
+ * Apply the field [[Extractor]] to the field value to extract key information we want to
+ * represents a table, column list, etc.
+ * @param v
+ * @return
+ */
+ def extract(v: AnyRef): AnyRef
+
+ final def error(v: AnyRef, e: Throwable): String = {
+ val resourceName = getClass.getSimpleName.stripSuffix("Desc")
+ val objectClass = v.getClass.getName
+ s"[Spark$SPARK_VERSION] failed to get $resourceName from $objectClass by" +
+ s" $fieldExtractor/$fieldName, due to ${e.getMessage}"
+ }
+}
+
+/**
+ * Column Descriptor
+ *
+ * @param fieldName the field name or method name of this column field
+ * @param fieldExtractor the key of a [[ColumnExtractor]] instance
+ */
+case class ColumnDesc(
+ fieldName: String,
+ fieldExtractor: String) extends Descriptor {
+ override def extract(v: AnyRef): Seq[String] = {
+ val columnsVal = invoke(v, fieldName)
+ val columnExtractor = columnExtractors(fieldExtractor)
+ columnExtractor(columnsVal)
+ }
+}
+
+/**
+ * Database Descriptor
+ *
+ * @param fieldName the field name or method name of this database field
+ * @param fieldExtractor the key of a [[DatabaseExtractor]] instance
+ * @param isInput read or write
+ */
+case class DatabaseDesc(
+ fieldName: String,
+ fieldExtractor: String,
+ isInput: Boolean = false) extends Descriptor {
+ override def extract(v: AnyRef): String = {
+ val databaseVal = invoke(v, fieldName)
+ val databaseExtractor = dbExtractors(fieldExtractor)
+ databaseExtractor(databaseVal)
+ }
+}
+
+/**
+ * Function Type Descriptor
+ *
+ * @param fieldName the field name or method name of this function type field
+ * @param fieldExtractor the key of a [[FunctionTypeExtractor]] instance
+ * @param skipTypes which kinds of functions are skipped checking
+ */
+case class FunctionTypeDesc(
+ fieldName: String,
+ fieldExtractor: String,
+ skipTypes: Seq[String]) extends Descriptor {
+ override def extract(v: AnyRef): FunctionType = {
+ getValue(v, SparkSession.active)
+ }
+
+ def getValue(v: AnyRef, spark: SparkSession): FunctionType = {
+ val functionTypeVal = invoke(v, fieldName)
+ val functionTypeExtractor = functionTypeExtractors(fieldExtractor)
+ functionTypeExtractor(functionTypeVal, spark)
+ }
+
+ def skip(v: AnyRef, spark: SparkSession): Boolean = {
+ skipTypes.exists(skipType => getValue(v, spark) == FunctionType.withName(skipType))
+ }
+}
+
+/**
+ * Function Descriptor
+ *
+ * @param fieldName the field name or method name of this function field
+ * @param fieldExtractor the key of a [[FunctionExtractor]] instance
+ * @param databaseDesc which kinds of functions are skipped checking
+ * @param functionTypeDesc indicates the function type if necessary
+ * @param isInput read or write
+ */
+case class FunctionDesc(
+ fieldName: String,
+ fieldExtractor: String,
+ databaseDesc: Option[DatabaseDesc] = None,
+ functionTypeDesc: Option[FunctionTypeDesc] = None,
+ isInput: Boolean = false) extends Descriptor {
+ override def extract(v: AnyRef): Function = {
+ val functionVal = invoke(v, fieldName)
+ val functionExtractor = functionExtractors(fieldExtractor)
+ var function = functionExtractor(functionVal)
+ if (function.database.isEmpty) {
+ function = function.copy(database = databaseDesc.map(_.extract(v)))
+ }
+ function
+ }
+}
+
+/**
+ * Query Descriptor which represents one or more query fields of a command
+ *
+ * @param fieldName the field name or method name of this query field
+ * @param fieldExtractor the key of a [[QueryExtractor]] instance
+ * The default value is [[LogicalPlanQueryExtractor]] which
+ * return the original plan directly.
+ */
+case class QueryDesc(
+ fieldName: String,
+ fieldExtractor: String = "LogicalPlanQueryExtractor") extends Descriptor {
+ override def extract(v: AnyRef): LogicalPlan = {
+ val queryVal = invoke(v, fieldName)
+ val queryExtractor = queryExtractors(fieldExtractor)
+ queryExtractor(queryVal)
+ }
+}
+
+/**
+ * Table Type Descriptor
+ *
+ * @param fieldName the field name or method name of this table type field
+ * @param fieldExtractor the key of a [[TableTypeExtractor]] instance
+ * @param skipTypes which kinds of table or view are skipped checking
+ */
+case class TableTypeDesc(
+ fieldName: String,
+ fieldExtractor: String,
+ skipTypes: Seq[String]) extends Descriptor {
+ override def extract(v: AnyRef): TableType = {
+ extract(v, SparkSession.active)
+ }
+
+ def extract(v: AnyRef, spark: SparkSession): TableType = {
+ val tableTypeVal = invoke(v, fieldName)
+ val tableTypeExtractor = tableTypeExtractors(fieldExtractor)
+ tableTypeExtractor(tableTypeVal, spark)
+ }
+
+ def skip(v: AnyRef): Boolean = {
+ skipTypes.exists(skipType => extract(v) == TableType.withName(skipType))
+ }
+}
+
+/**
+ * Table Descriptor
+ *
+ * @param fieldName the field name or method name of this table field
+ * @param fieldExtractor the key of a [[TableExtractor]] instance
+ * @param columnDesc optional [[ColumnDesc]] instance if columns field are specified
+ * @param actionTypeDesc optional [[ActionTypeDesc]] indicates the action type
+ * @param tableTypeDesc optional [[TableTypeDesc]] indicates the table type
+ * @param isInput read or write
+ * @param setCurrentDatabaseIfMissing whether to use current database if the database
+ * field is missing
+ */
+case class TableDesc(
+ fieldName: String,
+ fieldExtractor: String,
+ columnDesc: Option[ColumnDesc] = None,
+ actionTypeDesc: Option[ActionTypeDesc] = None,
+ tableTypeDesc: Option[TableTypeDesc] = None,
+ isInput: Boolean = false,
+ setCurrentDatabaseIfMissing: Boolean = false) extends Descriptor {
+ override def extract(v: AnyRef): Option[Table] = {
+ extract(v, SparkSession.active)
+ }
+
+ def extract(v: AnyRef, spark: SparkSession): Option[Table] = {
+ val tableVal = invoke(v, fieldName)
+ val tableExtractor = tableExtractors(fieldExtractor)
+ tableExtractor(spark, tableVal)
+ }
+}
+
+/**
+ * Action Type Descriptor
+ *
+ * @param fieldName the field name or method name of this action type field
+ * @param fieldExtractor the key of a [[ActionTypeExtractor]] instance
+ * @param actionType the explicitly given action type which take precedence over extracting
+ */
+case class ActionTypeDesc(
+ fieldName: String,
+ fieldExtractor: String,
+ actionType: Option[String] = None) extends Descriptor {
+ override def extract(v: AnyRef): PrivilegeObjectActionType = {
+ actionType.map(PrivilegeObjectActionType.withName).getOrElse {
+ val actionTypeVal = invoke(v, fieldName)
+ val extractor = actionTypeExtractors(fieldExtractor)
+ extractor(actionTypeVal)
+ }
+ }
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/Extractor.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/Extractor.scala
new file mode 100644
index 000000000..0fe1d5268
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/Extractor.scala
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.serde
+
+/**
+ * Base trait for an field extractor
+ */
+trait Extractor {
+
+ /**
+ * the key string of this extractor
+ */
+ def key: String = getClass.getSimpleName
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/Function.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/Function.scala
new file mode 100644
index 000000000..b7a0010b4
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/Function.scala
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.serde
+
+/**
+ * :: Developer API ::
+ *
+ * Represents a function identity
+ *
+ * @param database
+ * @param functionName
+ */
+case class Function(database: Option[String], functionName: String)
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/Table.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/Table.scala
new file mode 100644
index 000000000..ab50e3622
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/Table.scala
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.serde
+
+/**
+ * :: Developer API ::
+ *
+ * Represents a table identity with owner
+ *
+ * TODO(kent yao): How about adding a catalog name field here
+ *
+ * @param database database name or None
+ * @param table table name
+ * @param owner table owner if any, otherwise None
+ */
+case class Table(
+ database: Option[String],
+ table: String,
+ owner: Option[String])
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/actionTypeExtractors.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/actionTypeExtractors.scala
new file mode 100644
index 000000000..e01a6d399
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/actionTypeExtractors.scala
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.serde
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType._
+
+trait ActionTypeExtractor extends (AnyRef => PrivilegeObjectActionType) with Extractor
+
+object ActionTypeExtractor {
+ val actionTypeExtractors: Map[String, ActionTypeExtractor] = {
+ ServiceLoader.load(classOf[ActionTypeExtractor])
+ .iterator()
+ .asScala
+ .map(e => (e.key, e))
+ .toMap
+ }
+}
+
+class OverwriteOrInsertActionTypeExtractor extends ActionTypeExtractor {
+ override def apply(v1: AnyRef): PrivilegeObjectActionType = {
+ if (v1.asInstanceOf[Boolean]) INSERT_OVERWRITE else INSERT
+ }
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/columnExtractors.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/columnExtractors.scala
new file mode 100644
index 000000000..23ef665d8
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/columnExtractors.scala
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.serde
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.sql.types.StructField
+
+trait ColumnExtractor extends (AnyRef => Seq[String]) with Extractor
+
+object ColumnExtractor {
+ val columnExtractors: Map[String, ColumnExtractor] = {
+ ServiceLoader.load(classOf[ColumnExtractor])
+ .iterator()
+ .asScala
+ .map(e => (e.key, e))
+ .toMap
+ }
+}
+class StringColumnExtractor extends ColumnExtractor {
+ override def apply(v1: AnyRef): Seq[String] = {
+ Seq(v1.asInstanceOf[String])
+ }
+}
+
+class StringSeqColumnExtractor extends ColumnExtractor {
+ override def apply(v1: AnyRef): Seq[String] = {
+ v1.asInstanceOf[Seq[String]]
+ }
+}
+
+class StringSeqLastColumnExtractor extends ColumnExtractor {
+ override def apply(v1: AnyRef): Seq[String] = {
+ v1.asInstanceOf[Seq[String]].takeRight(1)
+ }
+}
+
+class StringSeqOptionColumnExtractor extends ColumnExtractor {
+ override def apply(v1: AnyRef): Seq[String] = {
+ v1.asInstanceOf[Option[Seq[String]]].getOrElse(Nil)
+ }
+}
+
+class StructFieldSeqColumnExtractor extends ColumnExtractor {
+ override def apply(v1: AnyRef): Seq[String] = {
+ v1.asInstanceOf[Seq[StructField]].map(_.name)
+ }
+}
+
+class PartitionColumnExtractor extends ColumnExtractor {
+ override def apply(v1: AnyRef): Seq[String] = {
+ v1.asInstanceOf[Map[String, _]].keySet.toSeq
+ }
+}
+
+class PartitionOptionColumnExtractor extends ColumnExtractor {
+ override def apply(v1: AnyRef): Seq[String] = {
+ v1.asInstanceOf[Option[Map[String, _]]].toSeq.flatMap(_.keySet)
+ }
+}
+
+class PartitionSeqColumnExtractor extends ColumnExtractor {
+ override def apply(v1: AnyRef): Seq[String] = {
+ v1.asInstanceOf[Seq[Map[String, _]]].flatMap(_.keySet).distinct
+ }
+}
+
+class PartitionLocsSeqColumnExtractor extends ColumnExtractor {
+ override def apply(v1: AnyRef): Seq[String] = {
+ v1.asInstanceOf[Seq[(Map[String, _], _)]].flatMap(_._1.keySet).distinct
+ }
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/databaseExtractors.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/databaseExtractors.scala
new file mode 100644
index 000000000..b4b600b5b
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/databaseExtractors.scala
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.serde
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+trait DatabaseExtractor extends (AnyRef => String) with Extractor
+
+object DatabaseExtractor {
+ val dbExtractors: Map[String, DatabaseExtractor] = {
+ ServiceLoader.load(classOf[DatabaseExtractor])
+ .iterator()
+ .asScala
+ .map(e => (e.key, e))
+ .toMap
+ }
+}
+
+/**
+ * String
+ */
+class StringDatabaseExtractor extends DatabaseExtractor {
+ override def apply(v1: AnyRef): String = {
+ v1.asInstanceOf[String]
+ }
+}
+
+/**
+ * Option[String]
+ */
+class StringOptionDatabaseExtractor extends DatabaseExtractor {
+ override def apply(v1: AnyRef): String = {
+ v1.asInstanceOf[Option[String]].orNull
+ }
+}
+
+/**
+ * Seq[String]
+ */
+class StringSeqDatabaseExtractor extends DatabaseExtractor {
+ override def apply(v1: AnyRef): String = {
+ quote(v1.asInstanceOf[Seq[String]])
+ }
+}
+
+/**
+ * Option[Seq[String]]
+ */
+class StringSeqOptionDatabaseExtractor extends DatabaseExtractor {
+ override def apply(v1: AnyRef): String = {
+ v1.asInstanceOf[Option[Seq[String]]].map(quote).orNull
+ }
+}
+
+/**
+ * org.apache.spark.sql.catalyst.analysis.ResolvedNamespace
+ */
+class ResolvedNamespaceDatabaseExtractor extends DatabaseExtractor {
+ override def apply(v1: AnyRef): String = {
+ val namespace = getFieldVal[Seq[String]](v1, "namespace")
+ quote(namespace)
+ }
+}
+
+class ResolvedDBObjectNameDatabaseExtractor extends DatabaseExtractor {
+ override def apply(v1: AnyRef): String = {
+ val namespace = getFieldVal[Seq[String]](v1, "nameParts")
+ quote(namespace)
+ }
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/functionExtractors.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/functionExtractors.scala
new file mode 100644
index 000000000..971203a59
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/functionExtractors.scala
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.serde
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.sql.catalyst.FunctionIdentifier
+import org.apache.spark.sql.catalyst.expressions.ExpressionInfo
+
+trait FunctionExtractor extends (AnyRef => Function) with Extractor
+
+object FunctionExtractor {
+ val functionExtractors: Map[String, FunctionExtractor] = {
+ ServiceLoader.load(classOf[FunctionExtractor])
+ .iterator()
+ .asScala
+ .map(e => (e.key, e))
+ .toMap
+ }
+}
+
+class StringFunctionExtractor extends FunctionExtractor {
+ override def apply(v1: AnyRef): Function = {
+ Function(None, v1.asInstanceOf[String])
+ }
+}
+
+class FunctionIdentifierFunctionExtractor extends FunctionExtractor {
+ override def apply(v1: AnyRef): Function = {
+ val identifier = v1.asInstanceOf[FunctionIdentifier]
+ Function(identifier.database, identifier.funcName)
+ }
+}
+
+class ExpressionInfoFunctionExtractor extends FunctionExtractor {
+ override def apply(v1: AnyRef): Function = {
+ val info = v1.asInstanceOf[ExpressionInfo]
+ Function(Option(info.getDb), info.getName)
+ }
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/functionTypeExtractors.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/functionTypeExtractors.scala
new file mode 100644
index 000000000..4509ed6de
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/functionTypeExtractors.scala
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.serde
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.FunctionIdentifier
+
+import org.apache.kyuubi.plugin.spark.authz.serde.FunctionType.{FunctionType, PERMANENT, SYSTEM, TEMP}
+
+object FunctionType extends Enumeration {
+ type FunctionType = Value
+ val PERMANENT, TEMP, SYSTEM = Value
+}
+
+trait FunctionTypeExtractor extends ((AnyRef, SparkSession) => FunctionType) with Extractor
+
+object FunctionTypeExtractor {
+ val functionTypeExtractors: Map[String, FunctionTypeExtractor] = {
+ ServiceLoader.load(classOf[FunctionTypeExtractor])
+ .iterator()
+ .asScala
+ .map(e => (e.key, e))
+ .toMap
+ }
+}
+
+/**
+ * org.apache.spark.sql.catalyst.analysis.ViewType
+ */
+class TempMarkerFunctionTypeExtractor extends FunctionTypeExtractor {
+ override def apply(v1: AnyRef, spark: SparkSession): FunctionType = {
+ if (v1.toString.toBoolean) {
+ TEMP
+ } else {
+ PERMANENT
+ }
+ }
+}
+
+class ExpressionInfoFunctionTypeExtractor extends FunctionTypeExtractor {
+ override def apply(v1: AnyRef, spark: SparkSession): FunctionType = {
+ val function = new ExpressionInfoFunctionExtractor().apply(v1)
+ val fi = FunctionIdentifier(function.functionName, function.database)
+ new FunctionIdentifierFunctionTypeExtractor().apply(fi, spark)
+ }
+}
+
+class FunctionIdentifierFunctionTypeExtractor extends FunctionTypeExtractor {
+ override def apply(v1: AnyRef, spark: SparkSession): FunctionType = {
+ val catalog = spark.sessionState.catalog
+ val fi = v1.asInstanceOf[FunctionIdentifier]
+ if (catalog.isTemporaryFunction(fi)) {
+ TEMP
+ } else if (catalog.isPersistentFunction(fi)) {
+ PERMANENT
+ } else if (catalog.isRegisteredFunction(fi)) {
+ SYSTEM
+ } else {
+ TEMP
+ }
+ }
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/package.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/package.scala
new file mode 100644
index 000000000..a5fb4997c
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/package.scala
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import com.fasterxml.jackson.databind.json.JsonMapper
+import com.fasterxml.jackson.module.scala.{ClassTagExtensions, DefaultScalaModule}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType.{OperationType, QUERY}
+
+package object serde {
+
+ final val mapper = JsonMapper.builder()
+ .addModule(DefaultScalaModule)
+ .build() :: ClassTagExtensions
+
+ final lazy val DB_COMMAND_SPECS: Map[String, DatabaseCommandSpec] = {
+ val is = getClass.getClassLoader.getResourceAsStream("database_command_spec.json")
+ mapper.readValue[Array[DatabaseCommandSpec]](is).map(e => (e.classname, e)).toMap
+ }
+
+ final lazy val TABLE_COMMAND_SPECS: Map[String, TableCommandSpec] = {
+ val is = getClass.getClassLoader.getResourceAsStream("table_command_spec.json")
+ mapper.readValue[Array[TableCommandSpec]](is).map(e => (e.classname, e)).toMap
+ }
+
+ final lazy val FUNCTION_COMMAND_SPECS: Map[String, FunctionCommandSpec] = {
+ val is = getClass.getClassLoader.getResourceAsStream("function_command_spec.json")
+ mapper.readValue[Array[FunctionCommandSpec]](is).map(e => (e.classname, e)).toMap
+ }
+
+ def operationType(plan: LogicalPlan): OperationType = {
+ val classname = plan.getClass.getName
+ TABLE_COMMAND_SPECS.get(classname)
+ .orElse(DB_COMMAND_SPECS.get(classname))
+ .orElse(FUNCTION_COMMAND_SPECS.get(classname))
+ .map(s => s.operationType)
+ .getOrElse(QUERY)
+ }
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/queryExtractors.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/queryExtractors.scala
new file mode 100644
index 000000000..1cf625cf8
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/queryExtractors.scala
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.serde
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+
+trait QueryExtractor extends (AnyRef => LogicalPlan) with Extractor
+
+object QueryExtractor {
+ val queryExtractors: Map[String, QueryExtractor] = {
+ ServiceLoader.load(classOf[QueryExtractor])
+ .iterator()
+ .asScala
+ .map(e => (e.key, e))
+ .toMap
+ }
+}
+
+class LogicalPlanQueryExtractor extends QueryExtractor {
+ override def apply(v1: AnyRef): LogicalPlan = {
+ v1.asInstanceOf[LogicalPlan]
+ }
+}
+
+class LogicalPlanOptionQueryExtractor extends QueryExtractor {
+ override def apply(v1: AnyRef): LogicalPlan = {
+ v1.asInstanceOf[Option[LogicalPlan]].orNull
+ }
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/tableExtractors.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/tableExtractors.scala
new file mode 100644
index 000000000..02cafad39
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/tableExtractors.scala
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.serde
+
+import java.util.{Map => JMap, ServiceLoader}
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+/**
+ * A trait for extracting database and table as string tuple
+ * from the give object whose class type is define by `key`.
+ */
+trait TableExtractor extends ((SparkSession, AnyRef) => Option[Table]) with Extractor
+
+object TableExtractor {
+ val tableExtractors: Map[String, TableExtractor] = {
+ ServiceLoader.load(classOf[TableExtractor])
+ .iterator()
+ .asScala
+ .map(e => (e.key, e))
+ .toMap
+ }
+
+ /**
+ * Get table owner from table properties
+ * @param v a object contains a org.apache.spark.sql.connector.catalog.Table
+ * @return owner
+ */
+ def getOwner(v: AnyRef): Option[String] = {
+ // org.apache.spark.sql.connector.catalog.Table
+ val table = invoke(v, "table")
+ val properties = invoke(table, "properties").asInstanceOf[JMap[String, String]].asScala
+ properties.get("owner")
+ }
+}
+
+/**
+ * TableIdentifier ->
+ */
+class TableIdentifierTableExtractor extends TableExtractor {
+ override def apply(spark: SparkSession, v1: AnyRef): Option[Table] = {
+ val identifier = v1.asInstanceOf[TableIdentifier]
+ val owner =
+ try {
+ val catalogTable = spark.sessionState.catalog.getTableMetadata(identifier)
+ Option(catalogTable.owner).filter(_.nonEmpty)
+ } catch {
+ case _: Exception => None
+ }
+ Some(Table(identifier.database, identifier.table, owner))
+ }
+}
+
+/**
+ * CatalogTable ->
+ */
+class CatalogTableTableExtractor extends TableExtractor {
+ override def apply(spark: SparkSession, v1: AnyRef): Option[Table] = {
+ val catalogTable = v1.asInstanceOf[CatalogTable]
+ val identifier = catalogTable.identifier
+ val owner = Option(catalogTable.owner).filter(_.nonEmpty)
+ Some(Table(identifier.database, identifier.table, owner))
+ }
+}
+
+/**
+ * org.apache.spark.sql.catalyst.analysis.ResolvedTable
+ */
+class ResolvedTableTableExtractor extends TableExtractor {
+ override def apply(spark: SparkSession, v1: AnyRef): Option[Table] = {
+ val identifier = invoke(v1, "identifier")
+ val maybeTable = new IdentifierTableExtractor().apply(spark, identifier)
+ val maybeOwner = TableExtractor.getOwner(v1)
+ maybeTable.map(_.copy(owner = maybeOwner))
+ }
+}
+
+/**
+ * org.apache.spark.sql.connector.catalog.Identifier
+ */
+class IdentifierTableExtractor extends TableExtractor {
+ override def apply(spark: SparkSession, v1: AnyRef): Option[Table] = {
+ val namespace = invoke(v1, "namespace").asInstanceOf[Array[String]]
+ val table = invoke(v1, "name").asInstanceOf[String]
+ Some(Table(Some(quote(namespace)), table, None))
+ }
+}
+
+/**
+ * DataSourceV2Relation ->
+ */
+class DataSourceV2RelationTableExtractor extends TableExtractor {
+ override def apply(spark: SparkSession, v1: AnyRef): Option[Table] = {
+ val plan = v1.asInstanceOf[LogicalPlan]
+ val v2Relation = plan.find(_.getClass.getSimpleName == "DataSourceV2Relation")
+ if (v2Relation.isEmpty) {
+ None
+ } else {
+ val maybeIdentifier = invoke(v2Relation.get, "identifier").asInstanceOf[Option[AnyRef]]
+ maybeIdentifier.flatMap { id =>
+ val maybeTable = new IdentifierTableExtractor().apply(spark, id)
+ val maybeOwner = TableExtractor.getOwner(v2Relation.get)
+ maybeTable.map(_.copy(owner = maybeOwner))
+ }
+ }
+ }
+}
+
+/**
+ * org.apache.spark.sql.execution.datasources.LogicalRelation
+ */
+class LogicalRelationTableExtractor extends TableExtractor {
+ override def apply(spark: SparkSession, v1: AnyRef): Option[Table] = {
+ val maybeCatalogTable = invoke(v1, "catalogTable").asInstanceOf[Option[AnyRef]]
+ maybeCatalogTable.flatMap { ct =>
+ new CatalogTableTableExtractor().apply(spark, ct)
+ }
+ }
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/tableTypeExtractors.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/tableTypeExtractors.scala
new file mode 100644
index 000000000..62ad46944
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/tableTypeExtractors.scala
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.serde
+
+import java.util.ServiceLoader
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.TableIdentifier
+
+import org.apache.kyuubi.plugin.spark.authz.serde.TableType.TableType
+
+object TableType extends Enumeration {
+ type TableType = Value
+ val TABLE, PERMANENT_VIEW, TEMP_VIEW, GLOBAL_TEMP_VIEW = Value
+}
+
+trait TableTypeExtractor extends ((AnyRef, SparkSession) => TableType) with Extractor
+
+object TableTypeExtractor {
+ val tableTypeExtractors: Map[String, TableTypeExtractor] = {
+ ServiceLoader.load(classOf[TableTypeExtractor])
+ .iterator()
+ .asScala
+ .map(e => (e.key, e))
+ .toMap
+ }
+}
+
+/**
+ * org.apache.spark.sql.catalyst.analysis.ViewType
+ */
+class ViewTypeTableTypeExtractor extends TableTypeExtractor {
+ override def apply(v1: AnyRef, spark: SparkSession): TableType = {
+ v1.toString match {
+ case "LocalTempView" => TableType.TEMP_VIEW
+ case "GlobalTempView" => TableType.GLOBAL_TEMP_VIEW
+ case _ => TableType.PERMANENT_VIEW
+ }
+ }
+}
+
+class TableIdentifierTableTypeExtractor extends TableTypeExtractor {
+ override def apply(v1: AnyRef, spark: SparkSession): TableType = {
+ val catalog = spark.sessionState.catalog
+ val identifier = v1.asInstanceOf[TableIdentifier]
+ val nameParts: Seq[String] = identifier.database.toSeq :+ identifier.table
+ if (catalog.isTempView(nameParts)) {
+ TableType.TEMP_VIEW
+ } else {
+ TableType.PERMANENT_VIEW
+ }
+ }
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala
deleted file mode 100644
index 05f3018a3..000000000
--- a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala
+++ /dev/null
@@ -1,397 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.kyuubi.plugin.spark.authz
-
-import scala.collection.mutable.ArrayBuffer
-
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.connector.catalog.Identifier
-import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
-
-import org.apache.kyuubi.plugin.spark.authz.OperationType._
-import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
-import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
-import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
-import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
-import org.apache.kyuubi.plugin.spark.authz.v2Commands.CommandType.{CommandType, HasChildAsIdentifier, HasQueryAsLogicalPlan, HasTableAsIdentifier, HasTableAsIdentifierOption, HasTableNameAsIdentifier}
-
-/**
- * Building privilege objects
- * for Spark Datasource V2 commands
- */
-object v2Commands extends Enumeration {
-
- /**
- * Command type enum
- * with naming rule as `HasFieldAsReturnType`
- * for hinting privileges building of inputObjs or outputObjs
- */
- object CommandType extends Enumeration {
- type CommandType = Value
- val HasChildAsIdentifier, HasQueryAsLogicalPlan, HasTableAsIdentifier,
- HasTableAsIdentifierOption, HasTableNameAsIdentifier = Value
- }
-
- import scala.language.implicitConversions
-
- implicit def valueToCmdPrivilegeBuilder(x: Value): CmdPrivilegeBuilder =
- x.asInstanceOf[CmdPrivilegeBuilder]
-
- /**
- * check whether commandName is implemented with supported privilege builders
- * and pass the requirement checks (e.g. Spark version)
- *
- * @param commandName name of command
- * @return true if so, false else
- */
- def accept(commandName: String): Boolean = {
- try {
- val command = v2Commands.withName(commandName)
-
- // check spark version requirements
- passSparkVersionCheck(command.mostVer, command.leastVer)
-
- } catch {
- case _: NoSuchElementException => false
- }
- }
-
- val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[CommandType]) => Unit =
- (plan, inputObjs, commandTypes) => {
- commandTypes.foreach {
- case HasQueryAsLogicalPlan =>
- val query = getFieldVal[LogicalPlan](plan, "query")
- buildQuery(query, inputObjs)
- case _ =>
- }
- }
-
- val defaultBuildOutput: (
- LogicalPlan,
- ArrayBuffer[PrivilegeObject],
- Seq[CommandType],
- PrivilegeObjectActionType,
- Boolean) => Unit =
- (plan, outputObjs, commandTypes, outputObjsActionType, resolveTableOwner) => {
- commandTypes.foreach {
- case HasTableNameAsIdentifier =>
- val table = invoke(plan, "tableName").asInstanceOf[Identifier]
- val owner = if (resolveTableOwner) getTableOwnerFromV2Plan(plan, table) else None
- outputObjs += v2TablePrivileges(table, owner = owner)
-
- case HasTableAsIdentifierOption =>
- val datasourceV2Relation = getFieldVal[LogicalPlan](plan, "table")
- .find(_.isInstanceOf[DataSourceV2Relation])
- val tableIdent = datasourceV2Relation.flatMap { r =>
- getFieldVal[Option[Identifier]](r, "identifier")
- }
- if (tableIdent.isDefined) {
- val owner =
- if (resolveTableOwner) getDatasourceV2TableOwner(datasourceV2Relation.get) else None
- outputObjs += v2TablePrivileges(
- tableIdent.get,
- owner = owner,
- actionType = outputObjsActionType)
- }
-
- case HasTableAsIdentifier =>
- val resolvedTable = getFieldVal[LogicalPlan](plan, "table")
- val tableIdent = getFieldVal[Identifier](resolvedTable, "identifier")
- val owner = if (resolveTableOwner) getDatasourceV2TableOwner(resolvedTable) else None
- outputObjs += v2TablePrivileges(tableIdent, owner = owner)
-
- case HasChildAsIdentifier =>
- val resolvedTable = getFieldVal[LogicalPlan](plan, "child")
- val tableIdent = getFieldVal[Identifier](resolvedTable, "identifier")
- val owner = if (resolveTableOwner) getDatasourceV2TableOwner(resolvedTable) else None
- outputObjs += v2TablePrivileges(tableIdent, owner = owner)
-
- case _ =>
- }
- }
-
- /**
- * Command privilege builder
- *
- * @param operationType OperationType for converting accessType
- * @param leastVer minimum Spark version required
- * @param mostVer maximum Spark version supported
- * @param commandTypes Seq of [[CommandType]] hinting privilege building
- * @param buildInput input [[PrivilegeObject]] for privilege check
- * @param buildOutput output [[PrivilegeObject]] for privilege check
- * @param outputActionType [[PrivilegeObjectActionType]] for output [[PrivilegeObject]]
- * @param resolveOutputTableOwner Whether to resolve table owner for output [[PrivilegeObject]]
- */
- case class CmdPrivilegeBuilder(
- operationType: OperationType = QUERY,
- leastVer: Option[String] = None,
- mostVer: Option[String] = None,
- commandTypes: Seq[CommandType] = Seq.empty,
- buildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[CommandType]) => Unit =
- defaultBuildInput,
- buildOutput: (
- LogicalPlan,
- ArrayBuffer[PrivilegeObject],
- Seq[CommandType],
- PrivilegeObjectActionType,
- Boolean) => Unit = defaultBuildOutput,
- outputActionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER,
- resolveOutputTableOwner: Boolean = true)
- extends super.Val {
-
- def buildPrivileges(
- plan: LogicalPlan,
- inputObjs: ArrayBuffer[PrivilegeObject],
- outputObjs: ArrayBuffer[PrivilegeObject]): Unit = {
- this.buildInput(plan, inputObjs, commandTypes)
- this.buildOutput(
- plan,
- outputObjs,
- commandTypes,
- outputActionType,
- resolveOutputTableOwner)
- }
- }
-
- def v2TablePrivileges(
- table: Identifier,
- columns: Seq[String] = Nil,
- owner: Option[String] = None,
- actionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER): PrivilegeObject = {
- PrivilegeObject(
- TABLE_OR_VIEW,
- actionType,
- quote(table.namespace()),
- table.name(),
- columns,
- owner)
- }
-
- // namespace commands
-
- val CreateNamespace: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = CREATEDATABASE,
- buildOutput = (plan, outputObjs, _, _, _) => {
- if (isSparkVersionAtLeast("3.3")) {
- val resolvedNamespace = getFieldVal[Any](plan, "name")
- val databases = getFieldVal[Seq[String]](resolvedNamespace, "nameParts")
- outputObjs += databasePrivileges(quote(databases))
- } else {
- val namespace = getFieldVal[Seq[String]](plan, "namespace")
- outputObjs += databasePrivileges(quote(namespace))
- }
- })
-
- val DropNamespace: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = DROPDATABASE,
- buildOutput = (plan, outputObjs, _, _, _) => {
- val resolvedNamespace = getFieldVal[LogicalPlan](plan, "namespace")
- val databases = getFieldVal[Seq[String]](resolvedNamespace, "namespace")
- outputObjs += databasePrivileges(quote(databases))
- })
-
- // with V2CreateTablePlan
-
- val CreateTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = CREATETABLE,
- commandTypes = Seq(HasTableNameAsIdentifier),
- leastVer = Some("3.3"),
- resolveOutputTableOwner = false)
-
- val CreateV2Table: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = CREATETABLE,
- commandTypes = Seq(HasTableNameAsIdentifier),
- mostVer = Some("3.2"),
- resolveOutputTableOwner = false)
-
- val CreateTableAsSelect: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = CREATETABLE,
- commandTypes = Seq(HasTableNameAsIdentifier, HasQueryAsLogicalPlan),
- resolveOutputTableOwner = false)
-
- val ReplaceTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = CREATETABLE,
- commandTypes = Seq(HasTableNameAsIdentifier),
- resolveOutputTableOwner = false)
-
- val ReplaceTableAsSelect: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = CREATETABLE,
- commandTypes = Seq(HasTableNameAsIdentifier, HasQueryAsLogicalPlan),
- resolveOutputTableOwner = false)
-
- // with V2WriteCommand
-
- val AppendData: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- commandTypes = Seq(HasTableAsIdentifierOption, HasQueryAsLogicalPlan),
- outputActionType = PrivilegeObjectActionType.INSERT)
-
- val UpdateTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- commandTypes = Seq(HasTableAsIdentifierOption),
- outputActionType = PrivilegeObjectActionType.UPDATE)
-
- val DeleteFromTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- commandTypes = Seq(HasTableAsIdentifierOption),
- outputActionType = PrivilegeObjectActionType.UPDATE)
-
- val OverwriteByExpression: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- commandTypes = Seq(HasTableAsIdentifierOption, HasQueryAsLogicalPlan),
- outputActionType = PrivilegeObjectActionType.UPDATE)
-
- val OverwritePartitionsDynamic: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- commandTypes = Seq(HasTableAsIdentifierOption, HasQueryAsLogicalPlan),
- outputActionType = PrivilegeObjectActionType.UPDATE)
-
- // with V2PartitionCommand
-
- val AddPartitions: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = OperationType.ALTERTABLE_ADDPARTS,
- leastVer = Some("3.2"),
- commandTypes = Seq(HasTableAsIdentifier))
-
- val DropPartitions: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = OperationType.ALTERTABLE_DROPPARTS,
- leastVer = Some("3.2"),
- commandTypes = Seq(HasTableAsIdentifier))
-
- val RenamePartitions: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = OperationType.ALTERTABLE_ADDPARTS,
- leastVer = Some("3.2"),
- commandTypes = Seq(HasTableAsIdentifier))
-
- val TruncatePartition: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = OperationType.ALTERTABLE_DROPPARTS,
- leastVer = Some("3.2"),
- commandTypes = Seq(HasTableAsIdentifier))
-
- // other table commands
-
- val CacheTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = CREATEVIEW,
- leastVer = Some("3.2"),
- buildInput = (plan, inputObjs, _) => {
- val query = getFieldVal[LogicalPlan](plan, "table") // table to cache
- buildQuery(query, inputObjs)
- })
-
- val CacheTableAsSelect: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = CREATEVIEW,
- leastVer = Some("3.2"),
- buildInput = (plan, inputObjs, _) => {
- val query = getFieldVal[LogicalPlan](plan, "plan")
- buildQuery(query, inputObjs)
- })
-
- val CommentOnNamespace: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = ALTERDATABASE,
- buildOutput = (plan, outputObjs, _, _, _) => {
- val resolvedNamespace = getFieldVal[AnyRef](plan, "child")
- val namespace = getFieldVal[Seq[String]](resolvedNamespace, "namespace")
- outputObjs += databasePrivileges(quote(namespace))
- })
-
- val CommentOnTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = ALTERTABLE_PROPERTIES,
- commandTypes = Seq(
- if (isSparkVersionAtLeast("3.2")) HasTableAsIdentifier else HasChildAsIdentifier))
-
- val DropTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = DROPTABLE,
- buildOutput = (plan, outputObjs, _, _, _) => {
- val (tableIdent, owner) =
- if (isSparkVersionAtLeast("3.1")) {
- val resolvedTable = getFieldVal[LogicalPlan](plan, "child")
- (
- getFieldVal[Identifier](resolvedTable, "identifier"),
- getDatasourceV2TableOwner(resolvedTable))
- } else {
- (getFieldVal[Identifier](plan, "ident"), None)
- }
- outputObjs += v2TablePrivileges(tableIdent, owner = owner)
- })
- val MergeIntoTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- buildInput = (plan, inputObjs, _) => {
- val table = getFieldVal[LogicalPlan](plan, "sourceTable")
- buildQuery(table, inputObjs)
- },
- buildOutput = (plan, outputObjs, _, _, _) => {
- val aliasOrRelation = getFieldVal[LogicalPlan](plan, "targetTable")
- aliasOrRelation.find(_.isInstanceOf[DataSourceV2Relation]).foreach {
- case table: DataSourceV2Relation if table.identifier.isDefined =>
- outputObjs += v2TablePrivileges(
- table.identifier.get,
- owner = getDatasourceV2TableOwner(table),
- actionType = PrivilegeObjectActionType.UPDATE)
- }
- })
-
- val RepairTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = ALTERTABLE_ADDPARTS,
- leastVer = Some("3.2"),
- commandTypes = Seq(HasChildAsIdentifier))
-
- val TruncateTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- leastVer = Some("3.2"),
- buildOutput = (plan, outputObjs, _, _, _) => {
- val resolvedTable = getFieldVal[LogicalPlan](plan, "table")
- val tableIdent = getFieldVal[Identifier](resolvedTable, "identifier")
- val owner = getDatasourceV2TableOwner(resolvedTable)
- outputObjs += v2TablePrivileges(
- tableIdent,
- owner = owner,
- actionType = PrivilegeObjectActionType.UPDATE)
- })
-
- // with V2AlterTableCommand
-
- val AlterTable: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = ALTERTABLE_ADDCOLS,
- mostVer = Some("3.1"),
- buildOutput = (plan, outputObjs, _, _, _) => {
- val table = getFieldVal[LogicalPlan](plan, "table")
- val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
- if (tableIdent.isDefined) {
- val owner = getDatasourceV2TableOwner(table)
- outputObjs += v2TablePrivileges(tableIdent.get, owner = owner)
- }
- })
-
- val AddColumns: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = ALTERTABLE_ADDCOLS,
- leastVer = Some("3.2"),
- commandTypes = Seq(HasTableAsIdentifier))
-
- val AlterColumn: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = ALTERTABLE_ADDCOLS,
- leastVer = Some("3.2"),
- commandTypes = Seq(HasTableAsIdentifier))
-
- val DropColumns: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = ALTERTABLE_ADDCOLS,
- leastVer = Some("3.2"),
- commandTypes = Seq(HasTableAsIdentifier))
-
- val ReplaceColumns: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = ALTERTABLE_REPLACECOLS,
- leastVer = Some("3.2"),
- commandTypes = Seq(HasTableAsIdentifier))
-
- val RenameColumn: CmdPrivilegeBuilder = CmdPrivilegeBuilder(
- operationType = ALTERTABLE_RENAMECOL,
- leastVer = Some("3.2"),
- commandTypes = Seq(HasTableAsIdentifier))
-}
diff --git a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/IcebergCatalogPrivilegesBuilderSuite.scala b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/IcebergCatalogPrivilegesBuilderSuite.scala
index a5ec2cfcc..db9f91099 100644
--- a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/IcebergCatalogPrivilegesBuilderSuite.scala
+++ b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/IcebergCatalogPrivilegesBuilderSuite.scala
@@ -58,10 +58,8 @@ class IcebergCatalogPrivilegesBuilderSuite extends V2CommandsPrivilegesSuite {
test("DeleteFromIcebergTable") {
val plan = sql(s"DELETE FROM $catalogTable WHERE key = 1 ").queryExecution.analyzed
assert(IcebergCommands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
-
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
assert(inputs.isEmpty)
assert(outputs.size === 1)
val po = outputs.head
@@ -78,10 +76,8 @@ class IcebergCatalogPrivilegesBuilderSuite extends V2CommandsPrivilegesSuite {
test("UpdateIcebergTable") {
val plan = sql(s"UPDATE $catalogTable SET value = 'b' WHERE key = 1 ").queryExecution.analyzed
assert(IcebergCommands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
-
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
assert(inputs.isEmpty)
assert(outputs.size === 1)
val po = outputs.head
@@ -104,11 +100,8 @@ class IcebergCatalogPrivilegesBuilderSuite extends V2CommandsPrivilegesSuite {
s"ON t.key = s.key " +
s"WHEN MATCHED THEN UPDATE SET t.value = s.value " +
s"WHEN NOT MATCHED THEN INSERT *").queryExecution.analyzed
- assert(IcebergCommands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
-
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
assert(inputs.size === 1)
val po0 = inputs.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
diff --git a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilderSuite.scala b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilderSuite.scala
index 3d4bc3db6..77f21ec30 100644
--- a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilderSuite.scala
+++ b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilderSuite.scala
@@ -54,7 +54,7 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
}
protected def checkColumns(plan: LogicalPlan, cols: Seq[String]): Unit = {
- val (in, out) = PrivilegesBuilder.build(plan, spark)
+ val (in, out, _) = PrivilegesBuilder.build(plan, spark)
assert(out.isEmpty, "Queries shall not check output privileges")
val po = in.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
@@ -110,19 +110,16 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
test("AlterDatabasePropertiesCommand") {
val plan = sql("ALTER DATABASE default SET DBPROPERTIES (abc = '123')").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === ALTERDATABASE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.isEmpty)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(in.isEmpty)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.DATABASE)
assert(po.dbname === "default")
assert(po.objectName === "default")
assert(po.columns.isEmpty)
- val accessType = AccessType(po, operationType, isInput = false)
- assert(accessType === AccessType.ALTER)
}
test("AlterTableRenameCommand") {
@@ -138,20 +135,18 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
s" RENAME TO $t").queryExecution.analyzed
// re-create oldTable because we needs to get table owner from table metadata later
sql(create)
-
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === ALTERTABLE_RENAME)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.isEmpty)
- assert(tuple._2.size === 2)
- tuple._2.foreach { po =>
+ assert(in.isEmpty)
+ assert(out.size === 2)
+ out.foreach { po =>
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
assert(Set(oldTableShort, "efg").contains(po.objectName))
assert(po.columns.isEmpty)
val accessType = ranger.AccessType(po, operationType, isInput = false)
assert(Set(AccessType.CREATE, AccessType.DROP).contains(accessType))
- if (accessType === AccessType.DROP) {
+ if (accessType == AccessType.DROP) {
checkTableOwner(po)
}
}
@@ -162,12 +157,11 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
test("CreateDatabaseCommand") {
withDatabase("CreateDatabaseCommand") { db =>
val plan = sql(s"CREATE DATABASE $db").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === CREATEDATABASE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.isEmpty)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(in.isEmpty)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.DATABASE)
assert(po.dbname === "CreateDatabaseCommand")
@@ -182,12 +176,11 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
withDatabase("DropDatabaseCommand") { db =>
sql(s"CREATE DATABASE $db")
val plan = sql(s"DROP DATABASE DropDatabaseCommand").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === DROPDATABASE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.isEmpty)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(in.isEmpty)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.DATABASE)
assert(po.dbname === "DropDatabaseCommand")
@@ -201,12 +194,10 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
test("AlterTableAddPartitionCommand") {
val plan = sql(s"ALTER TABLE $reusedPartTable ADD IF NOT EXISTS PARTITION (pid=1)")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
- assert(operationType === ALTERTABLE_ADDPARTS)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.isEmpty)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(in.isEmpty)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -220,12 +211,10 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
test("AlterTableDropPartitionCommand") {
val plan = sql(s"ALTER TABLE $reusedPartTable DROP IF EXISTS PARTITION (pid=1)")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
- assert(operationType === ALTERTABLE_DROPPARTS)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.isEmpty)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(in.isEmpty)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -252,10 +241,8 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
|MSCK REPAIR TABLE $tableName
|""".stripMargin
val plan = sql(sqlStr).queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === MSCK)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
-
assert(inputs.isEmpty)
assert(outputs.size === 1)
@@ -278,12 +265,10 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val plan = sql(s"ALTER TABLE $reusedPartTable PARTITION (pid=1) " +
s"RENAME TO PARTITION (PID=10)")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
- assert(operationType === ALTERTABLE_RENAMEPART)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.isEmpty)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(in.isEmpty)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -300,12 +285,12 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val plan = sql(s"ALTER TABLE $reusedPartTable PARTITION (pid=1)" +
s" SET LOCATION '$newLoc'")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
- assert(operationType === ALTERTABLE_LOCATION)
val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.isEmpty)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
+
+ assert(in.isEmpty)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname === reusedDb)
@@ -322,12 +307,11 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
"UNSET TBLPROPERTIES (key)").foreach { clause =>
val plan = sql(s"ALTER TABLE $reusedTable $clause")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
- assert(operationType === ALTERTABLE_PROPERTIES)
val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.isEmpty)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(in.isEmpty)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname === reusedDb)
@@ -343,11 +327,10 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
sql(s"CREATE VIEW AlterViewAsCommand AS SELECT * FROM $reusedTable")
val plan = sql(s"ALTER VIEW AlterViewAsCommand AS SELECT * FROM $reusedPartTable")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
- assert(operationType === ALTERVIEW_AS)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
+
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -360,8 +343,8 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname === (if (isSparkV2) null else "default"))
@@ -375,11 +358,10 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
test("AnalyzeColumnCommand") {
val plan = sql(s"ANALYZE TABLE $reusedPartTable PARTITION (pid=1)" +
s" COMPUTE STATISTICS FOR COLUMNS key").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === ANALYZE_TABLE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -390,17 +372,17 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("AnalyzePartitionCommand") {
val plan = sql(s"ANALYZE TABLE $reusedPartTable" +
s" PARTITION (pid = 1) COMPUTE STATISTICS").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === ANALYZE_TABLE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -411,17 +393,17 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("AnalyzeTableCommand") {
val plan = sql(s"ANALYZE TABLE $reusedPartTable COMPUTE STATISTICS")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
+
assert(operationType === ANALYZE_TABLE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -432,18 +414,17 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("AnalyzeTablesCommand") {
assume(isSparkV32OrGreater)
val plan = sql(s"ANALYZE TABLES IN $reusedDb COMPUTE STATISTICS")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === ANALYZE_TABLE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.DATABASE)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -453,16 +434,15 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("RefreshTableCommand / RefreshTable") {
val plan = sql(s"REFRESH TABLE $reusedTable").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -472,40 +452,16 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 0)
- }
-
- test("CacheTable") {
- val plan = sql(s"CACHE LAZY TABLE $reusedTable").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
- assert(operationType === CREATEVIEW)
- val tuple = PrivilegesBuilder.build(plan, spark)
- if (isSparkV32OrGreater) {
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
- assert(po0.actionType === PrivilegeObjectActionType.OTHER)
- assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
- assert(po0.dbname equalsIgnoreCase reusedDb)
- assert(po0.objectName equalsIgnoreCase reusedDb)
- assert(po0.columns.head === "key")
- checkTableOwner(po0)
- val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
- assert(accessType0 === AccessType.SELECT)
- } else {
- assert(tuple._1.isEmpty)
- }
-
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("CacheTableAsSelect") {
val plan = sql(s"CACHE TABLE CacheTableAsSelect AS SELECT * FROM $reusedTable")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === CREATEVIEW)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -519,17 +475,16 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("CreateViewCommand") {
val plan = sql(s"CREATE VIEW CreateViewCommand(a, b) AS SELECT key, value FROM $reusedTable")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === CREATEVIEW)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -543,8 +498,8 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname === (if (isSparkV2) null else "default"))
@@ -559,12 +514,11 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
withTable(tableName) { _ =>
val plan = sql(s"CREATE TABLE $tableName(a int, b string) USING parquet")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === CREATETABLE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 0)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(in.size === 0)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname === (if (isSparkV2) null else "default"))
@@ -579,42 +533,38 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val plan = sql("CREATE TEMPORARY FUNCTION CreateTempFunction AS" +
"'org.apache.hadoop.hive.ql.udf.generic.GenericUDFMaskHash'")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === CREATEFUNCTION)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 0)
- assert(tuple._2.size === 0)
+ assert(in.size === 0)
+ assert(out.size === 0)
}
test("Describe Temporary Function") {
val plan = sql("DESCRIBE FUNCTION CreateTempFunction")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === DESCFUNCTION)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 0)
- assert(tuple._2.size === 0)
+ assert(in.size === 0)
+ assert(out.size === 0)
}
test("Drop Temporary Function") {
val plan = sql("DROP TEMPORARY FUNCTION CreateTempFunction")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === DROPFUNCTION)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 0)
- assert(tuple._2.size === 0)
+ assert(in.size === 0)
+ assert(out.size === 0)
}
test("CreateFunctionCommand") {
val plan = sql("CREATE FUNCTION CreateFunctionCommand AS 'class_name'")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === CREATEFUNCTION)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 0)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(in.size === 0)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.FUNCTION)
val db = if (isSparkV33OrGreater) "default" else null
@@ -626,25 +576,26 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
}
test("Describe Persistent Function") {
- val plan = sql("DESCRIBE FUNCTION CreateFunctionCommand")
+ sql("DROP FUNCTION IF EXISTS default.DescribePersistentFunction")
+ sql("CREATE FUNCTION default.DescribePersistentFunction AS 'class_name'")
+ val plan = sql("DESCRIBE FUNCTION DescribePersistentFunction")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === DESCFUNCTION)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- assert(tuple._2.size === 0)
+ assert(in.size === 1)
+ assert(out.size === 0)
}
test("DropFunctionCommand") {
- sql("CREATE FUNCTION DropFunctionCommand AS 'class_name'")
+ sql("DROP FUNCTION IF EXISTS default.DropFunctionCommand")
+ sql("CREATE FUNCTION default.DropFunctionCommand AS 'class_name'")
val plan = sql("DROP FUNCTION DropFunctionCommand")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === DROPFUNCTION)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 0)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(in.size === 0)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.FUNCTION)
val db = if (isSparkV33OrGreater) "default" else null
@@ -660,12 +611,11 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
sql(s"CREATE FUNCTION RefreshFunctionCommand AS '${getClass.getCanonicalName}'")
val plan = sql("REFRESH FUNCTION RefreshFunctionCommand")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === RELOADFUNCTION)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 0)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(in.size === 0)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.FUNCTION)
val db = if (isSparkV33OrGreater) "default" else null
@@ -679,12 +629,10 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
test("CreateTableLikeCommand") {
withTable(reusedDb + ".CreateTableLikeCommand") { t =>
val plan = sql(s"CREATE TABLE $t LIKE $reusedTable").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
-
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === CREATETABLE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -694,8 +642,8 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -711,12 +659,10 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
withTable("CreateTableLikeCommandWithoutDatabase") { t =>
sql(s"USE ${reusedDb}")
val plan = sql(s"CREATE TABLE $t LIKE ${reusedTableShort}").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
-
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === CREATETABLE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -726,8 +672,8 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -741,21 +687,18 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
test("CreateTempViewUsing") {
val plan = sql("CREATE TEMPORARY VIEW CreateTempViewUsing (a int, b string) USING parquet")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
-
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === CREATEVIEW)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 0)
- assert(tuple._2.size === 0)
+ assert(in.size === 0)
+ assert(out.size === 0)
}
test("DescribeColumnCommand") {
val plan = sql(s"DESC TABLE $reusedTable key").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === DESCTABLE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po = tuple._1.head
+ assert(in.size === 1)
+ val po = in.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -765,16 +708,15 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType = ranger.AccessType(po, operationType, isInput = false)
assert(accessType === AccessType.SELECT)
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("DescribeTableCommand") {
val plan = sql(s"DESC TABLE $reusedTable").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === DESCTABLE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po = tuple._1.head
+ assert(in.size === 1)
+ val po = in.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -784,16 +726,15 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType = ranger.AccessType(po, operationType, isInput = false)
assert(accessType === AccessType.SELECT)
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("DescribeDatabaseCommand") {
val plan = sql(s"DESC DATABASE $reusedDb").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === DESCDATABASE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po = tuple._1.head
+ assert(in.size === 1)
+ val po = in.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.DATABASE)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -802,18 +743,17 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType = ranger.AccessType(po, operationType, isInput = false)
assert(accessType === AccessType.USE)
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("SetDatabaseCommand") {
try {
val plan = sql(s"USE $reusedDb").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === SWITCHDATABASE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
+ assert(in.size === 1)
- val po0 = tuple._1.head
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.DATABASE)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -822,7 +762,7 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = false)
assert(accessType0 === AccessType.USE)
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
} finally {
sql("USE default")
}
@@ -831,12 +771,11 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
test("TruncateTableCommand") {
val plan = sql(s"TRUNCATE TABLE $reusedPartTable PARTITION (pid=1)")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === TRUNCATETABLE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.isEmpty)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(in.isEmpty)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -849,11 +788,10 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
test("ShowColumnsCommand") {
val plan = sql(s"SHOW COLUMNS IN $reusedTable").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === SHOWCOLUMNS)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -863,16 +801,15 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("ShowCreateTableCommand") {
val plan = sql(s"SHOW CREATE TABLE $reusedTable").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === SHOW_CREATETABLE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -882,16 +819,15 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("ShowTablePropertiesCommand") {
val plan = sql(s"SHOW TBLPROPERTIES $reusedTable ").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === SHOW_TBLPROPERTIES)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -901,26 +837,16 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 0)
- }
-
- test("ShowFunctionsCommand") {
- val plan = sql("SHOW FUNCTIONS").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
- assert(operationType === SHOWFUNCTIONS)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 0)
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("ShowPartitionsCommand") {
val plan = sql(s"SHOW PARTITIONS $reusedPartTable PARTITION (pid=1)")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === SHOWPARTITIONS)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -930,7 +856,7 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("RepairTableCommand") {
@@ -949,22 +875,21 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
|MSCK REPAIR TABLE $tableName
|""".stripMargin
val plan = sql(sqlStr).queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === MSCK)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
assert(inputs.isEmpty)
assert(outputs.size === 1)
outputs.foreach { po =>
- assert(po.actionType === PrivilegeObjectActionType.INSERT)
+ assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
assert(po.objectName equalsIgnoreCase tableName.split("\\.").last)
assert(po.columns.isEmpty)
checkTableOwner(po)
val accessType = ranger.AccessType(po, operationType, isInput = false)
- assert(accessType === AccessType.UPDATE)
+ assert(accessType === AccessType.ALTER)
}
}
}
@@ -1026,7 +951,7 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
| FROM $reusedPartTable WHERE pid > 1)
|""".stripMargin).queryExecution.optimizedPlan
- val (in, _) = PrivilegesBuilder.build(plan, spark)
+ val (in, _, _) = PrivilegesBuilder.build(plan, spark)
assert(in.size === 2)
assert(in.find(_.objectName equalsIgnoreCase reusedTableShort).head.columns ===
@@ -1043,11 +968,10 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
|RANK() OVER (PARTITION BY key ORDER BY pid) AS rank
|FROM $reusedPartTable)
|""".stripMargin).queryExecution.optimizedPlan
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- tuple._1.foreach { po =>
+ assert(in.size === 1)
+ in.foreach { po =>
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -1059,7 +983,7 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType = ranger.AccessType(po, operationType, isInput = true)
assert(accessType === AccessType.SELECT)
}
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("Query: Subquery With Order") {
@@ -1070,11 +994,10 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
|FROM $reusedPartTable
|ORDER BY key, pid)
|""".stripMargin).queryExecution.optimizedPlan
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- tuple._1.foreach { po =>
+ assert(in.size === 1)
+ in.foreach { po =>
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -1086,7 +1009,7 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType = ranger.AccessType(po, operationType, isInput = true)
assert(accessType === AccessType.SELECT)
}
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("Query: INNER JOIN") {
@@ -1099,12 +1022,11 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
| """.stripMargin
val plan = sql(sqlStr).queryExecution.optimizedPlan
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 2)
- tuple._1.foreach { po =>
+ assert(in.size === 2)
+ in.foreach { po =>
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -1116,7 +1038,7 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType = ranger.AccessType(po, operationType, isInput = true)
assert(accessType === AccessType.SELECT)
}
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("Query: WHERE Without Project") {
@@ -1128,12 +1050,11 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
| """.stripMargin
val plan = sql(sqlStr).queryExecution.optimizedPlan
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- tuple._1.foreach { po =>
+ assert(in.size === 1)
+ in.foreach { po =>
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -1145,7 +1066,7 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType = ranger.AccessType(po, operationType, isInput = true)
assert(accessType === AccessType.SELECT)
}
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("Query: JOIN Without Project") {
@@ -1158,12 +1079,11 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
| """.stripMargin
val plan = sql(sqlStr).queryExecution.optimizedPlan
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 2)
- tuple._1.foreach { po =>
+ assert(in.size === 2)
+ in.foreach { po =>
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -1175,7 +1095,7 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType = ranger.AccessType(po, operationType, isInput = true)
assert(accessType === AccessType.SELECT)
}
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("Query: Window Without Project") {
@@ -1185,11 +1105,10 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
|RANK() OVER (PARTITION BY key ORDER BY value) AS rank
|FROM $reusedTable
|""".stripMargin).queryExecution.optimizedPlan
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- tuple._1.foreach { po =>
+ assert(in.size === 1)
+ in.foreach { po =>
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -1201,7 +1120,7 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType = ranger.AccessType(po, operationType, isInput = true)
assert(accessType === AccessType.SELECT)
}
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("Query: Order By Without Project") {
@@ -1211,11 +1130,10 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
|FROM $reusedPartTable
|ORDER BY key, value, pid
|""".stripMargin).queryExecution.optimizedPlan
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- tuple._1.foreach { po =>
+ assert(in.size === 1)
+ in.foreach { po =>
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -1227,7 +1145,7 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType = ranger.AccessType(po, operationType, isInput = true)
assert(accessType === AccessType.SELECT)
}
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
test("Query: Union") {
@@ -1239,7 +1157,7 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
|SELECT pid, value
|FROM $reusedPartTable
|""".stripMargin).queryExecution.optimizedPlan
- val (in, _) = PrivilegesBuilder.build(plan, spark)
+ val (in, _, _) = PrivilegesBuilder.build(plan, spark)
assert(in.size === 2)
}
@@ -1254,12 +1172,11 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
test("AlterTableAddColumnsCommand") {
val plan = sql(s"ALTER TABLE $reusedTable" +
s" ADD COLUMNS (a int)").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === ALTERTABLE_ADDCOLS)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.isEmpty)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(in.isEmpty)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -1270,16 +1187,15 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
assert(accessType === AccessType.ALTER)
}
- test("AlterTableChangeColumnsCommand") {
+ test("AlterTableChangeColumnCommand") {
assume(!isSparkV2)
val plan = sql(s"ALTER TABLE $reusedTable" +
s" ALTER COLUMN value COMMENT 'alter column'").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === ALTERTABLE_REPLACECOLS)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.isEmpty)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(in.isEmpty)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase reusedDb)
@@ -1300,12 +1216,11 @@ class InMemoryPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
val newLoc = spark.conf.get("spark.sql.warehouse.dir") + "/new_db_location"
val plan = sql(s"ALTER DATABASE default SET LOCATION '$newLoc'")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === ALTERDATABASE_LOCATION)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.isEmpty)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(in.isEmpty)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.DATABASE)
assert(po.dbname === "default")
@@ -1319,12 +1234,10 @@ class InMemoryPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
val plan = sql(s"CREATE TABLE CreateDataSourceTableAsSelectCommand USING parquet" +
s" AS SELECT key, value FROM $reusedTable")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
-
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === CREATETABLE_AS_SELECT)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -1334,8 +1247,8 @@ class InMemoryPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname === (if (isSparkV2) null else "default"))
@@ -1362,12 +1275,11 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
val plan = sql(s"ALTER TABLE $t PARTITION (pid=1)" +
s" SET SERDEPROPERTIES ( key1 = 'some key')")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === ALTERTABLE_SERDEPROPERTIES)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.isEmpty)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(in.isEmpty)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname === "default")
@@ -1384,12 +1296,11 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
withTable("CreateTableCommand") { _ =>
val plan = sql(s"CREATE TABLE CreateTableCommand(a int, b string) USING hive")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === CREATETABLE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 0)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(in.size === 0)
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname === "default")
@@ -1405,12 +1316,11 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
val plan = sql(s"CREATE TABLE CreateHiveTableAsSelectCommand USING hive" +
s" AS SELECT key, value FROM $reusedTable")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === CREATETABLE_AS_SELECT)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -1420,8 +1330,8 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname === "default")
@@ -1433,7 +1343,7 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
test("LoadDataCommand") {
assume(!isSparkV2)
- val dataPath = getClass.getResource("/data.txt").getPath
+ val dataPath = getClass.getClassLoader.getResource("data.txt").getPath
val tableName = reusedDb + "." + "LoadDataToTable"
withTable(tableName) { _ =>
sql(
@@ -1444,13 +1354,12 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
|""".stripMargin)
val plan = sql(s"LOAD DATA INPATH '$dataPath' OVERWRITE INTO TABLE $tableName")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === LOAD)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.isEmpty)
+ assert(in.isEmpty)
- assert(tuple._2.size === 1)
- val po0 = tuple._2.head
+ assert(out.size === 1)
+ val po0 = out.head
assert(po0.actionType === PrivilegeObjectActionType.INSERT_OVERWRITE)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -1472,11 +1381,10 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
|USING parquet
|SELECT * FROM $reusedPartTable""".stripMargin)
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -1486,7 +1394,7 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.isEmpty)
+ assert(out.isEmpty)
}
test("InsertIntoDataSourceCommand") {
@@ -1517,10 +1425,8 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
|SELECT key, value FROM $reusedTable
|""".stripMargin
val plan = sql(sqlStr).queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
-
assert(inputs.size == 1)
inputs.foreach { po =>
assert(po.actionType === PrivilegeObjectActionType.OTHER)
@@ -1559,9 +1465,8 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
|SELECT key, value FROM $reusedTable
|""".stripMargin
val plan = sql(sqlStr).queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
assert(inputs.size == 1)
inputs.foreach { po =>
@@ -1589,11 +1494,10 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
|USING parquet
|SELECT * FROM $reusedPartTable""".stripMargin)
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname equalsIgnoreCase reusedDb)
@@ -1603,7 +1507,7 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.isEmpty)
+ assert(out.isEmpty)
}
test("InsertIntoHiveTableCommand") {
@@ -1616,10 +1520,9 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
|INSERT INTO $tableName VALUES (1, "KYUUBI")
|""".stripMargin
val plan = sql(sqlStr).queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
- assert(operationType === QUERY)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(operationType === QUERY)
assert(inputs.isEmpty)
assert(outputs.size === 1)
@@ -1628,7 +1531,7 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname equalsIgnoreCase "default")
assert(po.objectName equalsIgnoreCase tableName)
- assert(po.columns.isEmpty)
+ assert(po.columns === Seq("a", "b"))
checkTableOwner(po)
val accessType = ranger.AccessType(po, operationType, isInput = false)
assert(accessType === AccessType.UPDATE)
@@ -1641,11 +1544,10 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
withTable("ShowCreateTableAsSerdeCommand") { t =>
sql(s"CREATE TABLE $t (key int, pid int) USING hive PARTITIONED BY (pid)")
val plan = sql(s"SHOW CREATE TABLE $t AS SERDE").queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === SHOW_CREATETABLE)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 1)
- val po0 = tuple._1.head
+ assert(in.size === 1)
+ val po0 = in.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
assert(po0.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po0.dbname === "default")
@@ -1655,7 +1557,7 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(tuple._2.size === 0)
+ assert(out.size === 0)
}
}
@@ -1664,14 +1566,13 @@ class HiveCatalogPrivilegeBuilderSuite extends PrivilegesBuilderSuite {
val plan = sql(
s"CREATE TABLE OptimizedCreateHiveTableAsSelectCommand STORED AS parquet AS SELECT 1 as a")
.queryExecution.analyzed
- val operationType = OperationType(plan.nodeName)
+ val (in, out, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === CREATETABLE_AS_SELECT)
- val tuple = PrivilegesBuilder.build(plan, spark)
- assert(tuple._1.size === 0)
+ assert(in.size === 0)
- assert(tuple._2.size === 1)
- val po = tuple._2.head
+ assert(out.size === 1)
+ val po = out.head
assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname === "default")
diff --git a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/V2CommandsPrivilegesSuite.scala b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/V2CommandsPrivilegesSuite.scala
index 18cbecda8..715fc10e2 100644
--- a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/V2CommandsPrivilegesSuite.scala
+++ b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/V2CommandsPrivilegesSuite.scala
@@ -88,11 +88,8 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
withV2Table(table) { tableId =>
val plan = executePlan(
s"CREATE TABLE IF NOT EXISTS $tableId (i int)").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === CREATETABLE)
-
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
assert(inputs.isEmpty)
assert(outputs.size === 1)
val po = outputs.head
@@ -113,11 +110,8 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
val plan = executePlan(
s"CREATE TABLE IF NOT EXISTS $tableId AS " +
s"SELECT * FROM $reusedTable").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
- assert(operationType === CREATETABLE)
-
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(operationType === CREATETABLE_AS_SELECT)
assert(inputs.size === 1)
val po0 = inputs.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
@@ -145,11 +139,9 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
withV2Table(table) { tableId =>
sql(s"CREATE TABLE IF NOT EXISTS $tableId (i int)")
val plan = executePlan(s"REPLACE TABLE $tableId (j int)").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
- assert(operationType === CREATETABLE)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(operationType === CREATETABLE)
assert(inputs.size === 0)
assert(outputs.size === 1)
val po = outputs.head
@@ -170,11 +162,8 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
sql(s"CREATE TABLE IF NOT EXISTS $tableId (i int)")
val plan =
executePlan(s"REPLACE TABLE $tableId AS SELECT * FROM $reusedTable").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
- assert(operationType === CREATETABLE)
-
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(operationType === CREATETABLE_AS_SELECT)
assert(inputs.size === 1)
val po0 = inputs.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
@@ -201,11 +190,8 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
test("AppendData") {
val plan = executePlan(s"INSERT INTO $catalogTable VALUES (0, 'a')").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
-
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
assert(inputs.size === 0)
assert(outputs.size === 1)
val po = outputs.head
@@ -224,11 +210,9 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
assume(supportsUpdateTable)
val plan = executePlan(s"UPDATE $catalogTable SET value = 'a' WHERE key = 0").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
- assert(operationType === QUERY)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(operationType === QUERY)
assert(inputs.size === 0)
assert(outputs.size === 1)
val po = outputs.head
@@ -246,11 +230,8 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
assume(supportsDelete)
val plan = executePlan(s"DELETE FROM $catalogTable WHERE key = 0").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
-
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
assert(inputs.size === 0)
assert(outputs.size === 1)
val po = outputs.head
@@ -266,15 +247,12 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
test("OverwriteByExpression") {
val plan = executePlan(s"INSERT OVERWRITE TABLE $catalogTable VALUES (0, 1)").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
-
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
assert(inputs.size === 0)
assert(outputs.size === 1)
val po = outputs.head
- assert(po.actionType === PrivilegeObjectActionType.UPDATE)
+ assert(po.actionType === PrivilegeObjectActionType.INSERT_OVERWRITE)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname === namespace)
assert(po.objectName === catalogTableShort)
@@ -291,15 +269,12 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
sql("SET spark.sql.sources.partitionOverwriteMode=dynamic")
val plan = executePlan(s"INSERT OVERWRITE TABLE $catalogPartTable PARTITION (dt)" +
s"VALUES (0, 1, '2022-01-01')").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
-
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
assert(inputs.size === 0)
assert(outputs.size === 1)
val po = outputs.head
- assert(po.actionType === PrivilegeObjectActionType.UPDATE)
+ assert(po.actionType === PrivilegeObjectActionType.INSERT_OVERWRITE)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname === namespace)
assert(po.objectName === catalogPartTableShort)
@@ -318,11 +293,8 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
val plan = executePlan(s"ALTER TABLE $catalogPartTable " +
s"ADD PARTITION (dt='2022-01-01')").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === ALTERTABLE_ADDPARTS)
-
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
assert(inputs.size === 0)
assert(outputs.size === 1)
val po = outputs.head
@@ -342,11 +314,8 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
val plan = executePlan(s"ALTER TABLE $catalogPartTable " +
s"DROP PARTITION (dt='2022-01-01')").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === ALTERTABLE_DROPPARTS)
-
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
assert(inputs.size === 0)
assert(outputs.size === 1)
val po = outputs.head
@@ -366,11 +335,8 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
val plan = executePlan(s"ALTER TABLE $catalogPartTable " +
s"PARTITION (dt='2022-01-01') RENAME TO PARTITION (dt='2022-01-02')").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === ALTERTABLE_ADDPARTS)
-
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
assert(inputs.size === 0)
assert(outputs.size === 1)
val po = outputs.head
@@ -390,11 +356,9 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
val plan = executePlan(s"ALTER TABLE $catalogPartTable " +
s"PARTITION (dt='2022-01-01') RENAME TO PARTITION (dt='2022-01-02')").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
- assert(operationType === ALTERTABLE_DROPPARTS)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(operationType === ALTERTABLE_DROPPARTS)
assert(inputs.size === 0)
assert(outputs.size === 1)
val po = outputs.head
@@ -412,11 +376,9 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
test("CommentOnTable") {
val plan = executePlan(s"COMMENT ON TABLE $catalogTable IS 'text'").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
- assert(operationType === ALTERTABLE_PROPERTIES)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(operationType === ALTERTABLE_PROPERTIES)
assert(inputs.isEmpty)
assert(outputs.size === 1)
val po = outputs.head
@@ -435,11 +397,9 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
withV2Table(table) { tableId =>
sql(s"CREATE TABLE $tableId (i int)")
val plan = executePlan(s"DROP TABLE $tableId").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
- assert(operationType === DROPTABLE)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(operationType === DROPTABLE)
assert(inputs.isEmpty)
assert(outputs.size === 1)
val po = outputs.head
@@ -465,11 +425,8 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
s"ON t.key = s.key " +
s"WHEN MATCHED THEN UPDATE SET t.value = s.value " +
s"WHEN NOT MATCHED THEN INSERT *").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
assert(operationType === QUERY)
-
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
assert(inputs.size == 1)
val po0 = inputs.head
assert(po0.actionType === PrivilegeObjectActionType.OTHER)
@@ -497,11 +454,9 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
assume(isSparkV32OrGreater)
val plan = executePlan(s"MSCK REPAIR TABLE $catalogPartTable").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
- assert(operationType === ALTERTABLE_ADDPARTS)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(operationType === MSCK)
assert(inputs.isEmpty)
assert(outputs.size === 1)
val po = outputs.head
@@ -519,15 +474,13 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
assume(isSparkV32OrGreater)
val plan = executePlan(s"TRUNCATE TABLE $catalogTable").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
- assert(operationType === QUERY)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(operationType === TRUNCATETABLE)
assert(inputs.isEmpty)
assert(outputs.size === 1)
val po = outputs.head
- assert(po.actionType === PrivilegeObjectActionType.UPDATE)
+ assert(po.actionType === PrivilegeObjectActionType.OTHER)
assert(po.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
assert(po.dbname === namespace)
assert(po.objectName === catalogTableShort)
@@ -546,11 +499,9 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
withV2Table(table) { tableId =>
sql(s"CREATE TABLE $tableId (i int)")
val plan = executePlan(s"ALTER TABLE $tableId ADD COLUMNS (j int)").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
- assert(operationType === ALTERTABLE_ADDCOLS)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(operationType === ALTERTABLE_ADDCOLS)
assert(inputs.isEmpty)
assert(outputs.size === 1)
val po = outputs.head
@@ -572,11 +523,9 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
withV2Table(table) { tableId =>
sql(s"CREATE TABLE $tableId (i int)")
val plan = executePlan(s"ALTER TABLE $tableId ALTER COLUMN i TYPE int").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
- assert(operationType === ALTERTABLE_ADDCOLS)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(operationType === ALTERTABLE_ADDCOLS)
assert(inputs.isEmpty)
assert(outputs.size === 1)
val po = outputs.head
@@ -598,11 +547,9 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
withV2Table(table) { tableId =>
sql(s"CREATE TABLE $tableId (i int, j int)")
val plan = executePlan(s"ALTER TABLE $tableId DROP COLUMN i").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
- assert(operationType === ALTERTABLE_ADDCOLS)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(operationType === ALTERTABLE_ADDCOLS)
assert(inputs.isEmpty)
assert(outputs.size === 1)
val po = outputs.head
@@ -624,11 +571,9 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
withV2Table(table) { tableId =>
sql(s"CREATE TABLE $tableId (i int, j int)")
val plan = executePlan(s"ALTER TABLE $tableId REPLACE COLUMNS (i String)").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
- assert(operationType === ALTERTABLE_REPLACECOLS)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(operationType === ALTERTABLE_REPLACECOLS)
assert(inputs.isEmpty)
assert(outputs.size === 1)
val po = outputs.head
@@ -650,11 +595,9 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
withV2Table(table) { tableId =>
sql(s"CREATE TABLE $tableId (i int, j int)")
val plan = executePlan(s"ALTER TABLE $tableId RENAME COLUMN i TO k").analyzed
- assert(v2Commands.accept(plan.nodeName))
- val operationType = OperationType(plan.nodeName)
- assert(operationType === ALTERTABLE_RENAMECOL)
- val (inputs, outputs) = PrivilegesBuilder.build(plan, spark)
+ val (inputs, outputs, operationType) = PrivilegesBuilder.build(plan, spark)
+ assert(operationType === ALTERTABLE_RENAMECOL)
assert(inputs.isEmpty)
assert(outputs.size === 1)
val po = outputs.head
diff --git a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/DatabaseCommands.scala b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/DatabaseCommands.scala
new file mode 100644
index 000000000..d6a72dbbb
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/DatabaseCommands.scala
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.gen
+
+import org.apache.kyuubi.plugin.spark.authz.serde.{DatabaseCommandSpec, DatabaseDesc}
+
+object DatabaseCommands {
+
+ val AlterDatabaseProperties = {
+ DatabaseCommandSpec(
+ "org.apache.spark.sql.execution.command.AlterDatabasePropertiesCommand",
+ Seq(DatabaseDesc("databaseName", "StringDatabaseExtractor")),
+ "ALTERDATABASE")
+ }
+
+ val CommentOnNamespace = {
+ DatabaseCommandSpec(
+ "org.apache.spark.sql.catalyst.plans.logical.CommentOnNamespace",
+ Seq(DatabaseDesc("child", "ResolvedNamespaceDatabaseExtractor")),
+ "ALTERDATABASE")
+ }
+
+ val SetNamespaceProperties = {
+ DatabaseCommandSpec(
+ "org.apache.spark.sql.catalyst.plans.logical.SetNamespaceProperties",
+ Seq(DatabaseDesc("namespace", "ResolvedNamespaceDatabaseExtractor")),
+ "ALTERDATABASE")
+ }
+
+ val SetNamespaceLocation = {
+ DatabaseCommandSpec(
+ "org.apache.spark.sql.catalyst.plans.logical.SetNamespaceLocation",
+ Seq(DatabaseDesc("namespace", "ResolvedNamespaceDatabaseExtractor")),
+ "ALTERDATABASE_LOCATION")
+ }
+
+ val CreateNamespace = {
+ val databaseDesc1 = DatabaseDesc("name", "ResolvedDBObjectNameDatabaseExtractor")
+ val databaseDesc2 = DatabaseDesc("namespace", "StringSeqDatabaseExtractor")
+ DatabaseCommandSpec(
+ "org.apache.spark.sql.catalyst.plans.logical.CreateNamespace",
+ Seq(databaseDesc1, databaseDesc2),
+ "CREATEDATABASE")
+ }
+
+ val DropNamespace = {
+ DatabaseCommandSpec(
+ "org.apache.spark.sql.catalyst.plans.logical.DropNamespace",
+ Seq(DatabaseDesc("namespace", "ResolvedNamespaceDatabaseExtractor")),
+ "DROPDATABASE")
+ }
+
+ val AnalyzeTables = {
+ DatabaseCommandSpec(
+ "org.apache.spark.sql.execution.command.AnalyzeTablesCommand",
+ Seq(DatabaseDesc("databaseName", "StringOptionDatabaseExtractor", true)),
+ "ANALYZE_TABLE")
+ }
+
+ val SetDatabase = {
+ val cmd = "org.apache.spark.sql.execution.command.SetDatabaseCommand"
+ val databaseDesc = DatabaseDesc("databaseName", "StringDatabaseExtractor", true)
+ DatabaseCommandSpec(cmd, Seq(databaseDesc), "SWITCHDATABASE")
+ }
+
+ val DescribeDatabase = {
+ val cmd = "org.apache.spark.sql.execution.command.DescribeDatabaseCommand"
+ val databaseDesc = DatabaseDesc("databaseName", "StringDatabaseExtractor", true)
+ DatabaseCommandSpec(cmd, Seq(databaseDesc), "DESCDATABASE")
+ }
+
+ val SetCatalogAndNamespace = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.SetCatalogAndNamespace"
+ val databaseDesc1 = DatabaseDesc("child", "ResolvedDBObjectNameDatabaseExtractor", true)
+ val databaseDesc2 = DatabaseDesc("namespace", "StringSeqOptionDatabaseExtractor", true)
+ DatabaseCommandSpec(cmd, Seq(databaseDesc1, databaseDesc2), "SWITCHDATABASE")
+ }
+
+ val SetNamespace = {
+ val cmd = "org.apache.spark.sql.execution.command.SetNamespaceCommand"
+ val databaseDesc = DatabaseDesc("namespace", "StringSeqDatabaseExtractor", true)
+ DatabaseCommandSpec(cmd, Seq(databaseDesc), "SWITCHDATABASE")
+ }
+
+ val DescribeNamespace = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.DescribeNamespace"
+ val databaseDesc = DatabaseDesc("namespace", "ResolvedNamespaceDatabaseExtractor", true)
+ DatabaseCommandSpec(cmd, Seq(databaseDesc), "DESCDATABASE")
+ }
+
+ val data = Array(
+ AlterDatabaseProperties,
+ AlterDatabaseProperties.copy(
+ classname = "org.apache.spark.sql.execution.command.AlterDatabaseSetLocationCommand",
+ opType = "ALTERDATABASE_LOCATION"),
+ AlterDatabaseProperties.copy(
+ classname = "org.apache.spark.sql.execution.command.CreateDatabaseCommand",
+ opType = "CREATEDATABASE"),
+ AlterDatabaseProperties.copy(
+ classname = "org.apache.spark.sql.execution.command.DropDatabaseCommand",
+ opType = "DROPDATABASE"),
+ AnalyzeTables,
+ CreateNamespace,
+ CommentOnNamespace,
+ DescribeDatabase,
+ DescribeNamespace,
+ DropNamespace,
+ SetCatalogAndNamespace,
+ SetDatabase,
+ SetNamespace,
+ SetNamespaceProperties,
+ SetNamespaceLocation)
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/FunctionCommands.scala b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/FunctionCommands.scala
new file mode 100644
index 000000000..41bafedcb
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/FunctionCommands.scala
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.gen
+
+import org.apache.kyuubi.plugin.spark.authz.serde.{DatabaseDesc, FunctionCommandSpec, FunctionDesc, FunctionTypeDesc, StringOptionDatabaseExtractor, TempMarkerFunctionTypeExtractor}
+
+object FunctionCommands {
+
+ val CreateFunction = {
+ val cmd = "org.apache.spark.sql.execution.command.CreateFunctionCommand"
+ val functionTypeDesc = FunctionTypeDesc(
+ "isTemp",
+ classOf[TempMarkerFunctionTypeExtractor].getSimpleName,
+ Seq("TEMP"))
+ val databaseDesc = DatabaseDesc(
+ "databaseName",
+ classOf[StringOptionDatabaseExtractor].getSimpleName)
+ val functionDesc = FunctionDesc(
+ "functionName",
+ "StringFunctionExtractor",
+ Some(databaseDesc),
+ Some(functionTypeDesc))
+ FunctionCommandSpec(cmd, Seq(functionDesc), "CREATEFUNCTION")
+ }
+
+ val DescribeFunction = {
+ val cmd = "org.apache.spark.sql.execution.command.DescribeFunctionCommand"
+ val skips = Seq("TEMP", "SYSTEM")
+ val functionTypeDesc1 = FunctionTypeDesc("info", "ExpressionInfoFunctionTypeExtractor", skips)
+ val functionDesc1 = FunctionDesc(
+ "info",
+ "ExpressionInfoFunctionExtractor",
+ functionTypeDesc = Some(functionTypeDesc1),
+ isInput = true)
+
+ val functionTypeDesc2 =
+ FunctionTypeDesc("functionName", "FunctionIdentifierFunctionTypeExtractor", skips)
+ val functionDesc2 = FunctionDesc(
+ "functionName",
+ "FunctionIdentifierFunctionExtractor",
+ functionTypeDesc = Some(functionTypeDesc2),
+ isInput = true)
+ FunctionCommandSpec(cmd, Seq(functionDesc1, functionDesc2), "DESCFUNCTION")
+ }
+
+ val DropFunction = {
+ val cmd = "org.apache.spark.sql.execution.command.DropFunctionCommand"
+ CreateFunction.copy(cmd, opType = "DROPFUNCTION")
+ }
+
+ val RefreshFunction = {
+ val cmd = "org.apache.spark.sql.execution.command.RefreshFunctionCommand"
+ val databaseDesc = DatabaseDesc("databaseName", "StringOptionDatabaseExtractor")
+ val functionDesc = FunctionDesc(
+ "functionName",
+ "StringFunctionExtractor",
+ Some(databaseDesc))
+ FunctionCommandSpec(cmd, Seq(functionDesc), "RELOADFUNCTION")
+ }
+
+ val data = Array(
+ CreateFunction,
+ DropFunction,
+ DescribeFunction,
+ RefreshFunction).sortBy(_.classname)
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/IcebergCommands.scala b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/IcebergCommands.scala
new file mode 100644
index 000000000..66e7d3a25
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/IcebergCommands.scala
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.gen
+
+import org.apache.kyuubi.plugin.spark.authz.serde.{ActionTypeDesc, QueryDesc, TableCommandSpec, TableDesc}
+
+object IcebergCommands {
+
+ val DeleteFromIcebergTable = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.DeleteFromIcebergTable"
+ val actionTypeDesc = ActionTypeDesc(null, null, Some("UPDATE"))
+ val tableDesc =
+ TableDesc(
+ "table",
+ "DataSourceV2RelationTableExtractor",
+ actionTypeDesc = Some(actionTypeDesc))
+ TableCommandSpec(cmd, Seq(tableDesc), queryDescs = Seq(QueryDesc("query")))
+ }
+
+ val UpdateIcebergTable = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.UpdateIcebergTable"
+ DeleteFromIcebergTable.copy(cmd)
+ }
+
+ val MergeIntoIcebergTable = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable"
+ val actionTypeDesc = ActionTypeDesc(null, null, Some("UPDATE"))
+ val tableDesc = TableDesc(
+ "targetTable",
+ "DataSourceV2RelationTableExtractor",
+ actionTypeDesc = Some(actionTypeDesc))
+ val queryDesc = QueryDesc("sourceTable")
+ TableCommandSpec(cmd, Seq(tableDesc), queryDescs = Seq(queryDesc))
+ }
+
+ val data = Array(
+ DeleteFromIcebergTable,
+ UpdateIcebergTable,
+ MergeIntoIcebergTable)
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/JsonSpecFileGenerator.scala b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/JsonSpecFileGenerator.scala
new file mode 100644
index 000000000..101758f3c
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/JsonSpecFileGenerator.scala
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.gen
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Paths}
+
+import org.apache.kyuubi.plugin.spark.authz.serde.{mapper, CommandSpec}
+
+/**
+ * Generates the default command specs to src/main/resources dir.
+ *
+ * Usage:
+ * mvn scala:run -DmainClass=this class -pl :kyuubi-spark-authz_2.12
+ */
+object JsonSpecFileGenerator {
+
+ def main(args: Array[String]): Unit = {
+ write(DatabaseCommands.data, "database")
+ write(TableCommands.data ++ IcebergCommands.data, "table")
+ write(FunctionCommands.data, "function")
+ }
+
+ def write[T <: CommandSpec](data: Array[T], filename: String): Unit = {
+ val pluginHome = getClass.getProtectionDomain.getCodeSource.getLocation.getPath
+ .split("target").head
+ val writer = {
+ val p =
+ Paths.get(pluginHome, "src", "main", "resources", s"${filename}_command_spec.json")
+ Files.newBufferedWriter(p, StandardCharsets.UTF_8)
+ }
+ mapper.writerWithDefaultPrettyPrinter().writeValue(writer, data.sortBy(_.classname))
+ writer.close()
+ }
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/TableCommands.scala b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/TableCommands.scala
new file mode 100644
index 000000000..922c4eba8
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/TableCommands.scala
@@ -0,0 +1,600 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.gen
+
+import org.apache.kyuubi.plugin.spark.authz.serde._
+
+object TableCommands {
+ // table extractors
+ val tite = classOf[TableIdentifierTableExtractor].getSimpleName
+ val tableNameDesc = TableDesc("tableName", tite)
+ val tableIdentDesc = TableDesc("tableIdent", tite)
+ val resolvedTableDesc = TableDesc("child", "ResolvedTableTableExtractor")
+
+ val overwriteActionTypeDesc =
+ ActionTypeDesc("overwrite", "OverwriteOrInsertActionTypeExtractor")
+
+ val AlterTable = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.AlterTable"
+ val tableDesc = TableDesc("ident", classOf[IdentifierTableExtractor].getSimpleName)
+ TableCommandSpec(cmd, Seq(tableDesc), "ALTERTABLE_PROPERTIES")
+ }
+
+ val AlterTableAddColumns = {
+ val cmd = "org.apache.spark.sql.execution.command.AlterTableAddColumnsCommand"
+ val columnDesc = ColumnDesc("colsToAdd", "StructFieldSeqColumnExtractor")
+ val tableDesc = TableDesc("table", tite, Some(columnDesc))
+ TableCommandSpec(cmd, Seq(tableDesc), "ALTERTABLE_ADDCOLS")
+ }
+
+ val AddColumns = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.AddColumns"
+ TableCommandSpec(cmd, Seq(resolvedTableDesc), "ALTERTABLE_ADDCOLS")
+ }
+
+ val AlterColumn = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.AlterColumn"
+ AddColumns.copy(classname = cmd)
+ }
+
+ val DropColumns = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.DropColumns"
+ AddColumns.copy(classname = cmd)
+ }
+
+ val ReplaceColumns = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.ReplaceColumns"
+ TableCommandSpec(cmd, Seq(resolvedTableDesc), "ALTERTABLE_REPLACECOLS")
+ }
+
+ val RenameColumn = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.RenameColumn"
+ TableCommandSpec(cmd, Seq(resolvedTableDesc), "ALTERTABLE_RENAMECOL")
+ }
+
+ val AlterTableAddPartition = {
+ val cmd = "org.apache.spark.sql.execution.command.AlterTableAddPartitionCommand"
+ val columnDesc = ColumnDesc("partitionSpecsAndLocs", "PartitionLocsSeqColumnExtractor")
+ TableCommandSpec(
+ cmd,
+ Seq(tableNameDesc.copy(columnDesc = Some(columnDesc))),
+ "ALTERTABLE_ADDPARTS")
+ }
+
+ val AlterTableChangeColumn = {
+ val cmd = "org.apache.spark.sql.execution.command.AlterTableChangeColumnCommand"
+ val columnDesc = ColumnDesc("columnName", "StringColumnExtractor")
+ TableCommandSpec(
+ cmd,
+ Seq(tableNameDesc.copy(columnDesc = Some(columnDesc))),
+ "ALTERTABLE_REPLACECOLS")
+ }
+
+ val AlterTableDropPartition = {
+ val cmd = "org.apache.spark.sql.execution.command.AlterTableDropPartitionCommand"
+ val columnDesc = ColumnDesc("specs", "PartitionSeqColumnExtractor")
+ TableCommandSpec(
+ cmd,
+ Seq(tableNameDesc.copy(columnDesc = Some(columnDesc))),
+ "ALTERTABLE_DROPPARTS")
+ }
+
+ val AlterTableRename = {
+ val cmd = "org.apache.spark.sql.execution.command.AlterTableRenameCommand"
+ val actionTypeDesc = ActionTypeDesc(null, null, Some("DELETE"))
+
+ val oldTableTableTypeDesc =
+ TableTypeDesc("oldName", "TableIdentifierTableTypeExtractor", Seq("TEMP_VIEW"))
+ val oldTableD = TableDesc(
+ "oldName",
+ tite,
+ tableTypeDesc = Some(oldTableTableTypeDesc),
+ actionTypeDesc = Some(actionTypeDesc))
+
+ val newTableD =
+ TableDesc("newName", tite, tableTypeDesc = Some(oldTableTableTypeDesc))
+ TableCommandSpec(cmd, Seq(oldTableD, newTableD), "ALTERTABLE_RENAME")
+ }
+
+ // this is for spark 3.1 or below
+ val AlterTableRecoverPartitions = {
+ val cmd = "org.apache.spark.sql.execution.command.AlterTableRecoverPartitionsCommand"
+ TableCommandSpec(cmd, Seq(tableNameDesc), "MSCK")
+ }
+
+ val RepairTable = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.RepairTable"
+ TableCommandSpec(cmd, Seq(resolvedTableDesc), "MSCK")
+ }
+
+ val AlterTableRenamePartition = {
+ val cmd = "org.apache.spark.sql.execution.command.AlterTableRenamePartitionCommand"
+ val columnDesc = ColumnDesc("oldPartition", "PartitionColumnExtractor")
+ TableCommandSpec(
+ cmd,
+ Seq(tableNameDesc.copy(columnDesc = Some(columnDesc))),
+ "ALTERTABLE_RENAMEPART")
+ }
+
+ val AlterTableSerDeProperties = {
+ val cmd = "org.apache.spark.sql.execution.command.AlterTableSerDePropertiesCommand"
+ val columnDesc = ColumnDesc("partSpec", "PartitionOptionColumnExtractor")
+ TableCommandSpec(
+ cmd,
+ Seq(tableNameDesc.copy(columnDesc = Some(columnDesc))),
+ "ALTERTABLE_SERDEPROPERTIES")
+ }
+
+ val AlterTableSetLocation = {
+ val cmd = "org.apache.spark.sql.execution.command.AlterTableSetLocationCommand"
+ val columnDesc = ColumnDesc("partitionSpec", "PartitionOptionColumnExtractor")
+ TableCommandSpec(
+ cmd,
+ Seq(tableNameDesc.copy(columnDesc = Some(columnDesc))),
+ "ALTERTABLE_LOCATION")
+ }
+
+ val AlterTableSetProperties = TableCommandSpec(
+ "org.apache.spark.sql.execution.command.AlterTableSetPropertiesCommand",
+ Seq(tableNameDesc),
+ "ALTERTABLE_PROPERTIES")
+
+ val AlterTableUnsetProperties = AlterTableSetProperties.copy(classname =
+ "org.apache.spark.sql.execution.command.AlterTableUnsetPropertiesCommand")
+
+ val AlterViewAs = {
+ val tableTypeDesc =
+ TableTypeDesc("name", "TableIdentifierTableTypeExtractor", Seq("TEMP_VIEW"))
+
+ TableCommandSpec(
+ "org.apache.spark.sql.execution.command.AlterViewAsCommand",
+ Seq(TableDesc("name", tite, tableTypeDesc = Some(tableTypeDesc))),
+ "ALTERVIEW_AS",
+ Seq(QueryDesc("query")))
+ }
+
+ val AnalyzeColumn = {
+ val cmd = "org.apache.spark.sql.execution.command.AnalyzeColumnCommand"
+ val cd1 = ColumnDesc("columnNames", "StringSeqColumnExtractor")
+ val cd2 = cd1.copy(fieldExtractor = "StringSeqOptionColumnExtractor")
+ val td1 = tableIdentDesc.copy(columnDesc = Some(cd1), isInput = true)
+ val td2 = td1.copy(columnDesc = Some(cd2))
+ TableCommandSpec(cmd, Seq(td1, td2), "ANALYZE_TABLE")
+ }
+
+ val AnalyzePartition = {
+ val cmd = "org.apache.spark.sql.execution.command.AnalyzePartitionCommand"
+ val columnDesc = ColumnDesc("partitionSpec", "PartitionColumnExtractor")
+ TableCommandSpec(
+ cmd,
+ Seq(tableIdentDesc.copy(columnDesc = Some(columnDesc), isInput = true)),
+ "ANALYZE_TABLE")
+ }
+
+ val AnalyzeTable = {
+ val cmd = "org.apache.spark.sql.execution.command.AnalyzeTableCommand"
+ TableCommandSpec(
+ cmd,
+ Seq(tableIdentDesc.copy(isInput = true)),
+ "ANALYZE_TABLE")
+ }
+
+ val CreateTableV2 = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.CreateTable"
+ val tableDesc = TableDesc("tableName", "IdentifierTableExtractor")
+ TableCommandSpec(cmd, Seq(tableDesc), "CREATETABLE")
+ }
+
+ val CreateTableAsSelectV2 = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.CreateTableAsSelect"
+ val tableDesc = TableDesc("tableName", "IdentifierTableExtractor")
+ TableCommandSpec(cmd, Seq(tableDesc), "CREATETABLE_AS_SELECT", Seq(QueryDesc("query")))
+ }
+
+ val CommentOnTable = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.CommentOnTable"
+ val tableDesc = resolvedTableDesc
+ TableCommandSpec(cmd, Seq(tableDesc), "ALTERTABLE_PROPERTIES")
+ }
+
+ val AppendDataV2 = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.AppendData"
+ val actionTypeDesc = ActionTypeDesc(null, null, Some("INSERT"))
+ val tableDesc =
+ TableDesc(
+ "table",
+ "DataSourceV2RelationTableExtractor",
+ actionTypeDesc = Some(actionTypeDesc))
+ TableCommandSpec(cmd, Seq(tableDesc), queryDescs = Seq(QueryDesc("query")))
+ }
+
+ val UpdateTable = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.UpdateTable"
+ val actionTypeDesc = ActionTypeDesc(null, null, Some("UPDATE"))
+ val tableDesc =
+ TableDesc(
+ "table",
+ "DataSourceV2RelationTableExtractor",
+ actionTypeDesc = Some(actionTypeDesc))
+ TableCommandSpec(cmd, Seq(tableDesc), queryDescs = Seq(QueryDesc("query")))
+ }
+
+ val DeleteFromTable = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.DeleteFromTable"
+ UpdateTable.copy(classname = cmd)
+ }
+
+ val OverwriteByExpression = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.OverwriteByExpression"
+ val actionTypeDesc = ActionTypeDesc(null, null, Some("INSERT_OVERWRITE"))
+ val tableDesc =
+ TableDesc(
+ "table",
+ "DataSourceV2RelationTableExtractor",
+ actionTypeDesc = Some(actionTypeDesc))
+ TableCommandSpec(cmd, Seq(tableDesc), queryDescs = Seq(QueryDesc("query")))
+ }
+
+ val OverwritePartitionsDynamic = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.OverwritePartitionsDynamic"
+ OverwriteByExpression.copy(classname = cmd)
+ }
+
+ val AddPartitions = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.AddPartitions"
+ // TODO: add column desc
+ val tableDesc = TableDesc("table", "DataSourceV2RelationTableExtractor")
+ TableCommandSpec(cmd, Seq(tableDesc), "ALTERTABLE_ADDPARTS")
+ }
+
+ val DropPartitions = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.DropPartitions"
+ // TODO: add column desc
+ val tableDesc = TableDesc("table", "DataSourceV2RelationTableExtractor")
+ TableCommandSpec(cmd, Seq(tableDesc), "ALTERTABLE_DROPPARTS")
+ }
+
+ val RenamePartitions = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.RenamePartitions"
+ // TODO: add column desc
+ val tableDesc = TableDesc("table", "DataSourceV2RelationTableExtractor")
+ TableCommandSpec(cmd, Seq(tableDesc), "ALTERTABLE_RENAMEPART")
+ }
+
+ val TruncatePartition = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.TruncatePartition"
+ // TODO: add column desc
+ val tableDesc = TableDesc("table", "DataSourceV2RelationTableExtractor")
+ TableCommandSpec(cmd, Seq(tableDesc), "ALTERTABLE_DROPPARTS")
+ }
+
+ val CacheTableAsSelect = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.CacheTableAsSelect"
+ TableCommandSpec(cmd, Nil, "CREATEVIEW", queryDescs = Seq(QueryDesc("plan")))
+ }
+
+ val CacheTable = {
+ val cmd = "org.apache.spark.sql.execution.command.CacheTableCommand"
+ val queryDesc = QueryDesc("plan", "LogicalPlanOptionQueryExtractor")
+ TableCommandSpec(cmd, Nil, "CREATEVIEW", queryDescs = Seq(queryDesc))
+ }
+
+ val CacheTableV2 = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.CacheTable"
+ TableCommandSpec(cmd, Nil, "CREATEVIEW", Seq(QueryDesc("table")))
+ }
+
+ val CreateView = {
+ val cmd = "org.apache.spark.sql.execution.command.CreateViewCommand"
+ val tableTypeDesc = TableTypeDesc(
+ "viewType",
+ "ViewTypeTableTypeExtractor",
+ Seq("TEMP_VIEW", "GLOBAL_TEMP_VIEW"))
+ val tableDesc = TableDesc(
+ "name",
+ "TableIdentifierTableExtractor",
+ tableTypeDesc = Some(tableTypeDesc))
+ val queryDesc1 = QueryDesc("plan")
+ val queryDesc2 = QueryDesc("child")
+ TableCommandSpec(cmd, Seq(tableDesc), "CREATEVIEW", queryDescs = Seq(queryDesc1, queryDesc2))
+ }
+
+ val CreateTempViewUsing = {
+ val cmd = "org.apache.spark.sql.execution.datasources.CreateTempViewUsing"
+ TableCommandSpec(cmd, Nil, "CREATEVIEW")
+ }
+
+ val CreateDataSourceTable = {
+ val cmd = "org.apache.spark.sql.execution.command.CreateDataSourceTableCommand"
+ val tableDesc = TableDesc("table", "CatalogTableTableExtractor")
+ TableCommandSpec(cmd, Seq(tableDesc), "CREATETABLE")
+ }
+
+ val CreateDataSourceTableAsSelect = {
+ val cmd = "org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand"
+ CreateDataSourceTable.copy(
+ classname = cmd,
+ opType = "CREATETABLE_AS_SELECT",
+ queryDescs = Seq(QueryDesc("query")))
+ }
+
+ val CreateHiveTableAsSelect = {
+ val cmd = "org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand"
+ val columnDesc = ColumnDesc("outputColumnNames", "StringSeqColumnExtractor")
+ val tableDesc = TableDesc("tableDesc", "CatalogTableTableExtractor", Some(columnDesc))
+ val queryDesc = QueryDesc("query")
+ TableCommandSpec(cmd, Seq(tableDesc), "CREATETABLE_AS_SELECT", queryDescs = Seq(queryDesc))
+ }
+
+ val CreateTableLike = {
+ val cmd = "org.apache.spark.sql.execution.command.CreateTableLikeCommand"
+ val tableDesc1 = TableDesc(
+ "targetTable",
+ "TableIdentifierTableExtractor",
+ setCurrentDatabaseIfMissing = true)
+ val tableDesc2 = TableDesc(
+ "sourceTable",
+ "TableIdentifierTableExtractor",
+ isInput = true,
+ setCurrentDatabaseIfMissing = true)
+ TableCommandSpec(cmd, Seq(tableDesc1, tableDesc2), "CREATETABLE")
+ }
+
+ val DescribeColumn = {
+ val cmd = "org.apache.spark.sql.execution.command.DescribeColumnCommand"
+ val columnDesc = ColumnDesc("colNameParts", "StringSeqLastColumnExtractor")
+ val tableDesc = TableDesc(
+ "table",
+ "TableIdentifierTableExtractor",
+ Some(columnDesc),
+ isInput = true)
+ TableCommandSpec(cmd, Seq(tableDesc), "DESCTABLE")
+ }
+
+ val DescribeTable = {
+ val cmd = "org.apache.spark.sql.execution.command.DescribeTableCommand"
+ val columnDesc = ColumnDesc("partitionSpec", "PartitionColumnExtractor")
+ val tableDesc = TableDesc(
+ "table",
+ "TableIdentifierTableExtractor",
+ Some(columnDesc),
+ isInput = true,
+ setCurrentDatabaseIfMissing = true)
+ TableCommandSpec(cmd, Seq(tableDesc), "DESCTABLE")
+ }
+
+ val DropTable = {
+ val cmd = "org.apache.spark.sql.execution.command.DropTableCommand"
+ val tableTypeDesc =
+ TableTypeDesc("tableName", "TableIdentifierTableTypeExtractor", Seq("TEMP_VIEW"))
+ TableCommandSpec(
+ cmd,
+ Seq(tableNameDesc.copy(tableTypeDesc = Some(tableTypeDesc))),
+ "DROPTABLE")
+ }
+
+ val DropTableV2 = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.DropTable"
+ val tableDesc1 = resolvedTableDesc
+ val tableDesc2 = TableDesc("ident", "IdentifierTableExtractor")
+ TableCommandSpec(cmd, Seq(tableDesc1, tableDesc2), "DROPTABLE")
+ }
+
+ val MergeIntoTable = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.MergeIntoTable"
+ val actionTypeDesc = ActionTypeDesc(null, null, Some("UPDATE"))
+ val tableDesc = TableDesc(
+ "targetTable",
+ "DataSourceV2RelationTableExtractor",
+ actionTypeDesc = Some(actionTypeDesc))
+ val queryDesc = QueryDesc("sourceTable")
+ TableCommandSpec(cmd, Seq(tableDesc), queryDescs = Seq(queryDesc))
+ }
+
+ val ShowColumns = {
+ val cmd = "org.apache.spark.sql.execution.command.ShowColumnsCommand"
+ val tableDesc = tableNameDesc.copy(isInput = true)
+ TableCommandSpec(cmd, Seq(tableDesc), "SHOWCOLUMNS")
+ }
+
+ val ShowCreateTable = {
+ val cmd = "org.apache.spark.sql.execution.command.ShowCreateTableCommand"
+ val tableDesc = tableNameDesc.copy(fieldName = "table", isInput = true)
+ TableCommandSpec(cmd, Seq(tableDesc), "SHOW_CREATETABLE")
+ }
+
+ val ShowTableProperties = {
+ val cmd = "org.apache.spark.sql.execution.command.ShowTablePropertiesCommand"
+ val tableDesc = tableNameDesc.copy(fieldName = "table", isInput = true)
+ TableCommandSpec(cmd, Seq(tableDesc), "SHOW_TBLPROPERTIES")
+ }
+
+ val ShowCreateTableV2 = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.ShowCreateTable"
+ val tableDesc = TableDesc("child", "ResolvedTableTableExtractor", isInput = true)
+ TableCommandSpec(cmd, Seq(tableDesc), "SHOW_CREATETABLE")
+ }
+
+ val ShowTablePropertiesV2 = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.ShowTableProperties"
+ val tableDesc = TableDesc("table", "ResolvedTableTableExtractor", isInput = true)
+ TableCommandSpec(cmd, Seq(tableDesc), "SHOW_TBLPROPERTIES")
+ }
+
+ val ShowPartitions = {
+ val cmd = "org.apache.spark.sql.execution.command.ShowPartitionsCommand"
+ val columnDesc = ColumnDesc("spec", "PartitionOptionColumnExtractor")
+ val tableDesc = tableNameDesc.copy(isInput = true, columnDesc = Some(columnDesc))
+ TableCommandSpec(cmd, Seq(tableDesc), "SHOWPARTITIONS")
+ }
+
+ val TruncateTable = {
+ val cmd = "org.apache.spark.sql.execution.command.TruncateTableCommand"
+ val columnDesc = ColumnDesc("partitionSpec", "PartitionOptionColumnExtractor")
+ val tableDesc = tableNameDesc.copy(columnDesc = Some(columnDesc))
+ TableCommandSpec(cmd, Seq(tableDesc), "TRUNCATETABLE")
+ }
+
+ val TruncateTableV2 = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.TruncateTable"
+ TableCommandSpec(cmd, Seq(resolvedTableDesc), "TRUNCATETABLE")
+ }
+
+ val InsertIntoDataSource = {
+ val cmd = "org.apache.spark.sql.execution.datasources.InsertIntoDataSourceCommand"
+ val actionTypeDesc = overwriteActionTypeDesc
+ val tableDesc = TableDesc(
+ "logicalRelation",
+ "LogicalRelationTableExtractor",
+ actionTypeDesc = Some(actionTypeDesc))
+ TableCommandSpec(cmd, Seq(tableDesc), queryDescs = Seq(QueryDesc("query")))
+ }
+
+ val InsertIntoHiveTable = {
+ val cmd = "org.apache.spark.sql.hive.execution.InsertIntoHiveTable"
+ val actionTypeDesc = overwriteActionTypeDesc
+ val columnDesc = ColumnDesc("outputColumnNames", "StringSeqColumnExtractor")
+ val tableDesc = TableDesc(
+ "table",
+ "CatalogTableTableExtractor",
+ Some(columnDesc),
+ Some(actionTypeDesc))
+ val queryDesc = QueryDesc("query")
+ TableCommandSpec(cmd, Seq(tableDesc), queryDescs = Seq(queryDesc))
+ }
+
+ val InsertIntoDataSourceDir = {
+ val cmd = "org.apache.spark.sql.execution.command.InsertIntoDataSourceDirCommand"
+ val queryDesc = QueryDesc("query")
+ TableCommandSpec(cmd, Nil, queryDescs = Seq(queryDesc))
+ }
+
+ val LoadData = {
+ val cmd = "org.apache.spark.sql.execution.command.LoadDataCommand"
+ val actionTypeDesc = overwriteActionTypeDesc.copy(fieldName = "isOverwrite")
+ val columnDesc = ColumnDesc("partition", "PartitionOptionColumnExtractor")
+ val tableDesc = tableIdentDesc.copy(
+ fieldName = "table",
+ columnDesc = Some(columnDesc),
+ actionTypeDesc = Some(actionTypeDesc))
+ TableCommandSpec(cmd, Seq(tableDesc), "LOAD")
+ }
+
+ val RefreshTable = {
+ val cmd = "org.apache.spark.sql.execution.command.RefreshTableCommand"
+ TableCommandSpec(cmd, Seq(tableIdentDesc.copy(isInput = true)))
+ }
+
+ val RefreshTableV2 = {
+ val cmd = "org.apache.spark.sql.catalyst.plans.logical.RefreshTable"
+ TableCommandSpec(cmd, Seq(resolvedTableDesc.copy(isInput = true)))
+ }
+
+ val RefreshTable3d0 = {
+ val cmd = "org.apache.spark.sql.execution.datasources.RefreshTable"
+ TableCommandSpec(cmd, Seq(tableIdentDesc.copy(isInput = true)))
+ }
+
+ val data = Array(
+ AddPartitions,
+ DropPartitions,
+ RenamePartitions,
+ TruncatePartition,
+ AddColumns,
+ AlterColumn,
+ DropColumns,
+ ReplaceColumns,
+ RenameColumn,
+ AlterTable,
+ AlterTableAddColumns,
+ AlterTableAddPartition,
+ AlterTableChangeColumn,
+ AlterTableDropPartition,
+ AlterTableRename,
+ AlterTableRecoverPartitions,
+ AlterTableRecoverPartitions.copy(classname =
+ "org.apache.spark.sql.execution.command.RepairTableCommand"),
+ AlterTableRenamePartition,
+ AlterTableSerDeProperties,
+ AlterTableSetLocation,
+ AlterTableSetProperties,
+ AlterTableUnsetProperties,
+ AlterViewAs,
+ AnalyzeColumn,
+ AnalyzePartition,
+ AnalyzeTable,
+ AnalyzeTable.copy(classname =
+ "org.apache.spark.sql.execution.command.AnalyzeTablesCommand"),
+ AppendDataV2,
+ CacheTable,
+ CacheTableAsSelect,
+ CacheTableV2,
+ CommentOnTable,
+ CreateDataSourceTable,
+ CreateDataSourceTable.copy(classname =
+ "org.apache.spark.sql.execution.command.CreateTableCommand"),
+ CreateDataSourceTableAsSelect,
+ CreateHiveTableAsSelect,
+ CreateHiveTableAsSelect.copy(classname =
+ "org.apache.spark.sql.hive.execution.OptimizedCreateHiveTableAsSelectCommand"),
+ CreateTableLike,
+ CreateTableV2,
+ CreateTableV2.copy(classname =
+ "org.apache.spark.sql.catalyst.plans.logical.ReplaceTable"),
+ CreateTableV2.copy(classname =
+ "org.apache.spark.sql.catalyst.plans.logical.CreateV2Table"),
+ CreateTableAsSelectV2,
+ CreateTableAsSelectV2.copy(classname =
+ "org.apache.spark.sql.catalyst.plans.logical.ReplaceTableAsSelect"),
+ CreateView,
+ CreateTempViewUsing,
+ DeleteFromTable,
+ DescribeColumn,
+ DescribeTable,
+ DropTable,
+ DropTableV2,
+ InsertIntoDataSource,
+ InsertIntoDataSourceDir,
+ InsertIntoDataSourceDir.copy(classname =
+ "org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand"),
+ // TODO: InsertIntoHadoopFsRelationCommand may have a CatalogTable
+ InsertIntoDataSourceDir.copy(classname =
+ "org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand"),
+ InsertIntoDataSourceDir.copy(classname =
+ "org.apache.spark.sql.execution.datasources.InsertIntoHiveDirCommand"),
+ InsertIntoHiveTable,
+ LoadData,
+ MergeIntoTable,
+ OverwriteByExpression,
+ OverwritePartitionsDynamic,
+ RepairTable,
+ RefreshTable,
+ RefreshTableV2,
+ RefreshTable3d0,
+ ShowColumns,
+ ShowCreateTable,
+ ShowCreateTable.copy(classname =
+ "org.apache.spark.sql.execution.command.ShowCreateTableAsSerdeCommand"),
+ ShowCreateTableV2,
+ ShowPartitions,
+ ShowTableProperties,
+ ShowTablePropertiesV2,
+ TruncateTable,
+ TruncateTableV2,
+ UpdateTable)
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RangerSparkExtensionSuite.scala b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RangerSparkExtensionSuite.scala
index a8d41a63c..e1a2fc6a0 100644
--- a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RangerSparkExtensionSuite.scala
+++ b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RangerSparkExtensionSuite.scala
@@ -823,7 +823,8 @@ class HiveCatalogRangerSparkExtensionSuite extends RangerSparkExtensionSuite {
s" [select] privilege on" +
s" [$db1/$srcTable1/id,$db1/$srcTable1/name,$db1/$srcTable1/city," +
s"$db1/$srcTable2/age,$db1/$srcTable2/id]," +
- s" [update] privilege on [$db1/$sinkTable1]"))
+ s" [update] privilege on [$db1/$sinkTable1/id,$db1/$sinkTable1/age," +
+ s"$db1/$sinkTable1/name,$db1/$sinkTable1/city]"))
} finally {
// revert to default value
SparkRangerAdminPlugin.getRangerConf.setBoolean(