You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/09/06 13:50:39 UTC
[GitHub] [incubator-kyuubi] bowenliang123 opened a new pull request, #3425: [KYUUBI #3424] [FEATURE] Support checking Access privilege for DatasourceV2
bowenliang123 opened a new pull request, #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
close #3424
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969512210
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
+}
+
+object v2Commands extends Enumeration {
+
+ import scala.language.implicitConversions
+
+ implicit def valueToV2CommandBuilder(x: Value): V2Command =
+ x.asInstanceOf[V2Command]
+
+ def accept(commandName: String): Boolean = {
+ try {
+ val cmd = v2Commands.withName(commandName)
+
+ // check spark version requirements
+ def passSparkVersionCheck: Boolean =
+ (cmd.mostVer.isEmpty || isSparkVersionAtMost(cmd.mostVer.get)) &&
+ (cmd.leastVer.isEmpty || isSparkVersionAtLeast(cmd.leastVer.get))
+
+ passSparkVersionCheck && cmd.enabled
+ } catch {
+ case _: NoSuchElementException => false
+ }
+ }
+
+ val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ (p, inputObjs, cmdTypes) => {
+ cmdTypes.foreach {
+ case HasQuery =>
+ val query = getFieldVal[LogicalPlan](p, "query")
+ buildQuery(query, inputObjs)
+ case _ =>
+ }
+ }
+
+ val defaultBuildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit =
+ (p, outputObjs, cmdTypes, outputObjsActionType) => {
+ cmdTypes.foreach {
+ case V2CreateTablePlan =>
+ val table = invoke(p, "tableName").asInstanceOf[Identifier]
+ outputObjs += v2TablePrivileges(table)
+
+ case V2WriteCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get, actionType = outputObjsActionType)
+ }
+
+ case V2AlterTableCommand if isSparkVersionAtLeast("3.2") =>
+ val table = getFieldVal[LogicalPlan](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2DdlTableCommand =>
+ val table = getFieldVal[AnyRef](p, "child")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2PartitionCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case _ =>
+ }
+ }
+
+ case class V2Command(
+ operationType: OperationType = QUERY,
+ leastVer: Option[String] = None,
+ mostVer: Option[String] = None,
+ cmdTypes: Seq[V2CommandType] = Seq(),
+ buildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ defaultBuildInput,
+ buildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit = defaultBuildOutput,
+ outputActionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER,
+ enabled: Boolean = true) {
+
+ def buildCommand(
+ plan: LogicalPlan,
+ inputObjs: ArrayBuffer[PrivilegeObject],
+ outputObjs: ArrayBuffer[PrivilegeObject]): Unit = {
+ this.buildInput(plan, inputObjs, cmdTypes)
+ this.buildOutput(plan, outputObjs, cmdTypes, outputActionType)
+ }
+ }
+
+ def v2TablePrivileges(
+ table: Identifier,
+ columns: Seq[String] = Nil,
+ actionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER): PrivilegeObject = {
+ PrivilegeObject(TABLE_OR_VIEW, actionType, quote(table.namespace()), table.name(), columns)
+ }
+
+ // /////////////////////////////////////////////////////////////////////////////////////////////
Review Comment:
remove this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966840289
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -47,6 +48,13 @@ object PrivilegesBuilder {
PrivilegeObject(TABLE_OR_VIEW, actionType, table.database.orNull, table.table, columns)
}
+ private def tablePrivilegesWithIdentifier(
Review Comment:
how about v2TablePrivileges and v1...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966848736
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -411,6 +469,26 @@ object PrivilegesBuilder {
outputObjs += tablePrivileges(table, actionType = actionType)
buildQuery(getQuery, inputObjs)
+ case "AppendData" => // DsV2 insert
Review Comment:
the comment here is useless.
How about we group v2 commands together with or without a new method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969465669
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
+}
+
+object v2Commands extends Enumeration {
+
+ import scala.language.implicitConversions
+
+ implicit def valueToV2CommandBuilder(x: Value): V2Command =
+ x.asInstanceOf[V2Command]
+
+ def accept(commandName: String): Boolean = {
+ try {
+ val cmd = v2Commands.withName(commandName)
+
+ // check spark version requirements
+ def passSparkVersionCheck: Boolean =
+ (StringUtils.isBlank(cmd.mostVer) || isSparkVersionAtMost(cmd.mostVer)) &&
+ (StringUtils.isBlank(cmd.leastVer) || isSparkVersionAtLeast(cmd.leastVer))
+
+ passSparkVersionCheck && cmd.enabled
+ } catch {
+ case _: NoSuchElementException => false
+ }
+ }
+
+ val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ (p, inputObjs, cmdTypes) => {
+ cmdTypes.foreach {
+ case HasQuery =>
+ val query = getFieldVal[LogicalPlan](p, "query")
+ buildQuery(query, inputObjs)
+ case _ =>
+ }
+ }
+
+ val defaultBuildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit =
+ (p, outputObjs, cmdTypes, outputObjsActionType) => {
+ cmdTypes.foreach {
+ case V2CreateTablePlan =>
+ val table = invoke(p, "tableName").asInstanceOf[Identifier]
+ outputObjs += v2TablePrivileges(table)
+
+ case V2WriteCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get, actionType = outputObjsActionType)
+ }
+
+ case V2AlterTableCommand if isSparkVersionAtLeast("3.2") =>
+ val table = getFieldVal[LogicalPlan](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2DdlTableCommand =>
+ val table = getFieldVal[AnyRef](p, "child")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2PartitionCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case _ =>
+ }
+ }
+
+ case class V2Command(
+ leastVer: String = "",
+ mostVer: String = "",
+ buildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ defaultBuildInput,
+ buildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit = defaultBuildOutput,
+ operType: OperationType = QUERY,
+ cmdTypes: Seq[V2CommandType] = Seq(),
+ outputActionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER,
+ enabled: Boolean = true)
+ extends super.Val {
+
+ def buildCommand(
+ plan: LogicalPlan,
+ inputObjs: ArrayBuffer[PrivilegeObject],
+ outputObjs: ArrayBuffer[PrivilegeObject]): Unit = {
+ this.buildInput(plan, inputObjs, cmdTypes)
+ this.buildOutput(plan, outputObjs, cmdTypes, outputActionType)
+ }
+ }
+
+ def v2TablePrivileges(
+ table: Identifier,
+ columns: Seq[String] = Nil,
+ actionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER): PrivilegeObject = {
+ PrivilegeObject(TABLE_OR_VIEW, actionType, quote(table.namespace()), table.name(), columns)
+ }
+
+ // /////////////////////////////////////////////////////////////////////////////////////////////
+ // commands
+
+ val CreateNamespace: V2Command = V2Command(
+ operType = CREATEDATABASE,
+ buildOutput = (plan, outputObjs, _, _) => {
+ if (isSparkVersionAtLeast("3.3")) {
+ val resolvedNamespace = getFieldVal[Any](plan, "name")
+ val databases = getFieldVal[Seq[String]](resolvedNamespace, "nameParts")
+ outputObjs += databasePrivileges(quote(databases))
+ } else {
+ val namespace = getFieldVal[Seq[String]](plan, "namespace")
+ outputObjs += databasePrivileges(quote(namespace))
+ }
+ })
+
+ val DropNamespace: V2Command = V2Command(
+ operType = DROPDATABASE,
+ buildOutput = (plan, outputObjs, _, _) => {
+ val resolvedNamespace = getFieldVal[LogicalPlan](plan, "namespace")
+ val databases = getFieldVal[Seq[String]](resolvedNamespace, "namespace")
+ outputObjs += databasePrivileges(quote(databases))
+ })
+
+ // with V2CreateTablePlan
+
+ val CreateTable: V2Command = V2Command(
+ operType = CREATETABLE,
+ cmdTypes = Seq(V2CreateTablePlan),
+ leastVer = "3.3")
+
+ val CreateV2Table: V2Command = V2Command(
+ operType = CREATETABLE,
+ cmdTypes = Seq(V2CreateTablePlan),
+ mostVer = "3.2")
+
+ val CreateTableAsSelect: V2Command = V2Command(
+ operType = CREATETABLE,
+ cmdTypes = Seq(V2CreateTablePlan, HasQuery))
+
+ val ReplaceTable: V2Command = V2Command(
+ operType = CREATETABLE,
+ cmdTypes = Seq(V2CreateTablePlan))
+
+ val ReplaceTableAsSelect: V2Command = V2Command(
+ operType = CREATETABLE,
+ cmdTypes = Seq(V2CreateTablePlan, HasQuery))
+
+ // with V2WriteCommand
+
+ val AppendData: V2Command = V2Command(
+ cmdTypes = Seq(V2WriteCommand, HasQuery),
+ outputActionType = PrivilegeObjectActionType.INSERT)
+
+ val UpdateTable: V2Command = V2Command(
+ cmdTypes = Seq(V2WriteCommand),
+ outputActionType = PrivilegeObjectActionType.UPDATE)
+
+ val DeleteFromTable: V2Command = V2Command(
+ cmdTypes = Seq(V2WriteCommand),
+ outputActionType = PrivilegeObjectActionType.UPDATE)
+
+ val OverwriteByExpression: V2Command = V2Command(
+ cmdTypes = Seq(V2WriteCommand, HasQuery),
+ outputActionType = PrivilegeObjectActionType.UPDATE)
+
+ val OverwritePartitionsDynamic: V2Command = V2Command(
+ cmdTypes = Seq(V2WriteCommand, HasQuery),
+ outputActionType = PrivilegeObjectActionType.UPDATE)
+
+ // with V2PartitionCommand
+
+ val AddPartitions: V2Command = V2Command(
+ operType = OperationType.ALTERTABLE_ADDPARTS,
+ leastVer = "3.2",
+ cmdTypes = Seq(V2PartitionCommand))
+
+ val DropPartitions: V2Command = V2Command(
+ operType = OperationType.ALTERTABLE_DROPPARTS,
+ leastVer = "3.2",
+ cmdTypes = Seq(V2PartitionCommand))
+
+ val RenamePartitions: V2Command = V2Command(
+ operType = OperationType.ALTERTABLE_ADDPARTS,
+ leastVer = "3.2",
+ cmdTypes = Seq(V2PartitionCommand))
+
+ val TruncatePartition: V2Command = V2Command(
+ operType = OperationType.ALTERTABLE_DROPPARTS,
+ leastVer = "3.2",
+ cmdTypes = Seq(V2PartitionCommand))
+
+ // other table commands
+
+ val DropTable: V2Command = V2Command(
+ operType = DROPTABLE,
+ buildOutput = (plan, outputObjs, _, _) => {
+ val tableIdent =
+ if (isSparkVersionAtLeast("3.1")) {
+ val resolvedTable = getFieldVal[LogicalPlan](plan, "child")
+ getFieldVal[Identifier](resolvedTable, "identifier")
+ } else {
+ getFieldVal[Identifier](plan, "ident")
+ }
+ outputObjs += v2TablePrivileges(tableIdent)
+ })
+
+ val CacheTable: V2Command = V2Command(
+ operType = CREATEVIEW,
+ leastVer = "3.2",
+ buildInput = (plan, inputObjs, _) => {
+ val query = getFieldVal[LogicalPlan](plan, "table") // table to cache
+ buildQuery(query, inputObjs)
+ })
+
+ val CacheTableAsSelect: V2Command = V2Command(
+ operType = CREATEVIEW,
+ leastVer = "3.2",
+ buildInput = (plan, inputObjs, _) => {
+ val query = getFieldVal[LogicalPlan](plan, "plan")
+ buildQuery(query, inputObjs)
+ })
+
+ val CommentOnNamespace: V2Command = V2Command(
+ operType = ALTERDATABASE,
+ buildOutput = (plan, outputObjs, _, _) => {
+ val resolvedNamespace = getFieldVal[AnyRef](plan, "child")
+ val namespace = getFieldVal[Seq[String]](resolvedNamespace, "namespace")
+ outputObjs += databasePrivileges(quote(namespace))
+ })
+
+ val CommentOnTable: V2Command = V2Command(
+ operType = ALTERTABLE_PROPERTIES,
+ cmdTypes = Seq(
+ if (isSparkVersionAtLeast("3.2")) V2AlterTableCommand else V2DdlTableCommand))
+
+ val MergeIntoTable: V2Command = V2Command(
+ buildInput = (plan, inputObjs, _) => {
+ val table = getFieldVal[DataSourceV2Relation](plan, "sourceTable")
+ buildQuery(table, inputObjs)
+ },
+ buildOutput = (plan, outputObjs, _, _) => {
+ val table = getFieldVal[DataSourceV2Relation](plan, "targetTable")
+ if (table.identifier.isDefined) {
+ outputObjs += v2TablePrivileges(
+ table.identifier.get,
+ actionType = PrivilegeObjectActionType.UPDATE)
+ }
+ })
+
+ val RepairTable: V2Command = V2Command(
+ operType = ALTERTABLE_ADDPARTS,
+ leastVer = "3.2",
+ cmdTypes = Seq(V2DdlTableCommand))
+
+ val TruncateTable: V2Command = V2Command(
+ leastVer = "3.2",
+ buildOutput = (plan, outputObjs, _, _) => {
+ val table = getFieldVal[Any](plan, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent, actionType = PrivilegeObjectActionType.UPDATE)
+ })
+
+ // with V2AlterTableCommand
+
+ val AlterTable: V2Command = V2Command(
+ operType = ALTERTABLE_ADDCOLS,
+ leastVer = "3.0",
+ mostVer = "3.1",
+ buildOutput = (plan, outputObjs, _, _) => {
+ val table = getFieldVal[Any](plan, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get)
+ }
Review Comment:
changed field order of V2Command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966871617
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -93,6 +101,16 @@ object PrivilegesBuilder {
}
}
+ def mergeProjectionWithIdentifier(table: Identifier, plan: LogicalPlan): Unit = {
Review Comment:
good. much clearer.
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -273,6 +308,18 @@ object PrivilegesBuilder {
inputObjs += databasePrivileges(db.get)
}
+ case "CreateNamespace" =>
+ // for Spark 3.1 and below
+ val namespace = getFieldValOption[Seq[String]](plan, "namespace")
Review Comment:
applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969436895
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
+}
+
+object v2Commands extends Enumeration {
+
+ import scala.language.implicitConversions
+
+ implicit def valueToV2CommandBuilder(x: Value): V2Command =
+ x.asInstanceOf[V2Command]
+
+ def accept(commandName: String): Boolean = {
+ try {
+ val cmd = v2Commands.withName(commandName)
+
+ // check spark version requirements
+ def passSparkVersionCheck: Boolean =
+ (StringUtils.isBlank(cmd.mostVer) || isSparkVersionAtMost(cmd.mostVer)) &&
+ (StringUtils.isBlank(cmd.leastVer) || isSparkVersionAtLeast(cmd.leastVer))
+
+ passSparkVersionCheck && cmd.enabled
+ } catch {
+ case _: NoSuchElementException => false
+ }
+ }
+
+ val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ (p, inputObjs, cmdTypes) => {
+ cmdTypes.foreach {
+ case HasQuery =>
+ val query = getFieldVal[LogicalPlan](p, "query")
+ buildQuery(query, inputObjs)
+ case _ =>
+ }
+ }
+
+ val defaultBuildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit =
+ (p, outputObjs, cmdTypes, outputObjsActionType) => {
+ cmdTypes.foreach {
+ case V2CreateTablePlan =>
+ val table = invoke(p, "tableName").asInstanceOf[Identifier]
+ outputObjs += v2TablePrivileges(table)
+
+ case V2WriteCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get, actionType = outputObjsActionType)
+ }
+
+ case V2AlterTableCommand if isSparkVersionAtLeast("3.2") =>
+ val table = getFieldVal[LogicalPlan](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2DdlTableCommand =>
+ val table = getFieldVal[AnyRef](p, "child")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2PartitionCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case _ =>
+ }
+ }
+
+ case class V2Command(
+ leastVer: String = "",
+ mostVer: String = "",
+ buildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ defaultBuildInput,
+ buildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit = defaultBuildOutput,
+ operType: OperationType = QUERY,
+ cmdTypes: Seq[V2CommandType] = Seq(),
+ outputActionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER,
+ enabled: Boolean = true)
+ extends super.Val {
Review Comment:
We have to keep this for ` withName` of enum for looking up value by names
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966850577
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/util/AuthZUtils.scala:
##########
@@ -98,14 +109,14 @@ private[authz] object AuthZUtils {
plan.nodeName == "DataSourceV2Relation" && plan.resolved
}
- def getDatasourceV2Identifier(plan: LogicalPlan): Option[TableIdentifier] = {
- // avoid importing DataSourceV2Relation for Spark version compatibility
- val identifier = getFieldVal[Option[AnyRef]](plan, "identifier")
- identifier.map { id =>
- val namespaces = invoke(id, "namespace").asInstanceOf[Array[String]]
- val table = invoke(id, "name").asInstanceOf[String]
- TableIdentifier(table, Some(quote(namespaces)))
- }
+ def getDatasourceV2Identifier(plan: LogicalPlan): Option[Identifier] = {
+ getFieldVal[Option[Identifier]](plan, "identifier")
+ }
+
+ def getTableIdentifierFromIdentifier(id: Identifier): TableIdentifier = {
+ val namespaces = invoke(id, "namespace").asInstanceOf[Array[String]]
Review Comment:
ok, it's a legacy usage from 1.6.0. let me fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966851849
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -273,6 +308,18 @@ object PrivilegesBuilder {
inputObjs += databasePrivileges(db.get)
}
+ case "CreateNamespace" =>
Review Comment:
the command here is out of the scope of this PR, at least it does match the PR title
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -273,6 +308,18 @@ object PrivilegesBuilder {
inputObjs += databasePrivileges(db.get)
}
+ case "CreateNamespace" =>
Review Comment:
the command here is out of the scope of this PR, at least it does not match the PR title
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966555975
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -93,6 +101,22 @@ object PrivilegesBuilder {
}
}
+ def mergeProjectionWithIdentifier(
Review Comment:
As method `mergeProjectionWithIdentifier` inside method `buildQuery`, 'private' modifier is unnecessary and not allowed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for databases and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r967696272
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -273,6 +308,18 @@ object PrivilegesBuilder {
inputObjs += databasePrivileges(db.get)
}
+ case "CreateNamespace" =>
Review Comment:
Change the PR and issue with extended command list for covering both databases/namespaces and tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r970259809
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,358 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+import org.apache.kyuubi.plugin.spark.authz.v2Commands.CommandType.{CommandType, HasChildAsIdentifier, HasQueryAsLogicPlan, HasTableAsIdentifier, HasTableAsIdentifierOption, HasTableNameAsIdentifier}
+
+/**
+ * Building privilege objects
+ * for Spark Datasource V2 commands
+ */
+object v2Commands extends Enumeration {
+
+ /**
+ * Command types
+ * for hinting privileges building of inputObjs or outputObjs
+ */
+ object CommandType extends Enumeration {
+ type CommandType = Value
+ val HasChildAsIdentifier, HasQueryAsLogicPlan, HasTableAsIdentifier, HasTableAsIdentifierOption,
Review Comment:
logical plan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969246565
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
+}
+
+object v2Commands extends Enumeration {
+
+ import scala.language.implicitConversions
+
+ implicit def valueToV2CommandBuilder(x: Value): V2Command =
+ x.asInstanceOf[V2Command]
+
+ def accept(commandName: String): Boolean = {
+ try {
+ val cmd = v2Commands.withName(commandName)
+
+ // check spark version requirements
+ def passSparkVersionCheck: Boolean =
+ (StringUtils.isBlank(cmd.mostVer) || isSparkVersionAtMost(cmd.mostVer)) &&
+ (StringUtils.isBlank(cmd.leastVer) || isSparkVersionAtLeast(cmd.leastVer))
+
+ passSparkVersionCheck && cmd.enabled
+ } catch {
+ case _: NoSuchElementException => false
+ }
+ }
+
+ val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ (p, inputObjs, cmdTypes) => {
+ cmdTypes.foreach {
+ case HasQuery =>
+ val query = getFieldVal[LogicalPlan](p, "query")
+ buildQuery(query, inputObjs)
+ case _ =>
+ }
+ }
+
+ val defaultBuildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit =
+ (p, outputObjs, cmdTypes, outputObjsActionType) => {
+ cmdTypes.foreach {
+ case V2CreateTablePlan =>
+ val table = invoke(p, "tableName").asInstanceOf[Identifier]
+ outputObjs += v2TablePrivileges(table)
+
+ case V2WriteCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get, actionType = outputObjsActionType)
+ }
+
+ case V2AlterTableCommand if isSparkVersionAtLeast("3.2") =>
+ val table = getFieldVal[LogicalPlan](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2DdlTableCommand =>
+ val table = getFieldVal[AnyRef](p, "child")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2PartitionCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case _ =>
+ }
+ }
+
+ case class V2Command(
+ leastVer: String = "",
+ mostVer: String = "",
+ buildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ defaultBuildInput,
+ buildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit = defaultBuildOutput,
+ operType: OperationType = QUERY,
+ cmdTypes: Seq[V2CommandType] = Seq(),
+ outputActionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER,
+ enabled: Boolean = true)
Review Comment:
what does the parameter `enabled` mean？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969436895
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
+}
+
+object v2Commands extends Enumeration {
+
+ import scala.language.implicitConversions
+
+ implicit def valueToV2CommandBuilder(x: Value): V2Command =
+ x.asInstanceOf[V2Command]
+
+ def accept(commandName: String): Boolean = {
+ try {
+ val cmd = v2Commands.withName(commandName)
+
+ // check spark version requirements
+ def passSparkVersionCheck: Boolean =
+ (StringUtils.isBlank(cmd.mostVer) || isSparkVersionAtMost(cmd.mostVer)) &&
+ (StringUtils.isBlank(cmd.leastVer) || isSparkVersionAtLeast(cmd.leastVer))
+
+ passSparkVersionCheck && cmd.enabled
+ } catch {
+ case _: NoSuchElementException => false
+ }
+ }
+
+ val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ (p, inputObjs, cmdTypes) => {
+ cmdTypes.foreach {
+ case HasQuery =>
+ val query = getFieldVal[LogicalPlan](p, "query")
+ buildQuery(query, inputObjs)
+ case _ =>
+ }
+ }
+
+ val defaultBuildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit =
+ (p, outputObjs, cmdTypes, outputObjsActionType) => {
+ cmdTypes.foreach {
+ case V2CreateTablePlan =>
+ val table = invoke(p, "tableName").asInstanceOf[Identifier]
+ outputObjs += v2TablePrivileges(table)
+
+ case V2WriteCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get, actionType = outputObjsActionType)
+ }
+
+ case V2AlterTableCommand if isSparkVersionAtLeast("3.2") =>
+ val table = getFieldVal[LogicalPlan](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2DdlTableCommand =>
+ val table = getFieldVal[AnyRef](p, "child")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2PartitionCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case _ =>
+ }
+ }
+
+ case class V2Command(
+ leastVer: String = "",
+ mostVer: String = "",
+ buildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ defaultBuildInput,
+ buildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit = defaultBuildOutput,
+ operType: OperationType = QUERY,
+ cmdTypes: Seq[V2CommandType] = Seq(),
+ outputActionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER,
+ enabled: Boolean = true)
+ extends super.Val {
Review Comment:
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966871442
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -47,6 +48,13 @@ object PrivilegesBuilder {
PrivilegeObject(TABLE_OR_VIEW, actionType, table.database.orNull, table.table, columns)
}
+ private def tablePrivilegesWithIdentifier(
Review Comment:
good suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#issuecomment-1240409237
CI resource is tight, you can do verify locally before pushing
```
./build/mvn clean test -pl extensions/spark/kyuubi-spark-authz -am -Pspark-3.1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966845694
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -273,6 +308,18 @@ object PrivilegesBuilder {
inputObjs += databasePrivileges(db.get)
}
+ case "CreateNamespace" =>
+ // for Spark 3.1 and below
+ val namespace = getFieldValOption[Seq[String]](plan, "namespace")
Review Comment:
use the spark version as condition instead of the current way, FYI, SetCatalogAndNamespace as an example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966875708
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -335,13 +381,20 @@ object PrivilegesBuilder {
case "CreateTableAsSelect" |
"ReplaceTableAsSelect" =>
- val left = getPlanField[LogicalPlan]("name")
- left.nodeName match {
- case "ResolvedDBObjectName" =>
- val nameParts = getPlanField[Seq[String]]("nameParts")
- val db = Some(quote(nameParts.init))
- outputObjs += tablePrivileges(TableIdentifier(nameParts.last, db))
- case _ =>
+ val tableIdent = getFieldValOption[Identifier](plan, "tableName")
Review Comment:
changed to getPlanField
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969670101
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
Review Comment:
Good Suggestion. Changed all the names from Spark command traits to more abstract identifications as HasChildAsIdentifier, HasQueryAsLogicPlan, HasTableAsIdentifier, HasTableAsIdentifierOption, HasTableNameAsIdentifier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969376618
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
+}
+
+object v2Commands extends Enumeration {
+
+ import scala.language.implicitConversions
+
+ implicit def valueToV2CommandBuilder(x: Value): V2Command =
+ x.asInstanceOf[V2Command]
+
+ def accept(commandName: String): Boolean = {
+ try {
+ val cmd = v2Commands.withName(commandName)
+
+ // check spark version requirements
+ def passSparkVersionCheck: Boolean =
+ (StringUtils.isBlank(cmd.mostVer) || isSparkVersionAtMost(cmd.mostVer)) &&
+ (StringUtils.isBlank(cmd.leastVer) || isSparkVersionAtLeast(cmd.leastVer))
+
+ passSparkVersionCheck && cmd.enabled
+ } catch {
+ case _: NoSuchElementException => false
+ }
+ }
+
+ val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ (p, inputObjs, cmdTypes) => {
+ cmdTypes.foreach {
+ case HasQuery =>
+ val query = getFieldVal[LogicalPlan](p, "query")
+ buildQuery(query, inputObjs)
+ case _ =>
+ }
+ }
+
+ val defaultBuildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit =
+ (p, outputObjs, cmdTypes, outputObjsActionType) => {
+ cmdTypes.foreach {
+ case V2CreateTablePlan =>
+ val table = invoke(p, "tableName").asInstanceOf[Identifier]
+ outputObjs += v2TablePrivileges(table)
+
+ case V2WriteCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get, actionType = outputObjsActionType)
+ }
+
+ case V2AlterTableCommand if isSparkVersionAtLeast("3.2") =>
+ val table = getFieldVal[LogicalPlan](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2DdlTableCommand =>
+ val table = getFieldVal[AnyRef](p, "child")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2PartitionCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case _ =>
+ }
+ }
+
+ case class V2Command(
+ leastVer: String = "",
+ mostVer: String = "",
+ buildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ defaultBuildInput,
+ buildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit = defaultBuildOutput,
+ operType: OperationType = QUERY,
+ cmdTypes: Seq[V2CommandType] = Seq(),
+ outputActionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER,
+ enabled: Boolean = true)
+ extends super.Val {
Review Comment:
do we need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969379703
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
+}
+
+object v2Commands extends Enumeration {
+
+ import scala.language.implicitConversions
+
+ implicit def valueToV2CommandBuilder(x: Value): V2Command =
+ x.asInstanceOf[V2Command]
+
+ def accept(commandName: String): Boolean = {
+ try {
+ val cmd = v2Commands.withName(commandName)
+
+ // check spark version requirements
+ def passSparkVersionCheck: Boolean =
+ (StringUtils.isBlank(cmd.mostVer) || isSparkVersionAtMost(cmd.mostVer)) &&
+ (StringUtils.isBlank(cmd.leastVer) || isSparkVersionAtLeast(cmd.leastVer))
+
+ passSparkVersionCheck && cmd.enabled
+ } catch {
+ case _: NoSuchElementException => false
+ }
+ }
+
+ val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ (p, inputObjs, cmdTypes) => {
+ cmdTypes.foreach {
+ case HasQuery =>
+ val query = getFieldVal[LogicalPlan](p, "query")
+ buildQuery(query, inputObjs)
+ case _ =>
+ }
+ }
+
+ val defaultBuildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit =
+ (p, outputObjs, cmdTypes, outputObjsActionType) => {
+ cmdTypes.foreach {
+ case V2CreateTablePlan =>
+ val table = invoke(p, "tableName").asInstanceOf[Identifier]
+ outputObjs += v2TablePrivileges(table)
+
+ case V2WriteCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get, actionType = outputObjsActionType)
+ }
+
+ case V2AlterTableCommand if isSparkVersionAtLeast("3.2") =>
+ val table = getFieldVal[LogicalPlan](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2DdlTableCommand =>
+ val table = getFieldVal[AnyRef](p, "child")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2PartitionCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case _ =>
+ }
+ }
+
+ case class V2Command(
+ leastVer: String = "",
+ mostVer: String = "",
+ buildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ defaultBuildInput,
+ buildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit = defaultBuildOutput,
+ operType: OperationType = QUERY,
Review Comment:
nit:opType or operationType
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#issuecomment-1240928345
I have change the v2 catalog class for test suite from InMemoryTableCatalog (3.2+) to `org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog`(3.1+) and made all ut passed with leverged differnences between 3.2+ and 3.1 and below.
As for Spark 3.0, I don't have good tools for e2e test as for 3.1+. V2 catalog seems not so mature in 3.0. I am considering leaving it behind and port back 3.0 in the future..
@hellojinsilei
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966648482
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -93,6 +101,22 @@ object PrivilegesBuilder {
}
}
+ def mergeProjectionWithIdentifier(
+ table: Identifier,
+ cols: Seq[Attribute],
Review Comment:
can be removed
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -93,6 +101,22 @@ object PrivilegesBuilder {
}
}
+ def mergeProjectionWithIdentifier(
+ table: Identifier,
+ cols: Seq[Attribute],
+ plan: LogicalPlan,
+ actionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER): Unit = {
Review Comment:
can be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966752972
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -93,6 +101,22 @@ object PrivilegesBuilder {
}
}
+ def mergeProjectionWithIdentifier(
+ table: Identifier,
+ cols: Seq[Attribute],
Review Comment:
removed and reuse plan for column output listing. thx for your code sample.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966850033
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/util/AuthZUtils.scala:
##########
@@ -98,14 +109,14 @@ private[authz] object AuthZUtils {
plan.nodeName == "DataSourceV2Relation" && plan.resolved
}
- def getDatasourceV2Identifier(plan: LogicalPlan): Option[TableIdentifier] = {
- // avoid importing DataSourceV2Relation for Spark version compatibility
- val identifier = getFieldVal[Option[AnyRef]](plan, "identifier")
- identifier.map { id =>
- val namespaces = invoke(id, "namespace").asInstanceOf[Array[String]]
- val table = invoke(id, "name").asInstanceOf[String]
- TableIdentifier(table, Some(quote(namespaces)))
- }
+ def getDatasourceV2Identifier(plan: LogicalPlan): Option[Identifier] = {
+ getFieldVal[Option[Identifier]](plan, "identifier")
+ }
+
+ def getTableIdentifierFromIdentifier(id: Identifier): TableIdentifier = {
+ val namespaces = invoke(id, "namespace").asInstanceOf[Array[String]]
Review Comment:
what not call id.namespace() directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969670101
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
Review Comment:
Good Suggestion. Changed all the names from Spark command traits to more abstract identifications, as HasChildAsIdentifier, HasQueryAsLogicPlan, HasTableAsIdentifier, HasTableAsIdentifierOption, HasTableNameAsIdentifier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r970268119
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,358 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+import org.apache.kyuubi.plugin.spark.authz.v2Commands.CommandType.{CommandType, HasChildAsIdentifier, HasQueryAsLogicPlan, HasTableAsIdentifier, HasTableAsIdentifierOption, HasTableNameAsIdentifier}
+
+/**
+ * Building privilege objects
+ * for Spark Datasource V2 commands
+ */
+object v2Commands extends Enumeration {
+
+ /**
+ * Command types
+ * for hinting privileges building of inputObjs or outputObjs
+ */
Review Comment:
Fixed with added comment with naming rule as `HasFieldAsReturnType`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#issuecomment-1246549567
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966841573
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -93,6 +101,16 @@ object PrivilegesBuilder {
}
}
+ def mergeProjectionWithIdentifier(table: Identifier, plan: LogicalPlan): Unit = {
Review Comment:
how about projectV2Table and projectV1Table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969378624
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
+}
+
+object v2Commands extends Enumeration {
+
+ import scala.language.implicitConversions
+
+ implicit def valueToV2CommandBuilder(x: Value): V2Command =
+ x.asInstanceOf[V2Command]
+
+ def accept(commandName: String): Boolean = {
+ try {
+ val cmd = v2Commands.withName(commandName)
+
+ // check spark version requirements
+ def passSparkVersionCheck: Boolean =
+ (StringUtils.isBlank(cmd.mostVer) || isSparkVersionAtMost(cmd.mostVer)) &&
+ (StringUtils.isBlank(cmd.leastVer) || isSparkVersionAtLeast(cmd.leastVer))
+
+ passSparkVersionCheck && cmd.enabled
+ } catch {
+ case _: NoSuchElementException => false
+ }
+ }
+
+ val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ (p, inputObjs, cmdTypes) => {
+ cmdTypes.foreach {
+ case HasQuery =>
+ val query = getFieldVal[LogicalPlan](p, "query")
+ buildQuery(query, inputObjs)
+ case _ =>
+ }
+ }
+
+ val defaultBuildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit =
+ (p, outputObjs, cmdTypes, outputObjsActionType) => {
+ cmdTypes.foreach {
+ case V2CreateTablePlan =>
+ val table = invoke(p, "tableName").asInstanceOf[Identifier]
+ outputObjs += v2TablePrivileges(table)
+
+ case V2WriteCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get, actionType = outputObjsActionType)
+ }
+
+ case V2AlterTableCommand if isSparkVersionAtLeast("3.2") =>
+ val table = getFieldVal[LogicalPlan](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2DdlTableCommand =>
+ val table = getFieldVal[AnyRef](p, "child")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2PartitionCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case _ =>
+ }
+ }
+
+ case class V2Command(
+ leastVer: String = "",
+ mostVer: String = "",
+ buildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ defaultBuildInput,
+ buildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit = defaultBuildOutput,
+ operType: OperationType = QUERY,
+ cmdTypes: Seq[V2CommandType] = Seq(),
+ outputActionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER,
+ enabled: Boolean = true)
+ extends super.Val {
+
+ def buildCommand(
+ plan: LogicalPlan,
+ inputObjs: ArrayBuffer[PrivilegeObject],
+ outputObjs: ArrayBuffer[PrivilegeObject]): Unit = {
+ this.buildInput(plan, inputObjs, cmdTypes)
+ this.buildOutput(plan, outputObjs, cmdTypes, outputActionType)
+ }
+ }
+
+ def v2TablePrivileges(
+ table: Identifier,
+ columns: Seq[String] = Nil,
+ actionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER): PrivilegeObject = {
+ PrivilegeObject(TABLE_OR_VIEW, actionType, quote(table.namespace()), table.name(), columns)
+ }
+
+ // /////////////////////////////////////////////////////////////////////////////////////////////
+ // commands
+
+ val CreateNamespace: V2Command = V2Command(
+ operType = CREATEDATABASE,
+ buildOutput = (plan, outputObjs, _, _) => {
+ if (isSparkVersionAtLeast("3.3")) {
+ val resolvedNamespace = getFieldVal[Any](plan, "name")
+ val databases = getFieldVal[Seq[String]](resolvedNamespace, "nameParts")
+ outputObjs += databasePrivileges(quote(databases))
+ } else {
+ val namespace = getFieldVal[Seq[String]](plan, "namespace")
+ outputObjs += databasePrivileges(quote(namespace))
+ }
+ })
+
+ val DropNamespace: V2Command = V2Command(
+ operType = DROPDATABASE,
+ buildOutput = (plan, outputObjs, _, _) => {
+ val resolvedNamespace = getFieldVal[LogicalPlan](plan, "namespace")
+ val databases = getFieldVal[Seq[String]](resolvedNamespace, "namespace")
+ outputObjs += databasePrivileges(quote(databases))
+ })
+
+ // with V2CreateTablePlan
+
+ val CreateTable: V2Command = V2Command(
+ operType = CREATETABLE,
+ cmdTypes = Seq(V2CreateTablePlan),
+ leastVer = "3.3")
+
+ val CreateV2Table: V2Command = V2Command(
+ operType = CREATETABLE,
+ cmdTypes = Seq(V2CreateTablePlan),
+ mostVer = "3.2")
+
+ val CreateTableAsSelect: V2Command = V2Command(
+ operType = CREATETABLE,
+ cmdTypes = Seq(V2CreateTablePlan, HasQuery))
+
+ val ReplaceTable: V2Command = V2Command(
+ operType = CREATETABLE,
+ cmdTypes = Seq(V2CreateTablePlan))
+
+ val ReplaceTableAsSelect: V2Command = V2Command(
+ operType = CREATETABLE,
+ cmdTypes = Seq(V2CreateTablePlan, HasQuery))
+
+ // with V2WriteCommand
+
+ val AppendData: V2Command = V2Command(
+ cmdTypes = Seq(V2WriteCommand, HasQuery),
+ outputActionType = PrivilegeObjectActionType.INSERT)
+
+ val UpdateTable: V2Command = V2Command(
+ cmdTypes = Seq(V2WriteCommand),
+ outputActionType = PrivilegeObjectActionType.UPDATE)
+
+ val DeleteFromTable: V2Command = V2Command(
+ cmdTypes = Seq(V2WriteCommand),
+ outputActionType = PrivilegeObjectActionType.UPDATE)
+
+ val OverwriteByExpression: V2Command = V2Command(
+ cmdTypes = Seq(V2WriteCommand, HasQuery),
+ outputActionType = PrivilegeObjectActionType.UPDATE)
+
+ val OverwritePartitionsDynamic: V2Command = V2Command(
+ cmdTypes = Seq(V2WriteCommand, HasQuery),
+ outputActionType = PrivilegeObjectActionType.UPDATE)
+
+ // with V2PartitionCommand
+
+ val AddPartitions: V2Command = V2Command(
+ operType = OperationType.ALTERTABLE_ADDPARTS,
+ leastVer = "3.2",
+ cmdTypes = Seq(V2PartitionCommand))
+
+ val DropPartitions: V2Command = V2Command(
+ operType = OperationType.ALTERTABLE_DROPPARTS,
+ leastVer = "3.2",
+ cmdTypes = Seq(V2PartitionCommand))
+
+ val RenamePartitions: V2Command = V2Command(
+ operType = OperationType.ALTERTABLE_ADDPARTS,
+ leastVer = "3.2",
+ cmdTypes = Seq(V2PartitionCommand))
+
+ val TruncatePartition: V2Command = V2Command(
+ operType = OperationType.ALTERTABLE_DROPPARTS,
+ leastVer = "3.2",
+ cmdTypes = Seq(V2PartitionCommand))
+
+ // other table commands
+
+ val DropTable: V2Command = V2Command(
+ operType = DROPTABLE,
+ buildOutput = (plan, outputObjs, _, _) => {
+ val tableIdent =
+ if (isSparkVersionAtLeast("3.1")) {
+ val resolvedTable = getFieldVal[LogicalPlan](plan, "child")
+ getFieldVal[Identifier](resolvedTable, "identifier")
+ } else {
+ getFieldVal[Identifier](plan, "ident")
+ }
+ outputObjs += v2TablePrivileges(tableIdent)
+ })
+
+ val CacheTable: V2Command = V2Command(
+ operType = CREATEVIEW,
+ leastVer = "3.2",
+ buildInput = (plan, inputObjs, _) => {
+ val query = getFieldVal[LogicalPlan](plan, "table") // table to cache
+ buildQuery(query, inputObjs)
+ })
+
+ val CacheTableAsSelect: V2Command = V2Command(
+ operType = CREATEVIEW,
+ leastVer = "3.2",
+ buildInput = (plan, inputObjs, _) => {
+ val query = getFieldVal[LogicalPlan](plan, "plan")
+ buildQuery(query, inputObjs)
+ })
+
+ val CommentOnNamespace: V2Command = V2Command(
+ operType = ALTERDATABASE,
+ buildOutput = (plan, outputObjs, _, _) => {
+ val resolvedNamespace = getFieldVal[AnyRef](plan, "child")
+ val namespace = getFieldVal[Seq[String]](resolvedNamespace, "namespace")
+ outputObjs += databasePrivileges(quote(namespace))
+ })
+
+ val CommentOnTable: V2Command = V2Command(
+ operType = ALTERTABLE_PROPERTIES,
+ cmdTypes = Seq(
+ if (isSparkVersionAtLeast("3.2")) V2AlterTableCommand else V2DdlTableCommand))
+
+ val MergeIntoTable: V2Command = V2Command(
+ buildInput = (plan, inputObjs, _) => {
+ val table = getFieldVal[DataSourceV2Relation](plan, "sourceTable")
+ buildQuery(table, inputObjs)
+ },
+ buildOutput = (plan, outputObjs, _, _) => {
+ val table = getFieldVal[DataSourceV2Relation](plan, "targetTable")
+ if (table.identifier.isDefined) {
+ outputObjs += v2TablePrivileges(
+ table.identifier.get,
+ actionType = PrivilegeObjectActionType.UPDATE)
+ }
+ })
+
+ val RepairTable: V2Command = V2Command(
+ operType = ALTERTABLE_ADDPARTS,
+ leastVer = "3.2",
+ cmdTypes = Seq(V2DdlTableCommand))
+
+ val TruncateTable: V2Command = V2Command(
+ leastVer = "3.2",
+ buildOutput = (plan, outputObjs, _, _) => {
+ val table = getFieldVal[Any](plan, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent, actionType = PrivilegeObjectActionType.UPDATE)
+ })
+
+ // with V2AlterTableCommand
+
+ val AlterTable: V2Command = V2Command(
+ operType = ALTERTABLE_ADDCOLS,
+ leastVer = "3.0",
+ mostVer = "3.1",
+ buildOutput = (plan, outputObjs, _, _) => {
+ val table = getFieldVal[Any](plan, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get)
+ }
Review Comment:
can we adjust the order? here and other related places
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r970268258
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,358 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+import org.apache.kyuubi.plugin.spark.authz.v2Commands.CommandType.{CommandType, HasChildAsIdentifier, HasQueryAsLogicPlan, HasTableAsIdentifier, HasTableAsIdentifierOption, HasTableNameAsIdentifier}
+
+/**
+ * Building privilege objects
+ * for Spark Datasource V2 commands
+ */
+object v2Commands extends Enumeration {
+
+ /**
+ * Command types
+ * for hinting privileges building of inputObjs or outputObjs
+ */
+ object CommandType extends Enumeration {
+ type CommandType = Value
+ val HasChildAsIdentifier, HasQueryAsLogicPlan, HasTableAsIdentifier, HasTableAsIdentifierOption,
Review Comment:
fixed. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969247119
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
Review Comment:
no, V2DdlTableCommand is customed type here for building outputs for extract target table by same pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn closed pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn closed pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
URL: https://github.com/apache/incubator-kyuubi/pull/3425
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966553439
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -93,6 +101,22 @@ object PrivilegesBuilder {
}
}
+ def mergeProjectionWithIdentifier(
Review Comment:
private
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #3425: [KYUUBI #3424] [FEATURE] Support checking Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#issuecomment-1238383767
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/3425?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3425](https://codecov.io/gh/apache/incubator-kyuubi/pull/3425?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ccaf744) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/f61806283f68c97cddb86e1f404c3cc8a552c740?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f618062) will **decrease** coverage by `0.04%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #3425 +/- ##
============================================
- Coverage 51.41% 51.37% -0.05%
Complexity 13 13
============================================
Files 481 481
Lines 26699 26718 +19
Branches 3731 3735 +4
============================================
- Hits 13728 13727 -1
- Misses 11631 11650 +19
- Partials 1340 1341 +1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/3425?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../kyuubi/plugin/spark/authz/PrivilegesBuilder.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1Ymktc3BhcmstYXV0aHovc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9reXV1YmkvcGx1Z2luL3NwYXJrL2F1dGh6L1ByaXZpbGVnZXNCdWlsZGVyLnNjYWxh) | `63.52% <0.00%> (-2.33%)` | :arrow_down: |
| [...he/kyuubi/plugin/spark/authz/util/AuthZUtils.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1Ymktc3BhcmstYXV0aHovc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9reXV1YmkvcGx1Z2luL3NwYXJrL2F1dGh6L3V0aWwvQXV0aFpVdGlscy5zY2FsYQ==) | `40.00% <0.00%> (-6.52%)` | :arrow_down: |
| [...rg/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWN0bC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jdGwvY21kL2xvZy9Mb2dCYXRjaENvbW1hbmQuc2NhbGE=) | `78.00% <0.00%> (-2.00%)` | :arrow_down: |
| [...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvUHJvY0J1aWxkZXIuc2NhbGE=) | `82.60% <0.00%> (-0.63%)` | :arrow_down: |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `97.40% <0.00%> (-0.09%)` | :arrow_down: |
| [...g/apache/kyuubi/operation/BatchJobSubmission.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vQmF0Y2hKb2JTdWJtaXNzaW9uLnNjYWxh) | `74.64% <0.00%> (ø)` | |
| [...yuubi/server/metadata/jdbc/JDBCMetadataStore.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbWV0YWRhdGEvamRiYy9KREJDTWV0YWRhdGFTdG9yZS5zY2FsYQ==) | `89.96% <0.00%> (+0.69%)` | :arrow_up: |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#issuecomment-1240427700
> CI resource is tight, you can do verification locally before pushing
>
> ```
> ./build/mvn clean test -pl extensions/spark/kyuubi-spark-authz -am -Pspark-3.1
> ```
Sorry for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r970260034
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,358 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+import org.apache.kyuubi.plugin.spark.authz.v2Commands.CommandType.{CommandType, HasChildAsIdentifier, HasQueryAsLogicPlan, HasTableAsIdentifier, HasTableAsIdentifierOption, HasTableNameAsIdentifier}
+
+/**
+ * Building privilege objects
+ * for Spark Datasource V2 commands
+ */
+object v2Commands extends Enumeration {
+
+ /**
+ * Command types
+ * for hinting privileges building of inputObjs or outputObjs
+ */
Review Comment:
better to elaborate a bit of the naming rule
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969406385
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
Review Comment:
V2CreateTablePlan is directly from traits of Spark's` v2Command.scala`. The V2DdlTableCommand is a customed cmds solely in Kyuubi Authz. They also differ in detail implements for building outputObjs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966847073
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -411,6 +469,26 @@ object PrivilegesBuilder {
outputObjs += tablePrivileges(table, actionType = actionType)
buildQuery(getQuery, inputObjs)
+ case "AppendData" => // DsV2 insert
+ val table = getPlanField[AnyRef]("table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ outputObjs += tablePrivilegesWithIdentifier(tableIdent.get, actionType = INSERT)
+ buildQuery(getQuery, inputObjs)
+
+ case "UpdateTable" => // DsV2 update
+ val table = getPlanField[AnyRef]("table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ outputObjs += tablePrivilegesWithIdentifier(tableIdent.get, actionType = UPDATE)
+ // todo INSERT ?
Review Comment:
if you add TODO, better to create an issue to track. otherwise, todo is never to be done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969422241
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
+}
+
+object v2Commands extends Enumeration {
+
+ import scala.language.implicitConversions
+
+ implicit def valueToV2CommandBuilder(x: Value): V2Command =
+ x.asInstanceOf[V2Command]
+
+ def accept(commandName: String): Boolean = {
+ try {
+ val cmd = v2Commands.withName(commandName)
+
+ // check spark version requirements
+ def passSparkVersionCheck: Boolean =
+ (StringUtils.isBlank(cmd.mostVer) || isSparkVersionAtMost(cmd.mostVer)) &&
+ (StringUtils.isBlank(cmd.leastVer) || isSparkVersionAtLeast(cmd.leastVer))
+
+ passSparkVersionCheck && cmd.enabled
+ } catch {
+ case _: NoSuchElementException => false
+ }
+ }
+
+ val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ (p, inputObjs, cmdTypes) => {
+ cmdTypes.foreach {
+ case HasQuery =>
+ val query = getFieldVal[LogicalPlan](p, "query")
+ buildQuery(query, inputObjs)
+ case _ =>
+ }
+ }
+
+ val defaultBuildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit =
+ (p, outputObjs, cmdTypes, outputObjsActionType) => {
+ cmdTypes.foreach {
+ case V2CreateTablePlan =>
+ val table = invoke(p, "tableName").asInstanceOf[Identifier]
+ outputObjs += v2TablePrivileges(table)
+
+ case V2WriteCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get, actionType = outputObjsActionType)
+ }
+
+ case V2AlterTableCommand if isSparkVersionAtLeast("3.2") =>
+ val table = getFieldVal[LogicalPlan](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2DdlTableCommand =>
+ val table = getFieldVal[AnyRef](p, "child")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2PartitionCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case _ =>
+ }
+ }
+
+ case class V2Command(
+ leastVer: String = "",
Review Comment:
ok, will follow it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969375602
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
+}
+
+object v2Commands extends Enumeration {
+
+ import scala.language.implicitConversions
+
+ implicit def valueToV2CommandBuilder(x: Value): V2Command =
+ x.asInstanceOf[V2Command]
+
+ def accept(commandName: String): Boolean = {
+ try {
+ val cmd = v2Commands.withName(commandName)
+
+ // check spark version requirements
+ def passSparkVersionCheck: Boolean =
+ (StringUtils.isBlank(cmd.mostVer) || isSparkVersionAtMost(cmd.mostVer)) &&
+ (StringUtils.isBlank(cmd.leastVer) || isSparkVersionAtLeast(cmd.leastVer))
+
+ passSparkVersionCheck && cmd.enabled
+ } catch {
+ case _: NoSuchElementException => false
+ }
+ }
+
+ val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ (p, inputObjs, cmdTypes) => {
+ cmdTypes.foreach {
+ case HasQuery =>
+ val query = getFieldVal[LogicalPlan](p, "query")
+ buildQuery(query, inputObjs)
+ case _ =>
+ }
+ }
+
+ val defaultBuildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit =
+ (p, outputObjs, cmdTypes, outputObjsActionType) => {
+ cmdTypes.foreach {
+ case V2CreateTablePlan =>
+ val table = invoke(p, "tableName").asInstanceOf[Identifier]
+ outputObjs += v2TablePrivileges(table)
+
+ case V2WriteCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get, actionType = outputObjsActionType)
+ }
+
+ case V2AlterTableCommand if isSparkVersionAtLeast("3.2") =>
+ val table = getFieldVal[LogicalPlan](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2DdlTableCommand =>
+ val table = getFieldVal[AnyRef](p, "child")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2PartitionCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case _ =>
+ }
+ }
+
+ case class V2Command(
+ leastVer: String = "",
Review Comment:
use Option[String]
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
+}
+
+object v2Commands extends Enumeration {
+
+ import scala.language.implicitConversions
+
+ implicit def valueToV2CommandBuilder(x: Value): V2Command =
+ x.asInstanceOf[V2Command]
+
+ def accept(commandName: String): Boolean = {
+ try {
+ val cmd = v2Commands.withName(commandName)
+
+ // check spark version requirements
+ def passSparkVersionCheck: Boolean =
+ (StringUtils.isBlank(cmd.mostVer) || isSparkVersionAtMost(cmd.mostVer)) &&
+ (StringUtils.isBlank(cmd.leastVer) || isSparkVersionAtLeast(cmd.leastVer))
+
+ passSparkVersionCheck && cmd.enabled
+ } catch {
+ case _: NoSuchElementException => false
+ }
+ }
+
+ val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ (p, inputObjs, cmdTypes) => {
+ cmdTypes.foreach {
+ case HasQuery =>
+ val query = getFieldVal[LogicalPlan](p, "query")
+ buildQuery(query, inputObjs)
+ case _ =>
+ }
+ }
+
+ val defaultBuildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit =
+ (p, outputObjs, cmdTypes, outputObjsActionType) => {
+ cmdTypes.foreach {
+ case V2CreateTablePlan =>
+ val table = invoke(p, "tableName").asInstanceOf[Identifier]
+ outputObjs += v2TablePrivileges(table)
+
+ case V2WriteCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get, actionType = outputObjsActionType)
+ }
+
+ case V2AlterTableCommand if isSparkVersionAtLeast("3.2") =>
+ val table = getFieldVal[LogicalPlan](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2DdlTableCommand =>
+ val table = getFieldVal[AnyRef](p, "child")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2PartitionCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case _ =>
+ }
+ }
+
+ case class V2Command(
+ leastVer: String = "",
+ mostVer: String = "",
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969381696
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
+}
+
+object v2Commands extends Enumeration {
+
+ import scala.language.implicitConversions
+
+ implicit def valueToV2CommandBuilder(x: Value): V2Command =
+ x.asInstanceOf[V2Command]
+
+ def accept(commandName: String): Boolean = {
+ try {
+ val cmd = v2Commands.withName(commandName)
+
+ // check spark version requirements
+ def passSparkVersionCheck: Boolean =
+ (StringUtils.isBlank(cmd.mostVer) || isSparkVersionAtMost(cmd.mostVer)) &&
+ (StringUtils.isBlank(cmd.leastVer) || isSparkVersionAtLeast(cmd.leastVer))
+
+ passSparkVersionCheck && cmd.enabled
+ } catch {
+ case _: NoSuchElementException => false
+ }
+ }
+
+ val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ (p, inputObjs, cmdTypes) => {
+ cmdTypes.foreach {
+ case HasQuery =>
+ val query = getFieldVal[LogicalPlan](p, "query")
+ buildQuery(query, inputObjs)
+ case _ =>
+ }
+ }
+
+ val defaultBuildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit =
+ (p, outputObjs, cmdTypes, outputObjsActionType) => {
+ cmdTypes.foreach {
+ case V2CreateTablePlan =>
+ val table = invoke(p, "tableName").asInstanceOf[Identifier]
+ outputObjs += v2TablePrivileges(table)
+
+ case V2WriteCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get, actionType = outputObjsActionType)
+ }
+
+ case V2AlterTableCommand if isSparkVersionAtLeast("3.2") =>
+ val table = getFieldVal[LogicalPlan](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2DdlTableCommand =>
+ val table = getFieldVal[AnyRef](p, "child")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2PartitionCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case _ =>
+ }
+ }
+
+ case class V2Command(
Review Comment:
CommandConverter/CommandDetail/CommandInfo etc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969435244
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
+}
+
+object v2Commands extends Enumeration {
+
+ import scala.language.implicitConversions
+
+ implicit def valueToV2CommandBuilder(x: Value): V2Command =
+ x.asInstanceOf[V2Command]
+
+ def accept(commandName: String): Boolean = {
+ try {
+ val cmd = v2Commands.withName(commandName)
+
+ // check spark version requirements
+ def passSparkVersionCheck: Boolean =
+ (StringUtils.isBlank(cmd.mostVer) || isSparkVersionAtMost(cmd.mostVer)) &&
+ (StringUtils.isBlank(cmd.leastVer) || isSparkVersionAtLeast(cmd.leastVer))
+
+ passSparkVersionCheck && cmd.enabled
+ } catch {
+ case _: NoSuchElementException => false
+ }
+ }
+
+ val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ (p, inputObjs, cmdTypes) => {
+ cmdTypes.foreach {
+ case HasQuery =>
+ val query = getFieldVal[LogicalPlan](p, "query")
+ buildQuery(query, inputObjs)
+ case _ =>
+ }
+ }
+
+ val defaultBuildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit =
+ (p, outputObjs, cmdTypes, outputObjsActionType) => {
+ cmdTypes.foreach {
+ case V2CreateTablePlan =>
+ val table = invoke(p, "tableName").asInstanceOf[Identifier]
+ outputObjs += v2TablePrivileges(table)
+
+ case V2WriteCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get, actionType = outputObjsActionType)
+ }
+
+ case V2AlterTableCommand if isSparkVersionAtLeast("3.2") =>
+ val table = getFieldVal[LogicalPlan](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2DdlTableCommand =>
+ val table = getFieldVal[AnyRef](p, "child")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2PartitionCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case _ =>
+ }
+ }
+
+ case class V2Command(
+ leastVer: String = "",
+ mostVer: String = "",
+ buildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ defaultBuildInput,
+ buildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit = defaultBuildOutput,
+ operType: OperationType = QUERY,
Review Comment:
ok, applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969670609
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
+}
+
+object v2Commands extends Enumeration {
+
+ import scala.language.implicitConversions
+
+ implicit def valueToV2CommandBuilder(x: Value): V2Command =
+ x.asInstanceOf[V2Command]
+
+ def accept(commandName: String): Boolean = {
+ try {
+ val cmd = v2Commands.withName(commandName)
+
+ // check spark version requirements
+ def passSparkVersionCheck: Boolean =
+ (StringUtils.isBlank(cmd.mostVer) || isSparkVersionAtMost(cmd.mostVer)) &&
+ (StringUtils.isBlank(cmd.leastVer) || isSparkVersionAtLeast(cmd.leastVer))
+
+ passSparkVersionCheck && cmd.enabled
+ } catch {
+ case _: NoSuchElementException => false
+ }
+ }
+
+ val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ (p, inputObjs, cmdTypes) => {
+ cmdTypes.foreach {
+ case HasQuery =>
+ val query = getFieldVal[LogicalPlan](p, "query")
+ buildQuery(query, inputObjs)
+ case _ =>
+ }
+ }
+
+ val defaultBuildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit =
+ (p, outputObjs, cmdTypes, outputObjsActionType) => {
+ cmdTypes.foreach {
+ case V2CreateTablePlan =>
+ val table = invoke(p, "tableName").asInstanceOf[Identifier]
+ outputObjs += v2TablePrivileges(table)
+
+ case V2WriteCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get, actionType = outputObjsActionType)
+ }
+
+ case V2AlterTableCommand if isSparkVersionAtLeast("3.2") =>
+ val table = getFieldVal[LogicalPlan](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2DdlTableCommand =>
+ val table = getFieldVal[AnyRef](p, "child")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2PartitionCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case _ =>
+ }
+ }
+
+ case class V2Command(
Review Comment:
changed to `CmdPrivilegeBuilder`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966704962
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -93,6 +101,22 @@ object PrivilegesBuilder {
}
}
+ def mergeProjectionWithIdentifier(
+ table: Identifier,
+ cols: Seq[Attribute],
+ plan: LogicalPlan,
+ actionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER): Unit = {
Review Comment:
removed, thx.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966846176
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -335,13 +381,20 @@ object PrivilegesBuilder {
case "CreateTableAsSelect" |
"ReplaceTableAsSelect" =>
- val left = getPlanField[LogicalPlan]("name")
- left.nodeName match {
- case "ResolvedDBObjectName" =>
- val nameParts = getPlanField[Seq[String]]("nameParts")
- val db = Some(quote(nameParts.init))
- outputObjs += tablePrivileges(TableIdentifier(nameParts.last, db))
- case _ =>
+ val tableIdent = getFieldValOption[Identifier](plan, "tableName")
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966855967
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -411,6 +469,26 @@ object PrivilegesBuilder {
outputObjs += tablePrivileges(table, actionType = actionType)
buildQuery(getQuery, inputObjs)
+ case "AppendData" => // DsV2 insert
Review Comment:
I prefer group v2commands in set or enum , but not to introduce new method for handling them. Stay in one method is good enough for now.
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -411,6 +469,26 @@ object PrivilegesBuilder {
outputObjs += tablePrivileges(table, actionType = actionType)
buildQuery(getQuery, inputObjs)
+ case "AppendData" => // DsV2 insert
Review Comment:
I prefer groupping v2commands in set or enum , but not to introduce new method for handling them. Stay in one method is good enough for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Support checking table Access privilege for DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r966570675
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -93,6 +101,22 @@ object PrivilegesBuilder {
}
}
+ def mergeProjectionWithIdentifier(
Review Comment:
sorry, my bad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969236740
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
Review Comment:
is V2CreateTablePlan a type of V2DdlTableCommand
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for databases and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#issuecomment-1242796814
With completely refractored v2 commands handling by dedicate privilige building handlers.
1. introduced command privilege handling framework/pattern for incoming changing commands from Spark 3.x+ or other Engine / Plugins, eg. Arctics.
2. resue possible building process and added support for least/most spark version , the command type feature grouping commands into `V2CreateTablePlan/V2CreateTablePlan/V2AlterTableCommand` for output checks and `HasQuery` for subquery input checks.
3. prevent unbound command name in string and group in command enums.
4. introducing minimum changes into PrivilegeBuilder as possible, by extended dedicate comand list and builders.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969408031
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
+
+ // custom types
+ val HasQuery, V2DdlTableCommand = Value
+}
+
+object v2Commands extends Enumeration {
+
+ import scala.language.implicitConversions
+
+ implicit def valueToV2CommandBuilder(x: Value): V2Command =
+ x.asInstanceOf[V2Command]
+
+ def accept(commandName: String): Boolean = {
+ try {
+ val cmd = v2Commands.withName(commandName)
+
+ // check spark version requirements
+ def passSparkVersionCheck: Boolean =
+ (StringUtils.isBlank(cmd.mostVer) || isSparkVersionAtMost(cmd.mostVer)) &&
+ (StringUtils.isBlank(cmd.leastVer) || isSparkVersionAtLeast(cmd.leastVer))
+
+ passSparkVersionCheck && cmd.enabled
+ } catch {
+ case _: NoSuchElementException => false
+ }
+ }
+
+ val defaultBuildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ (p, inputObjs, cmdTypes) => {
+ cmdTypes.foreach {
+ case HasQuery =>
+ val query = getFieldVal[LogicalPlan](p, "query")
+ buildQuery(query, inputObjs)
+ case _ =>
+ }
+ }
+
+ val defaultBuildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit =
+ (p, outputObjs, cmdTypes, outputObjsActionType) => {
+ cmdTypes.foreach {
+ case V2CreateTablePlan =>
+ val table = invoke(p, "tableName").asInstanceOf[Identifier]
+ outputObjs += v2TablePrivileges(table)
+
+ case V2WriteCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Option[Identifier]](table, "identifier")
+ if (tableIdent.isDefined) {
+ outputObjs += v2TablePrivileges(tableIdent.get, actionType = outputObjsActionType)
+ }
+
+ case V2AlterTableCommand if isSparkVersionAtLeast("3.2") =>
+ val table = getFieldVal[LogicalPlan](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2DdlTableCommand =>
+ val table = getFieldVal[AnyRef](p, "child")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case V2PartitionCommand =>
+ val table = getFieldVal[AnyRef](p, "table")
+ val tableIdent = getFieldVal[Identifier](table, "identifier")
+ outputObjs += v2TablePrivileges(tableIdent)
+
+ case _ =>
+ }
+ }
+
+ case class V2Command(
+ leastVer: String = "",
+ mostVer: String = "",
+ buildInput: (LogicalPlan, ArrayBuffer[PrivilegeObject], Seq[V2CommandType]) => Unit =
+ defaultBuildInput,
+ buildOutput: (
+ LogicalPlan,
+ ArrayBuffer[PrivilegeObject],
+ Seq[V2CommandType],
+ PrivilegeObjectActionType) => Unit = defaultBuildOutput,
+ operType: OperationType = QUERY,
+ cmdTypes: Seq[V2CommandType] = Seq(),
+ outputActionType: PrivilegeObjectActionType = PrivilegeObjectActionType.OTHER,
+ enabled: Boolean = true)
Review Comment:
It's meant to control whether enabled this command mapping. Handy for listing unimplemented commands like CreateView and etx. Also handy for debug and customized commands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3425: [KYUUBI #3424] [FEATURE] [AUTHZ] Access privilege checks for namespaces and tables of DatasourceV2
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3425:
URL: https://github.com/apache/incubator-kyuubi/pull/3425#discussion_r969516327
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/v2Commands.scala:
##########
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.connector.catalog.Identifier
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+import org.apache.kyuubi.plugin.spark.authz.OperationType._
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectActionType.PrivilegeObjectActionType
+import org.apache.kyuubi.plugin.spark.authz.PrivilegeObjectType.TABLE_OR_VIEW
+import org.apache.kyuubi.plugin.spark.authz.PrivilegesBuilder._
+import org.apache.kyuubi.plugin.spark.authz.V2CommandType.{HasQuery, V2AlterTableCommand, V2CommandType, V2CreateTablePlan, V2DdlTableCommand, V2PartitionCommand, V2WriteCommand}
+import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils._
+
+object V2CommandType extends Enumeration {
+ type V2CommandType = Value
+
+ // traits' name from Spark v2commands and v2AlterTableCommands
+ val V2CreateTablePlan, V2WriteCommand, V2AlterTableCommand, V2PartitionCommand = Value
Review Comment:
How about naming them all like HasQuery？ HasQueryField, HasTableNameAs... etc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org