You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ya...@apache.org on 2022/09/30 07:44:08 UTC
[incubator-kyuubi] branch master updated: [KYUUBI #3545][KYUUBI #3563] Support restrict spark configurations
This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new bb50c52c2 [KYUUBI #3545][KYUUBI #3563] Support restrict spark configurations
bb50c52c2 is described below
commit bb50c52c2f6f2068dbd9897cc04280b63b299f2f
Author: Fu Chen <cf...@gmail.com>
AuthorDate: Fri Sep 30 15:43:58 2022 +0800
[KYUUBI #3545][KYUUBI #3563] Support restrict spark configurations
### _Why are the changes needed?_
ban end-user from security settings
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #3564 from cfmcgrady/kyuubi-3563.
Closes #3545
Closes #3563
9d912b11 [Fu Chen] rename
ee44f7df [Fu Chen] fix ci
158f1552 [Fu Chen] address comment
3cbf4794 [Fu Chen] doc
0125a862 [Fu Chen] address comment
e4e554f5 [Kent Yao] Update extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AuthzConfigurationCheckExtension.scala
a8d35efa [Fu Chen] ban end-user from security settings
Lead-authored-by: Fu Chen <cf...@gmail.com>
Co-authored-by: Kent Yao <ya...@apache.org>
Signed-off-by: Kent Yao <ya...@apache.org>
---
docs/security/authorization/spark/overview.rst | 53 ++++++++++++++++++--
.../authz/ranger/AuthzConfigurationChecker.scala | 46 ++++++++++++++++++
.../spark/authz/ranger/RangerSparkExtension.scala | 1 +
.../ranger/AuthzConfigurationCheckerSuite.scala | 56 ++++++++++++++++++++++
4 files changed, 153 insertions(+), 3 deletions(-)
diff --git a/docs/security/authorization/spark/overview.rst b/docs/security/authorization/spark/overview.rst
index 109f7301f..fcbaa880b 100644
--- a/docs/security/authorization/spark/overview.rst
+++ b/docs/security/authorization/spark/overview.rst
@@ -28,7 +28,7 @@ Authorization in Kyuubi
-----------------------
Storage-based Authorization
-***************************
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
As Kyuubi supports multi tenancy, a tenant can only visit authorized resources,
including computing resources, data, etc.
@@ -41,7 +41,7 @@ as well as their permissions.
Storage-based authorization offers users with database, table and partition-level coarse-gained access control.
SQL-standard authorization with Ranger
-**************************************
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A SQL-standard authorization usually offers a row/colum-level fine-grained access control to meet the real-world data security need.
@@ -59,4 +59,51 @@ Kyuubi Spark Authz Plugin itself provides general purpose for ACL management for
It is not necessary to deploy it with the Kyuubi server and engine, and can be used as an extension for any Spark SQL jobs.
However, the authorization always requires a robust authentication layer and multi tenancy support, so Kyuubi is a perfect match.
-.. _Apache Ranger: https://ranger.apache.org/
\ No newline at end of file
+Restrict security configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+End-users can disable the AuthZ plugin by modifying Spark's configuration. For example:
+
+
+.. code-block:: sql
+
+ select * from parquet.`/path/to/table`
+
+.. code-block:: sql
+
+ set spark.sql.optimizer.excludedRules=org.apache.kyuubi.plugin.spark.authz.ranger.RuleAuthorization
+
+Kyuubi provides a mechanism to ban security configurations to enhance the security of production environments
+
+.. note:: How do we modify the Spark engine configurations please refer to the documentation `Spark Configurations`_
+
+
+
+Restrict session level config
+*****************************
+
+You can specify config `kyuubi.session.conf.ignore.list` values and config `kyuubi.session.conf.restrict.list` values to disable changing session+ level configuration on the server side. For example:
+
+.. code-block::
+
+ kyuubi.session.conf.ignore.list spark.driver.memory,spark.sql.optimizer.excludedRules
+
+.. code-block::
+
+ kyuubi.session.conf.restrict.list spark.driver.memory,spark.sql.optimizer.excludedRules
+
+Restrict operation level config
+*******************************
+
+You can specify config `spark.kyuubi.conf.restricted.list` values to disable changing operation level configuration on the engine side, this means that the config key in the restricted list cannot set dynamic configuration via SET syntax. For examples:
+
+.. code-block::
+
+ spark.kyuubi.conf.restricted.list spark.sql.adaptive.enabled,spark.sql.adaptive.skewJoin.enabled
+
+.. note::
+ 1. Note that config `spark.sql.runSQLOnFiles` values and config `spark.sql.extensions` values are by default in the engine restriction configuration list
+ 2. A set statement with key equal to `spark.sql.optimizer.excludedRules` and value containing `org.apache.kyuubi.plugin.spark.authz.ranger.*` also does not allow modification.
+
+.. _Apache Ranger: https://ranger.apache.org/
+.. _Spark Configurations: ../../../deployment/settings.html#spark-configurations
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AuthzConfigurationChecker.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AuthzConfigurationChecker.scala
new file mode 100644
index 000000000..56ab27d22
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AuthzConfigurationChecker.scala
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.ranger
+
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.command.SetCommand
+
+import org.apache.kyuubi.plugin.spark.authz.AccessControlException
+
+/**
+ * For banning end-users from set restricted spark configurations
+ */
+case class AuthzConfigurationChecker(spark: SparkSession) extends (LogicalPlan => Unit) {
+
+ final val RESTRICT_LIST_KEY = "spark.kyuubi.conf.restricted.list"
+
+ private val restrictedConfList: Set[String] =
+ Set(RESTRICT_LIST_KEY, "spark.sql.runSQLOnFiles", "spark.sql.extensions") ++
+ spark.conf.getOption(RESTRICT_LIST_KEY).map(_.split(',').toSet).getOrElse(Set.empty)
+
+ override def apply(plan: LogicalPlan): Unit = plan match {
+ case SetCommand(Some((
+ "spark.sql.optimizer.excludedRules",
+ Some(v)))) if v.contains("org.apache.kyuubi.plugin.spark.authz.ranger") =>
+ throw new AccessControlException("Excluding Authz security rules is not allowed")
+ case SetCommand(Some((k, Some(_)))) if restrictedConfList.contains(k) =>
+ throw new AccessControlException(s"Modifying config $k is not allowed")
+ case _ =>
+ }
+}
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RangerSparkExtension.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RangerSparkExtension.scala
index 3676dfb32..f4dcb3f9f 100644
--- a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RangerSparkExtension.scala
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RangerSparkExtension.scala
@@ -39,6 +39,7 @@ class RangerSparkExtension extends (SparkSessionExtensions => Unit) {
SparkRangerAdminPlugin.init()
override def apply(v1: SparkSessionExtensions): Unit = {
+ v1.injectCheckRule(AuthzConfigurationChecker)
v1.injectResolutionRule(_ => new RuleReplaceShowObjectCommands())
v1.injectResolutionRule(_ => new RuleApplyPermanentViewMarker())
v1.injectResolutionRule(new RuleApplyRowFilterAndDataMasking(_))
diff --git a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AuthzConfigurationCheckerSuite.scala b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AuthzConfigurationCheckerSuite.scala
new file mode 100644
index 000000000..cd5757e54
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AuthzConfigurationCheckerSuite.scala
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.spark.authz.ranger
+
+import org.scalatest.BeforeAndAfterAll
+// scalastyle:off
+import org.scalatest.funsuite.AnyFunSuite
+
+import org.apache.kyuubi.plugin.spark.authz.{AccessControlException, SparkSessionProvider}
+
+class AuthzConfigurationCheckerSuite extends AnyFunSuite with SparkSessionProvider
+ with BeforeAndAfterAll {
+
+ override protected val catalogImpl: String = "in-memory"
+ override def afterAll(): Unit = {
+ spark.stop()
+ super.afterAll()
+ }
+
+ test("apply spark configuration restriction rules") {
+ sql("set spark.kyuubi.conf.restricted.list=spark.sql.abc,spark.sql.xyz")
+ val extension = AuthzConfigurationChecker(spark)
+ val p1 = sql("set spark.sql.runSQLOnFiles=true").queryExecution.analyzed
+ intercept[AccessControlException](extension.apply(p1))
+ val p2 = sql("set spark.sql.runSQLOnFiles=false").queryExecution.analyzed
+ intercept[AccessControlException](extension.apply(p2))
+ val p3 = sql("set spark.sql.runSQLOnFiles").queryExecution.analyzed
+ extension.apply(p3)
+ val p4 = sql("set spark.sql.abc=xyz").queryExecution.analyzed
+ intercept[AccessControlException](extension.apply(p4))
+ val p5 = sql("set spark.sql.xyz=abc").queryExecution.analyzed
+ intercept[AccessControlException](extension.apply(p5))
+ val p6 = sql("set spark.kyuubi.conf.restricted.list=123").queryExecution.analyzed
+ intercept[AccessControlException](extension.apply(p6))
+ val p7 = sql("set spark.sql.efg=hijk").queryExecution.analyzed
+ extension.apply(p7)
+ val p8 = sql(
+ s"set spark.sql.optimizer.excludedRules=${classOf[RuleAuthorization].getName}").queryExecution.analyzed
+ intercept[AccessControlException](extension.apply(p8))
+ }
+}