You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2026/09/08 23:58:14 UTC
(spark) branch branch-4.x updated: [SPARK-59319][SQL] Add an opt-in restricted SQL execution mode
This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new a336c052dd54 [SPARK-59319][SQL] Add an opt-in restricted SQL execution mode
a336c052dd54 is described below
commit a336c052dd54a25e2c09879d9ef9d33c985f55c5
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Wed Sep 9 08:57:45 2026 +0900
[SPARK-59319][SQL] Add an opt-in restricted SQL execution mode
### What changes were proposed in this pull request?
Add `spark.sql.restrictedMode.enabled` (a static, internal config, default `false` = unchanged).
When enabled, the `reflect` / `java_method` / `try_reflect` functions and the `TRANSFORM ... USING`
clause are rejected during analysis with a new `UNSUPPORTED_FEATURE.SQL_RESTRICTED_MODE` error
condition.
The check walks the whole analyzed plan -- including already-analyzed sub-plans reused from a
temporary view or a cached Dataset, the bodies of analysis-only commands (CTAS,
`CACHE TABLE ... AS SELECT`, `CREATE`/`ALTER VIEW`) which move into `innerChildren` once analyzed,
and subquery plans -- so the features cannot slip through. It runs before `checkAnalysis0` by
design, so a restricted-mode rejection is reported ahead of ordinary analysis errors. Because it is
a static config, a session cannot turn it off for itself.
This PR does not touch `ADD JAR` / JAR-backed `CREATE FUNCTION` or the class-loading / URL JDBC
data-source options; gating those is left as a follow-up.
### Why are the changes needed?
Provides an opt-in mode that disables dynamic code and external-script execution features in SQL
for deployments that want a more constrained SQL surface. Off by default, so there is no behavior
change unless enabled.
### Does this PR introduce _any_ user-facing change?
No by default. When the mode is enabled, the listed features are rejected with a clear error.
### How was this patch tested?
New catalyst `RestrictedModeSuite`: `reflect` / `java_method` / `try_reflect` and `TRANSFORM ...
USING` are rejected when the mode is on and allowed when off (asserted with `checkError` on the
error condition and its parameters), and the rejection still fires for an analyzed sub-plan reused
under a fresh parent (the shape `checkAnalysis0` would skip).
New `sql/core` `RestrictedModeCommandSuite` (needs a session): `reflect` inside a `CREATE VIEW` body
and `TRANSFORM` inside a `CACHE TABLE ... AS SELECT` body are rejected (covering the analysis-only
command `innerChildren` path), and `SET spark.sql.restrictedMode.enabled=false` fails with
`CANNOT_MODIFY_STATIC_CONFIG`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
This pull request and its description were written by Isaac.
Co-authored-by: Isaac <no-replydatabricks.com>
Closes #58604 from HyukjinKwon/SPARK-59319-restricted-sql-mode.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
(cherry picked from commit dad9ee82213ff062b32cb5ca550755089339cd48)
Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
---
.../src/main/resources/error/error-conditions.json | 5 +
.../sql/catalyst/analysis/CheckAnalysis.scala | 42 ++++++++
.../spark/sql/errors/QueryCompilationErrors.scala | 9 ++
.../org/apache/spark/sql/internal/SQLConf.scala | 2 +
.../apache/spark/sql/internal/StaticSQLConf.scala | 13 +++
.../catalyst/analysis/RestrictedModeSuite.scala | 116 +++++++++++++++++++++
.../sql/execution/RestrictedModeCommandSuite.scala | 68 ++++++++++++
7 files changed, 255 insertions(+)
diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json
index 51965e15f663..d9c1795627f5 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -9241,6 +9241,11 @@
"SQL cursor operations (DECLARE CURSOR, OPEN, FETCH, CLOSE) are not supported."
]
},
+ "SQL_RESTRICTED_MODE" : {
+ "message" : [
+ "<feature> is not allowed because <config> is enabled."
+ ]
+ },
"SQL_SCRIPTING" : {
"message" : [
"SQL Scripting is under development and not all features are supported. SQL Scripting enables users to write procedural SQL including control flow and error handling. To enable existing features set <sqlScriptingEnabled> to `true`."
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
index 5c8e34bf8d69..685d5e1123ce 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
@@ -328,6 +328,9 @@ trait CheckAnalysis extends LookupCatalog with QueryErrorsBase with PlanToString
}
preemptedError.clear()
try {
+ if (SQLConf.get.restrictedModeEnabled) {
+ checkRestrictedMode(inlinedPlan)
+ }
checkAnalysis0(inlinedPlan)
preemptedError.getErrorOpt().foreach(throw _) // throw preempted error if any
} catch {
@@ -339,6 +342,45 @@ trait CheckAnalysis extends LookupCatalog with QueryErrorsBase with PlanToString
plan.setAnalyzed()
}
+ /**
+ * Rejects SQL features that load or execute externally provided code or scripts when the
+ * restricted execution mode is enabled. Unlike `checkAnalysis0`, which skips already-analyzed
+ * sub-plans (`case p if p.analyzed`), this walks the whole plan -- including analyzed sub-plans
+ * reused from a temporary view or a cached Dataset, and the bodies of analysis-only commands
+ * (CTAS, `CACHE TABLE ... AS SELECT`, `CREATE`/`ALTER VIEW`) which move into `innerChildren`
+ * once analyzed -- and descends into subquery plans, so these features cannot slip through.
+ */
+ private def checkRestrictedMode(plan: LogicalPlan): Unit = {
+ def checkExpression(expr: Expression): Unit = expr.foreach {
+ // A try_reflect call stays a RuntimeReplaceable wrapper until optimization, so match it
+ // before its CallMethodViaReflection replacement to report the right function name.
+ case t: TryReflect =>
+ throw QueryCompilationErrors.restrictedModeFeatureError(
+ s"The ${toSQLId(t.prettyName)} function")
+ case c: CallMethodViaReflection =>
+ throw QueryCompilationErrors.restrictedModeFeatureError(
+ s"The ${toSQLId(c.prettyName)} function")
+ case s: SubqueryExpression =>
+ checkPlan(s.plan)
+ case _ =>
+ }
+ def checkPlan(p: LogicalPlan): Unit = p.foreach {
+ case _: ScriptTransformation =>
+ throw QueryCompilationErrors.restrictedModeFeatureError(
+ "The TRANSFORM ... USING clause")
+ case node =>
+ node.expressions.foreach(checkExpression)
+ // The body of an analysis-only command (CTAS, `CACHE TABLE ... AS SELECT`,
+ // `CREATE`/`ALTER VIEW`) moves from `children` into `innerChildren` once the command is
+ // analyzed, so `foreach` (which follows `children`) would not otherwise reach it.
+ node.innerChildren.foreach {
+ case inner: LogicalPlan => checkPlan(inner)
+ case _ =>
+ }
+ }
+ checkPlan(plan)
+ }
+
def checkAnalysis0(plan: LogicalPlan): Unit = {
// The target table is not a child plan of the insert command. We should report errors for table
// not found first, instead of errors in the input query of the insert command, by doing a
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
index 309ab1598752..a4e8963b607f 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
@@ -39,6 +39,7 @@ import org.apache.spark.sql.connector.catalog.functions.{BoundFunction, UnboundF
import org.apache.spark.sql.connector.expressions.filter.Predicate
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.LEGACY_CTE_PRECEDENCE_POLICY
+import org.apache.spark.sql.internal.StaticSQLConf
import org.apache.spark.sql.sources.Filter
import org.apache.spark.sql.streaming.OutputMode
import org.apache.spark.sql.types._
@@ -329,6 +330,14 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat
messageParameters = Map.empty)
}
+ def restrictedModeFeatureError(feature: String): Throwable = {
+ new AnalysisException(
+ errorClass = "UNSUPPORTED_FEATURE.SQL_RESTRICTED_MODE",
+ messageParameters = Map(
+ "feature" -> feature,
+ "config" -> toSQLConf(StaticSQLConf.RESTRICTED_MODE_ENABLED.key)))
+ }
+
def binByRequiresTopLevelColumnError(reference: Expression): Throwable = {
new AnalysisException(
errorClass = "BIN_BY_REQUIRES_TOP_LEVEL_COLUMN",
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 642d06087ae6..2238ad9d0c6a 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -9363,6 +9363,8 @@ class SQLConf extends Serializable with Logging with SqlApiConf {
def crossJoinEnabled: Boolean = getConf(SQLConf.CROSS_JOINS_ENABLED)
+ def restrictedModeEnabled: Boolean = getConf(StaticSQLConf.RESTRICTED_MODE_ENABLED)
+
override def sessionLocalTimeZone: String = getConf(SQLConf.SESSION_LOCAL_TIMEZONE)
def jsonGeneratorIgnoreNullFields: Boolean = getConf(SQLConf.JSON_GENERATOR_IGNORE_NULL_FIELDS)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
index 4903c6a59de1..43dad9446c42 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
@@ -377,6 +377,19 @@ object StaticSQLConf {
"Every entry must be a valid regular expression.")
.createWithDefault(Nil)
+ val RESTRICTED_MODE_ENABLED = buildStaticConf("spark.sql.restrictedMode.enabled")
+ .internal()
+ .doc("When true, SQL features that load or execute externally provided code or scripts from " +
+ "the query itself are disabled: the reflect, java_method and try_reflect functions and the " +
+ "TRANSFORM ... USING clause are rejected during analysis. This is an opt-in profile for " +
+ "deployments that want a more constrained SQL surface. As a static configuration it can " +
+ "only be set when starting the driver, and not from a session, so that a session cannot " +
+ "turn it off for itself. The default of false preserves the previous behavior.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(false)
+
// Bounds on the environment a session may install in its Python workers through the reserved
// `spark.pythonWorkerEnv.` prefix. Static, so a session cannot raise its own limits. Their keys
// are deliberately not under that prefix: every SparkConf entry is copied into a new session's
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/RestrictedModeSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/RestrictedModeSuite.scala
new file mode 100644
index 000000000000..f9133e546a5a
--- /dev/null
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/RestrictedModeSuite.scala
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Literal}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project, ScriptInputOutputSchema, ScriptTransformation}
+import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf}
+import org.apache.spark.sql.types.StringType
+
+class RestrictedModeSuite extends AnalysisTest {
+
+ private val restricted = StaticSQLConf.RESTRICTED_MODE_ENABLED.key
+ // The config appears in the error message double-quoted, as `toSQLConf` renders it.
+ private val configName = "\"" + restricted + "\""
+
+ // The mode is a static config, so `withSQLConf` (which rejects static keys) cannot toggle it.
+ // Set it directly on the active conf instead, restoring the previous value afterwards.
+ private def withRestrictedMode[T](enabled: Boolean)(f: => T): T = {
+ val conf = SQLConf.get
+ val previous = if (conf.contains(restricted)) Some(conf.getConfString(restricted)) else None
+ conf.setConfString(restricted, enabled.toString)
+ try f finally {
+ previous match {
+ case Some(value) => conf.setConfString(restricted, value)
+ case None => conf.unsetConf(restricted)
+ }
+ }
+ }
+
+ private def functionProject(name: String): LogicalPlan =
+ Project(
+ Seq(UnresolvedAlias(
+ UnresolvedFunction(
+ name,
+ Seq(Literal("java.lang.Math"), Literal("abs"), Literal(-1)),
+ isDistinct = false))),
+ TestRelations.testRelation)
+
+ private def transformPlan: ScriptTransformation =
+ ScriptTransformation(
+ "cat",
+ Seq(AttributeReference("value", StringType)()),
+ TestRelations.testRelation,
+ ScriptInputOutputSchema(Nil, Nil, None, None, Nil, Nil, None, None, schemaLess = false))
+
+ private def checkRestrictedError(e: AnalysisException, feature: String): Unit = {
+ checkError(
+ exception = e,
+ condition = "UNSUPPORTED_FEATURE.SQL_RESTRICTED_MODE",
+ parameters = Map("feature" -> feature, "config" -> configName))
+ }
+
+ test("reflect / java_method / try_reflect are rejected only when restricted mode is enabled") {
+ Seq("reflect", "java_method", "try_reflect").foreach { fn =>
+ withRestrictedMode(true) {
+ val analyzer = getAnalyzer
+ val e = intercept[AnalysisException] {
+ analyzer.checkAnalysis(analyzer.execute(functionProject(fn)))
+ }
+ checkRestrictedError(e, s"The `$fn` function")
+ }
+ withRestrictedMode(false) {
+ val analyzer = getAnalyzer
+ analyzer.checkAnalysis(analyzer.execute(functionProject(fn)))
+ }
+ }
+ }
+
+ test("TRANSFORM ... USING is rejected only when restricted mode is enabled") {
+ withRestrictedMode(true) {
+ val analyzer = getAnalyzer
+ val e = intercept[AnalysisException] {
+ analyzer.checkAnalysis(analyzer.execute(transformPlan))
+ }
+ checkRestrictedError(e, "The TRANSFORM ... USING clause")
+ }
+ withRestrictedMode(false) {
+ val analyzer = getAnalyzer
+ // Analysis succeeds (no restricted-mode error) when the profile is off.
+ analyzer.checkAnalysis(analyzer.execute(transformPlan))
+ }
+ }
+
+ test("restricted mode is enforced for an analyzed sub-plan under a fresh parent") {
+ // An analyzed sub-plan reused under a new query (as a temporary view or a cached Dataset
+ // stores it): `setAnalyzed()` marks the sub-tree analyzed, so `checkAnalysis0`'s
+ // `case p if p.analyzed` skips it. A fresh, un-analyzed parent reproduces that shape;
+ // `checkRestrictedMode` must still reach the feature inside the analyzed child.
+ val analyzedChild = withRestrictedMode(false) {
+ val plan = getAnalyzer.execute(transformPlan)
+ plan.setAnalyzed()
+ plan
+ }
+ val parent = Project(analyzedChild.output, analyzedChild)
+ withRestrictedMode(true) {
+ val e = intercept[AnalysisException](getAnalyzer.checkAnalysis(parent))
+ checkRestrictedError(e, "The TRANSFORM ... USING clause")
+ }
+ }
+}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/RestrictedModeCommandSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/RestrictedModeCommandSuite.scala
new file mode 100644
index 000000000000..0ed52ffbd7fb
--- /dev/null
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/RestrictedModeCommandSuite.scala
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.internal.StaticSQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+
+/**
+ * Tests for the opt-in restricted SQL execution mode that need a running session -- in particular
+ * that the check reaches the body of an analysis-only command (whose body moves into
+ * `innerChildren` once analyzed), and that the static config cannot be turned off at runtime.
+ */
+class RestrictedModeCommandSuite extends QueryTest with SharedSparkSession {
+
+ private val restricted = StaticSQLConf.RESTRICTED_MODE_ENABLED.key
+ // The config is rendered double-quoted by `toSQLConf` in the error message.
+ private val configName = "\"" + restricted + "\""
+
+ override protected def sparkConf: SparkConf =
+ super.sparkConf.set(restricted, "true")
+
+ private def checkRestricted(sqlText: String, feature: String): Unit = {
+ checkError(
+ exception = intercept[AnalysisException](sql(sqlText)),
+ condition = "UNSUPPORTED_FEATURE.SQL_RESTRICTED_MODE",
+ parameters = Map("feature" -> feature, "config" -> configName))
+ }
+
+ test("restricted mode rejects reflect inside a CREATE VIEW body") {
+ withView("v") {
+ // CreateViewCommand is an analysis-only command: once analyzed, its body (which contains the
+ // reflect call) moves out of `children` into `innerChildren`.
+ checkRestricted(
+ "CREATE VIEW v AS SELECT reflect('java.lang.Math', 'abs', -1) AS c",
+ "The `reflect` function")
+ }
+ }
+
+ test("restricted mode rejects TRANSFORM inside a CACHE TABLE ... AS SELECT body") {
+ // CacheTableAsSelect is an analysis-only command whose query runs at command execution, so the
+ // gate must reach the body; the query is never executed because analysis rejects it first.
+ checkRestricted(
+ "CACHE TABLE c AS SELECT TRANSFORM(id) USING 'cat' AS (x) FROM range(1)",
+ "The TRANSFORM ... USING clause")
+ }
+
+ test("the restricted mode config cannot be turned off at runtime") {
+ val e = intercept[AnalysisException](sql(s"SET $restricted=false"))
+ assert(e.getCondition == "CANNOT_MODIFY_STATIC_CONFIG")
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org