(spark) branch master updated: [SPARK-57360][SQL] Block temporary variables in generated column expressions

[sz...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

szehon-ho pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 334952601082 [SPARK-57360][SQL] Block temporary variables in generated column expressions
334952601082 is described below

commit 334952601082732e809ec12cf11319a60dd32ea4
Author: Szehon Ho <sz...@gmail.com>
AuthorDate: Thu Jun 11 16:50:58 2026 -0700

    [SPARK-57360][SQL] Block temporary variables in generated column expressions
    
    ### What changes were proposed in this pull request?
    This PR adds a validation that rejects references to session (temporary) variables in a generated column's generation expression, for both `CREATE TABLE` and `REPLACE TABLE`. The check is added to the central `GeneratedColumnExpression.validate` and throws `UNSUPPORTED_EXPRESSION_GENERATED_COLUMN` with a clear reason.
    
    ### Why are the changes needed?
    This is a regression from #54126 ([SPARK-55347][SQL] Pass Generated Column as Expression to DSV2). Previously, the generation expression was analyzed by an independent (standalone) analyzer that did not resolve session variables, so a generated column referencing a temporary variable was not accepted. After #54126 moved the analysis inline into the main analyzer (similar to how constraint expressions are analyzed), the analyzer's last-resort variable resolution now resolves the refere [...]
    
    Persisting a session-scoped, mutable value into a generation expression is ill-defined: the value recomputed when reading or rewriting the column can differ across sessions or over time. The SQL standard restricts generation expressions to be deterministic and to reference only other columns of the same row, and other engines explicitly disallow variables:
    - **MySQL**: "Variables (system variables, user-defined variables, and stored program local variables) are not permitted."
    - **HSQLDB** (close to the standard): the expression "must reference only other, non-generated, columns of the table in the same row ... must be deterministic and must not access SQL-data."
    
    This restores the previous protection and aligns Spark with the standard.
    
    ### Does this PR introduce _any_ user-facing change?
    Yes. A `CREATE TABLE` / `REPLACE TABLE` whose generated column references a temporary variable now fails analysis with `UNSUPPORTED_EXPRESSION_GENERATED_COLUMN` (reason: "generation expression cannot reference temporary variables") instead of being accepted.
    
    Example:
    ```sql
    DECLARE my_var INT DEFAULT 1;
    CREATE TABLE t(a INT, b INT GENERATED ALWAYS AS (a + my_var)) USING foo;
    -- now fails with UNSUPPORTED_EXPRESSION_GENERATED_COLUMN
    ```
    
    ### How was this patch tested?
    Added unit tests in `DataSourceV2SQLSuite` covering both `CREATE TABLE` and `REPLACE TABLE`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    Generated-by: Cursor (Claude Opus 4.8)
    
    Closes #56421 from szehon-ho/block_temp_variable_from_generated_column.
    
    Authored-by: Szehon Ho <sz...@gmail.com>
    Signed-off-by: Szehon Ho <sz...@gmail.com>
---
 .../plans/logical/GeneratedColumnExpression.scala  | 10 ++++++++-
 .../spark/sql/connector/DataSourceV2SQLSuite.scala | 24 ++++++++++++++++++++++
 2 files changed, 33 insertions(+), 1 deletion(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala
index 28548dacdfbd..6253ccccafc3 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.catalyst.plans.logical
 
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.expressions.{AnalysisAwareExpression, AttributeReference, Cast, Expression, UnaryExpression, Unevaluable}
-import org.apache.spark.sql.catalyst.trees.TreePattern.{ANALYSIS_AWARE_EXPRESSION, PLAN_EXPRESSION, TreePattern}
+import org.apache.spark.sql.catalyst.trees.TreePattern.{ANALYSIS_AWARE_EXPRESSION, PLAN_EXPRESSION, TreePattern, VARIABLE_REFERENCE}
 import org.apache.spark.sql.catalyst.util.V2ExpressionBuilder
 import org.apache.spark.sql.connector.catalog.GenerationExpression
 import org.apache.spark.sql.internal.SQLConf
@@ -57,6 +57,7 @@ case class GeneratedColumnExpression(
    * - The expression must be deterministic
    * - The expression data type can be safely up-cast to the destination column data type
    * - No subquery expressions
+   * - No references to session (temporary) variables
    * - No non-UTF8 binary collation
    */
   def validate(
@@ -77,6 +78,13 @@ case class GeneratedColumnExpression(
       throw unsupportedExpressionError("subquery expressions are not allowed for generated columns")
     }
 
+    // Don't allow references to session (temporary) variables. They are session-scoped mutable
+    // state, so persisting them into a generation expression would be ill-defined.
+    if (child.containsPattern(VARIABLE_REFERENCE)) {
+      throw unsupportedExpressionError(
+        "generation expression cannot reference temporary variables")
+    }
+
     // Use the resolver to respect case sensitivity settings
     val resolver = SQLConf.get.resolver
 
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
index a5480d579a59..752c8ed6c22e 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
@@ -2020,6 +2020,30 @@ class DataSourceV2SQLSuiteV1Filter
     }
   }
 
+  test("SPARK-57360: generation expression cannot reference temporary variables") {
+    val tblName = "my_tab"
+    withSessionVariable("my_var") {
+      sql("DECLARE OR REPLACE VARIABLE my_var INT DEFAULT 1")
+      checkUnsupportedGenerationExpressionForCreate(
+        tblName,
+        "a + my_var",
+        "generation expression cannot reference temporary variables"
+      )
+    }
+  }
+
+  test("SPARK-57360: REPLACE TABLE - generation expression cannot reference temporary variables") {
+    val tblName = "my_tab"
+    withSessionVariable("my_var") {
+      sql("DECLARE OR REPLACE VARIABLE my_var INT DEFAULT 1")
+      checkUnsupportedGenerationExpressionForReplace(
+        tblName,
+        "a + my_var",
+        "generation expression cannot reference temporary variables"
+      )
+    }
+  }
+
   test("SPARK-44313: generation expression validation passes when there is a char/varchar column") {
     val tblName = "my_tab"
     // InMemoryTableCatalog.capabilities() = {SUPPORTS_CREATE_TABLE_WITH_GENERATED_COLUMNS}


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org