You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by jt...@apache.org on 2026/07/27 16:06:12 UTC

(spark) branch branch-4.x updated: [SPARK-57251][SDP] Validate SCD2 reserved framework columns at AutoCDC flow construction

This is an automated email from the ASF dual-hosted git repository.

jose-torres pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 17c6b74f3b50 [SPARK-57251][SDP] Validate SCD2 reserved framework columns at AutoCDC flow construction
17c6b74f3b50 is described below

commit 17c6b74f3b5006ed98d15562729fb4d48a509ab4
Author: Andreas Neumann <an...@databricks.com>
AuthorDate: Mon Jul 27 09:05:47 2026 -0700

    [SPARK-57251][SDP] Validate SCD2 reserved framework columns at AutoCDC flow construction
    
    ### What changes were proposed in this pull request?
    What changes were proposed in this pull request?
    
    This PR replaces https://github.com/apache/spark/pull/57488, which had to be abandoned due to severe semantic merge conflicts. It is an exact cherry-pick of the previous commits, plus a fix for the test cases that failed due to the merge conflicts.
    
    What were the merge conflicts? Two PRs disagreeing on the semantics when a source contains a reserved column but does not include that column in the column selection. This PR rejects that because it validates pre-column selection. The other PR (SPARK-58313) allowed that because it was doing the validation port column selection. I decided to fall back to the original behavior that already existed for SCD type 1 validate pre column selection. And we have SPARK-58325 to reconsider whethe [...]
    
    AutoCdcMergeFlow validates at construction time that a flow's source change-data feed does not carry columns that collide with AutoCDC's internal columns. Until now, that validation (requireReservedPrefixAbsentInSourceColumns) only rejected column names starting with the reserved prefix __spark_autocdc_.
    
      SCD2, however, persists two framework columns to the target that do not carry that prefix — __START_AT and __END_AT (see Scd2BatchProcessor.reservedFrameworkColNames). A source column named __START_AT or __END_AT therefore slipped past the guard and would be silently overwritten during microbatch preprocessing.
    
      This PR closes that gap (the TODO(SPARK-57251) in Scd2BatchProcessor):
    
      - Adds requireReservedFrameworkColumnsAbsentInSourceColumns() to the AutoCdcMergeFlow constructor. For SCD2 flows it rejects any source  column whose name collides (by exact name, resolver-aware so it respects spark.sql.caseSensitive) with a non-prefixed reserved framework column. SCD1 targets carry no such columns, so the check is a no-op for SCD1.
      - Runs the check before the flow's schema val is forced, so the actionable reserved-name error surfaces ahead of the temporary AUTOCDC_SCD2_NOT_SUPPORTED gate, and the check remains correct once SCD2 support lands.
      - Adds a new error condition AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT (SQLSTATE 42710), distinct from the existing prefix-based AUTOCDC_RESERVED_COLUMN_NAME_PREFIX_CONFLICT, since this collision is by exact name rather than by prefix.
      - Widens the visibility of Scd2BatchProcessor.reservedFrameworkColNames to private[pipelines] so the flow layer can validate against the single source of truth.
    
    ### Why are the changes needed?
     Without this check, a user whose CDC source happens to contain a __START_AT or __END_AT column would have that data silently overwritten by AutoCDC's SCD2 framework columns, with no error and no diagnostic — a data-correctness footgun. Failing fast at flow construction with a user-actionable error ("rename or remove the column") is the intended UX, consistent with the existing reserved-prefix guard.
    
    ### Does this PR introduce _any_ user-facing change?
    Yes. An AutoCDC SCD2 flow whose source change-data feed contains a column named __START_AT or __END_AT (subject to case-sensitivity settings) now fails at flow construction with AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT instead of silently overwriting the column. There is no change for SCD1 flows, and no change for SCD2 flows whose sources do not use these names. (Note: SCD2 AutoCDC flows are not yet generally supported on master — they are still gated by AUTOCDC_SCD2_NOT_SUPPORTED — so n [...]
    
    ### How was this patch tested?
    New unit tests in AutoCdcFlowSuite covering:
      - an SCD2 flow with a __START_AT/__END_AT source column is rejected with AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT;
      - the reserved-name check fires before the AUTOCDC_SCD2_NOT_SUPPORTED gate;
      - an SCD1 flow with the same column name is allowed and the column survives into the flow schema;
      - case-sensitivity behavior (spark.sql.caseSensitive true/false);
      - a guard test asserting the set of non-prefixed reserved names is exactly {__START_AT, __END_AT}, so a future rename can't silently un-cover the validation.
    
    ### Was this patch authored or co-authored using generative AI tooling?
     Generated-by: Claude Code (Opus 4.8)
    
    Closes #57527 from anew/spark-57251-reserved-columns-v2.
    
    Authored-by: Andreas Neumann <an...@databricks.com>
    Signed-off-by: Jose Torres <jt...@apache.org>
    (cherry picked from commit 18ad81147cbd86373ece320878b8469acd7f2e5f)
    Signed-off-by: Jose Torres <jt...@apache.org>
---
 .../src/main/resources/error/error-conditions.json |   6 +
 .../sql/pipelines/autocdc/Scd2BatchProcessor.scala |  15 +-
 .../apache/spark/sql/pipelines/graph/Flow.scala    |  40 ++++
 .../sql/pipelines/autocdc/AutoCdcFlowSuite.scala   | 205 +++++++++++++++------
 4 files changed, 204 insertions(+), 62 deletions(-)

diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json
index 05aec78e0919..04aa72c8cc8a 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -331,6 +331,12 @@
     ],
     "sqlState" : "22023"
   },
+  "AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT" : {
+    "message" : [
+      "The column `<columnName>` in the <schemaName> schema collides with a reserved AutoCDC <scdType> column name (using <caseSensitivity> column name comparison). The following column names are reserved by AutoCDC and cannot appear in the source: <reservedColumnNames>. Rename or remove the column."
+    ],
+    "sqlState" : "42710"
+  },
   "AUTOCDC_RESERVED_COLUMN_NAME_PREFIX_CONFLICT" : {
     "message" : [
       "The column `<columnName>` in the <schemaName> schema collides with the reserved AutoCDC column name prefix `<reservedColumnNamePrefix>` (using <caseSensitivity> column name comparison). Rename or remove the column."
diff --git a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala
index fbf9bc15127e..e41f223322bc 100644
--- a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala
+++ b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala
@@ -1381,15 +1381,16 @@ object Scd2BatchProcessor {
   private[pipelines] val endAtColName: String = "__END_AT"
 
   /**
-   * Column names reserved by AutoCDC that will be projected onto the microbatch and
-   * eventually persisted in the target table. If the user's source dataframe contains any of
-   * these columns, SCD2 reconciliation will fail.
+   * Column names reserved by AutoCDC that are projected onto the microbatch and persisted in the
+   * target table. A source dataframe must not contain any of them.
    *
-   * TODO(SPARK-57251): validate at [[AutoCdcMergeFlow]] construction time that the source
-   *   schema and column selection do not collide with these reserved names, so we fail fast
-   *   with a user-actionable error instead of silently overwriting them at preprocess time.
+   * [[startAtColName]] and [[endAtColName]] do NOT carry the reserved
+   * [[AutoCdcReservedNames.prefix]], so a source-column collision with them is not caught by the
+   * prefix-based guard; [[org.apache.spark.sql.pipelines.graph.AutoCdcMergeFlow]] validates the
+   * source schema against the non-prefixed names in this set at construction time, failing fast
+   * instead of silently overwriting them at preprocess time.
    */
-  private val reservedFrameworkColNames: Set[String] = Set(
+  private[pipelines] val reservedFrameworkColNames: Set[String] = Set(
     startAtColName,
     endAtColName,
     AutoCdcReservedNames.cdcMetadataColName
diff --git a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala
index 410b0486c787..bb996d527cdc 100644
--- a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala
+++ b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala
@@ -253,6 +253,7 @@ class AutoCdcMergeFlow(
     val funcResult: FlowFunctionResult
 ) extends ResolvedFlow {
   requireReservedPrefixAbsentInSourceColumns()
+  requireReservedFrameworkColumnsAbsentInSourceColumns()
 
   def changeArgs: ChangeArgs = flow.changeArgs
 
@@ -400,6 +401,45 @@ class AutoCdcMergeFlow(
     }
   }
 
+  /**
+   * Reject a source column that collides with an SCD2 reserved framework column not covered by
+   * [[requireReservedPrefixAbsentInSourceColumns]]: the prefix guard only rejects
+   * [[AutoCdcReservedNames.prefix]] names, but SCD2 also persists the non-prefixed
+   * [[Scd2BatchProcessor.startAtColName]] and [[Scd2BatchProcessor.endAtColName]]. Runs before
+   * [[schema]] is forced so the collision fails fast rather than being silently overwritten
+   * during preprocessing. No-op for SCD1, which has no such columns.
+   */
+  private def requireReservedFrameworkColumnsAbsentInSourceColumns(): Unit = {
+    val resolver = spark.sessionState.conf.resolver
+    val reservedPrefix = AutoCdcReservedNames.prefix
+
+    // Only the non-prefixed reserved names need checking here; prefixed ones are already rejected
+    // by requireReservedPrefixAbsentInSourceColumns.
+    val reservedNames: Set[String] = changeArgs.storedAsScdType match {
+      case ScdType.Type2 =>
+        Scd2BatchProcessor.reservedFrameworkColNames.filterNot(_.startsWith(reservedPrefix))
+      case ScdType.Type1 =>
+        Set.empty
+    }
+
+    df.schema.fieldNames
+      .find(name => reservedNames.exists(resolver(_, name)))
+      .foreach { conflictingColumnName =>
+        throw new AnalysisException(
+          errorClass = "AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT",
+          messageParameters = Map(
+            "caseSensitivity" -> CaseSensitivityLabels.of(
+              spark.sessionState.conf.caseSensitiveAnalysis
+            ),
+            "columnName" -> conflictingColumnName,
+            "schemaName" -> "changeDataFeed",
+            "scdType" -> changeArgs.storedAsScdType.label,
+            "reservedColumnNames" -> reservedNames.toSeq.sorted.mkString(", ")
+          )
+        )
+      }
+  }
+
   /**
    * Validate all keys specified in changeArgs are actually present in the user-selected schema.
    */
diff --git a/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/autocdc/AutoCdcFlowSuite.scala b/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/autocdc/AutoCdcFlowSuite.scala
index 5769b264993b..d31b9383354c 100644
--- a/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/autocdc/AutoCdcFlowSuite.scala
+++ b/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/autocdc/AutoCdcFlowSuite.scala
@@ -370,63 +370,42 @@ class AutoCdcFlowSuite extends QueryTest with SharedSparkSession {
     }
   }
 
-  test("AutoCdcMergeFlow.schema lets a selection exclude a source column that collides with " +
-    "a non-prefixed framework column, re-adding it from the framework") {
-    // Two of the three SCD2 framework columns -- __START_AT and __END_AT -- do not carry the
-    // reserved AutoCDC prefix, so a source change feed may legitimately contain columns with
-    // those names. The user excludes them via the column selection; they are dropped from the
-    // user-data portion and the engine's own framework columns are appended in their place, so
-    // each still appears exactly once in the output with the framework's type (the sequencing
-    // type), not the source column's type.
-    //
-    // The third framework column, _cdc_metadata, DOES carry the reserved prefix, so a source
-    // that contains it is rejected outright at flow construction -- it can never reach column
-    // selection and so is deliberately out of scope here. That rejection is covered separately by
-    // "AutoCdcMergeFlow rejects a source df column whose name equals the reserved CDC metadata
-    // column".
-    val session = spark
-    import session.implicits._
-    // Source carries String-typed __START_AT / __END_AT columns alongside the data columns.
-    val sourceDf = MemoryStream[(Int, String, Option[Long], String, String)]
-      .toDS()
-      .toDF(
-        "id",
-        "name",
-        "seq",
-        Scd2BatchProcessor.startAtColName,
-        Scd2BatchProcessor.endAtColName)
+  test("AutoCdcMergeFlow rejects a source column named after a non-prefixed framework column " +
+    "even when a selection would exclude it") {
+    // Behavior change introduced by this PR (SPARK-57251): __START_AT / __END_AT do not carry the
+    // reserved AutoCDC prefix, so before this change a source could legitimately contain columns
+    // with those names and exclude them via the column selection. The new
+    // requireReservedFrameworkColumnsAbsentInSourceColumns guard runs against the RAW source
+    // schema (before column selection is applied), so such a source is now rejected outright at
+    // flow construction -- an ExcludeColumns selection cannot rescue it. Allowing an explicit
+    // opt-out (validating post-selection instead) is tracked separately by SPARK-58325.
+    val sourceDf = sourceDfWithExtraColumns(
+      Scd2BatchProcessor.startAtColName -> StringType,
+      Scd2BatchProcessor.endAtColName -> StringType)
 
-    val resolvedFlow = newAutoCdcMergeFlow(
-      sourceDf = sourceDf,
-      storedAsScdType = ScdType.Type2,
-      columnSelection = Some(
-        ColumnSelection.ExcludeColumns(
-          Seq(
-            UnqualifiedColumnName(Scd2BatchProcessor.startAtColName),
-            UnqualifiedColumnName(Scd2BatchProcessor.endAtColName))
+    checkError(
+      exception = intercept[AnalysisException] {
+        newAutoCdcMergeFlow(
+          sourceDf = sourceDf,
+          storedAsScdType = ScdType.Type2,
+          columnSelection = Some(
+            ColumnSelection.ExcludeColumns(
+              Seq(
+                UnqualifiedColumnName(Scd2BatchProcessor.startAtColName),
+                UnqualifiedColumnName(Scd2BatchProcessor.endAtColName))
+            )
+          )
         )
-      )
-    )
-
-    // Each framework column appears exactly once (the source copy was excluded, the framework
-    // copy appended) ...
-    assert(
-      resolvedFlow.schema.fieldNames.count(_ == Scd2BatchProcessor.startAtColName) == 1)
-    assert(
-      resolvedFlow.schema.fieldNames.count(_ == Scd2BatchProcessor.endAtColName) == 1)
-    // ... and carries the framework (sequencing) type, not the source String type.
-    assert(resolvedFlow.schema(Scd2BatchProcessor.startAtColName).dataType == LongType)
-    assert(resolvedFlow.schema(Scd2BatchProcessor.endAtColName).dataType == LongType)
-    // Full expected shape: the retained data columns followed by the framework columns.
-    assert(
-      resolvedFlow.schema.fieldNames.toSeq ==
-      Seq(
-        "id",
-        "name",
-        "seq",
-        Scd2BatchProcessor.startAtColName,
-        Scd2BatchProcessor.endAtColName,
-        AutoCdcReservedNames.cdcMetadataColName
+      },
+      condition = "AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT",
+      sqlState = "42710",
+      parameters = Map(
+        "caseSensitivity" -> CaseSensitivityLabels.CaseInsensitive,
+        "columnName" -> Scd2BatchProcessor.startAtColName,
+        "schemaName" -> "changeDataFeed",
+        "scdType" -> ScdType.Type2.label,
+        "reservedColumnNames" ->
+          Seq(Scd2BatchProcessor.endAtColName, Scd2BatchProcessor.startAtColName).mkString(", ")
       )
     )
   }
@@ -703,6 +682,122 @@ class AutoCdcFlowSuite extends QueryTest with SharedSparkSession {
     }
   }
 
+  // ===========================================================================================
+  // AutoCdcMergeFlow reserved framework-column (non-prefixed) validation tests
+  //
+  // SCD2 persists framework columns __START_AT / __END_AT that do NOT carry the reserved
+  // AutoCDC prefix, so they are not caught by the prefix guard above. These tests lock in that a
+  // source column colliding with such a name is rejected at construction for SCD2, is allowed
+  // for SCD1 (which reserves no non-prefixed names), and that the check respects case-sensitivity.
+  // ===========================================================================================
+
+  /** The SCD2 reserved framework column names that are not covered by the reserved prefix. */
+  private val nonPrefixedScd2ReservedNames: Seq[String] =
+    Scd2BatchProcessor.reservedFrameworkColNames
+      .filterNot(_.startsWith(AutoCdcReservedNames.prefix))
+      .toSeq
+      .sorted
+
+  test("non-prefixed reserved names exist and are covered by this suite") {
+    // Guards against a future refactor renaming/removing __START_AT / __END_AT without updating
+    // the flow-construction validation: if this set ever empties, the tests below silently
+    // stop exercising anything.
+    assert(
+      nonPrefixedScd2ReservedNames == Seq("__END_AT", "__START_AT"),
+      s"Unexpected non-prefixed SCD2 reserved names: $nonPrefixedScd2ReservedNames"
+    )
+  }
+
+  test(
+    "an SCD2 flow with a source column colliding with a reserved framework column is rejected " +
+    "at construction"
+  ) {
+    nonPrefixedScd2ReservedNames.foreach { reservedName =>
+      val sourceDf = sourceDfWithExtraColumns(reservedName -> StringType)
+
+      checkError(
+        exception = intercept[AnalysisException] {
+          newAutoCdcMergeFlow(sourceDf, storedAsScdType = ScdType.Type2)
+        },
+        condition = "AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT",
+        sqlState = "42710",
+        parameters = Map(
+          "caseSensitivity" -> CaseSensitivityLabels.CaseInsensitive,
+          "columnName" -> reservedName,
+          "schemaName" -> "changeDataFeed",
+          "scdType" -> ScdType.Type2.label,
+          "reservedColumnNames" -> nonPrefixedScd2ReservedNames.mkString(", ")
+        )
+      )
+    }
+  }
+
+  test(
+    "the reserved framework-column check runs before the SCD2-not-supported gate"
+  ) {
+    // The reserved-name error is more actionable than AUTOCDC_SCD2_NOT_SUPPORTED, so it must win
+    // for an SCD2 flow that both is unsupported and carries a colliding source column. This also
+    // keeps the check meaningful today (before SCD2 is supported) and correct once it lands.
+    val sourceDf = sourceDfWithExtraColumns(Scd2BatchProcessor.startAtColName -> StringType)
+    val ex = intercept[AnalysisException] {
+      newAutoCdcMergeFlow(sourceDf, storedAsScdType = ScdType.Type2)
+    }
+    assert(ex.getCondition == "AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT")
+  }
+
+  test(
+    "an SCD1 flow with a source column matching an SCD2-only reserved name is allowed"
+  ) {
+    // SCD1 targets carry no non-prefixed framework columns, so __START_AT / __END_AT are ordinary
+    // user columns there. Construction succeeds and the column survives into the flow schema.
+    nonPrefixedScd2ReservedNames.foreach { reservedName =>
+      val sourceDf = sourceDfWithExtraColumns(reservedName -> StringType)
+      val resolvedFlow = newAutoCdcMergeFlow(sourceDf, storedAsScdType = ScdType.Type1)
+      assert(resolvedFlow.schema.fieldNames.contains(reservedName))
+    }
+  }
+
+  test(
+    "an uppercase reserved framework-column name is rejected for SCD2 when caseSensitive=false"
+  ) {
+    withSQLConf(SQLConf.CASE_SENSITIVE.key -> "false") {
+      val conflictingName = Scd2BatchProcessor.startAtColName.toLowerCase(Locale.ROOT)
+      val sourceDf = sourceDfWithExtraColumns(conflictingName -> StringType)
+
+      checkError(
+        exception = intercept[AnalysisException] {
+          newAutoCdcMergeFlow(sourceDf, storedAsScdType = ScdType.Type2)
+        },
+        condition = "AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT",
+        sqlState = "42710",
+        parameters = Map(
+          "caseSensitivity" -> CaseSensitivityLabels.CaseInsensitive,
+          "columnName" -> conflictingName,
+          "schemaName" -> "changeDataFeed",
+          "scdType" -> ScdType.Type2.label,
+          "reservedColumnNames" -> nonPrefixedScd2ReservedNames.mkString(", ")
+        )
+      )
+    }
+  }
+
+  test(
+    "a differently-cased reserved framework-column name does not trip the reserved check for " +
+    "SCD2 when caseSensitive=true"
+  ) {
+    // Under case-sensitive analysis, a lowercase variant is a distinct identifier and does not
+    // collide with the reserved (uppercase) framework name, consistent with the prefix guard.
+    // The reserved-name check therefore does NOT fire and the flow constructs successfully,
+    // keeping the lowercase column as an ordinary user column in the schema.
+    withSQLConf(SQLConf.CASE_SENSITIVE.key -> "true") {
+      val nonConflictingName = Scd2BatchProcessor.startAtColName.toLowerCase(Locale.ROOT)
+      val sourceDf = sourceDfWithExtraColumns(nonConflictingName -> StringType)
+
+      val resolvedFlow = newAutoCdcMergeFlow(sourceDf, storedAsScdType = ScdType.Type2)
+      assert(resolvedFlow.schema.fieldNames.contains(nonConflictingName))
+    }
+  }
+
   // ===========================================================================================
   // AutoCdcMergeFlow keys-presence validation tests (requireKeysPresentInSelectedSchema)
   // ===========================================================================================


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org