You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sz...@apache.org on 2026/09/15 19:55:21 UTC

(spark) branch master updated: [SPARK-59446][CONNECT] Support ignore-null updates in Auto CDC flows over Spark Connect

This is an automated email from the ASF dual-hosted git repository.

szehon-ho pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 6077852d08a9 [SPARK-59446][CONNECT] Support ignore-null updates in Auto CDC flows over Spark Connect
6077852d08a9 is described below

commit 6077852d08a90ea0907abe868aa3e21d318e08f6
Author: Andreas Neumann <an...@apache.org>
AuthorDate: Tue Sep 15 12:55:08 2026 -0700

    [SPARK-59446][CONNECT] Support ignore-null updates in Auto CDC flows over Spark Connect
    
    ### What changes were proposed in this pull request?
    
    Adds partial-update ("ignore null updates") support to Auto CDC flows defined over Spark Connect. Partial updates keep the existing target value for a column when an incoming CDC upsert carries `NULL` for it, instead of overwriting with `NULL`.
    
    - Adds an `optional bool ignore_null_updates` field (field 16) to the `PipelineCommand.DefineFlow.AutoCdcFlowDetails` message, and regenerates the Python protobuf stubs.
    - Wires all three ignore-null inputs through the server handler (`PipelinesHandler.buildAutoCdcFlow`) into `ChangeArgs.ignoreNullSelection`, mirroring the existing `track_history_*` handling:
      - `ignore_null_updates` (all columns) -> `ColumnSelection.ExcludeColumns(Seq.empty)`
      - `ignore_null_updates_column_list` -> `ColumnSelection.IncludeColumns`
      - `ignore_null_updates_except_column_list` -> `ColumnSelection.ExcludeColumns`
    - The three inputs are mutually exclusive; specifying more than one raises the new error condition `AUTOCDC_CONFLICTING_IGNORE_NULL_UPDATES_OPTIONS`.
    
    This removes the `TODO(SPARK-57093)` in the handler, which had left the two previously-declared column-list fields (14/15) unread.
    
    ### Why are the changes needed?
    
    Auto CDC "ignore null updates" has three user-visible states: off, a column subset, and all columns. The engine already models these as `ChangeArgs.ignoreNullSelection: Option[ColumnSelection]`, where "all columns" is `ExcludeColumns(Seq.empty)`, but the Connect handler discarded the two existing column-list fields behind a TODO, so none of it was reachable from a Connect client.
    
    The two repeated fields can express the subset cases but not "all columns": an empty repeated field is indistinguishable from an unset one on the wire, so an empty except-list collapses to "unset" (off). Expressing "ignore nulls on all columns" therefore requires an explicit scalar flag, `ignore_null_updates`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. Declarative pipeline flows are defined via Spark Connect; the Python API surface that sets these fields is added in a follow-up. The engine behavior itself is unchanged.
    
    ### How was this patch tested?
    
    New tests in `SparkDeclarativePipelinesServerSuite` drive the raw Connect stub and assert that the all-columns flag and an include list map to the expected `ignoreNullSelection`, and that conflicting options are rejected server-side. `SparkThrowableSuite` validates the new error condition.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    Generated-by: Opus 4.8
    
    Closes #58784 from anew/autocdc-ignore-null-updates-all-proto.
    
    Authored-by: Andreas Neumann <an...@apache.org>
    Signed-off-by: Szehon Ho <sz...@gmail.com>
---
 .../src/main/resources/error/error-conditions.json |   6 +
 python/pyspark/sql/connect/proto/pipelines_pb2.py  |  76 ++++++-------
 python/pyspark/sql/connect/proto/pipelines_pb2.pyi |  21 ++++
 .../main/protobuf/spark/connect/pipelines.proto    |   4 +
 .../sql/connect/pipelines/PipelinesHandler.scala   |  29 ++++-
 .../SparkDeclarativePipelinesServerSuite.scala     | 123 ++++++++++++++++++++-
 6 files changed, 216 insertions(+), 43 deletions(-)

diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json
index f52a28b00866..f355f13b8f2e 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -233,6 +233,12 @@
     ],
     "sqlState" : "42703"
   },
+  "AUTOCDC_CONFLICTING_IGNORE_NULL_UPDATES_OPTIONS" : {
+    "message" : [
+      "AutoCDC flow specifies more than one of `ignore_null_updates`, `ignore_null_updates_column_list`, and `ignore_null_updates_except_column_list`; at most one may be provided."
+    ],
+    "sqlState" : "42613"
+  },
   "AUTOCDC_EMPTY_KEYS" : {
     "message" : [
       "AutoCDC requires at least one key column to identify rows, but received an empty key set."
diff --git a/python/pyspark/sql/connect/proto/pipelines_pb2.py b/python/pyspark/sql/connect/proto/pipelines_pb2.py
index c32f27f26cdc..d68abadac519 100644
--- a/python/pyspark/sql/connect/proto/pipelines_pb2.py
+++ b/python/pyspark/sql/connect/proto/pipelines_pb2.py
@@ -44,7 +44,7 @@ from pyspark.sql.connect.proto import types_pb2 as spark_dot_connect_dot_types__
 
 
 DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
-    b'\n\x1dspark/connect/pipelines.proto\x12\rspark.connect\x1a\x19google/protobuf/any.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1aspark/connect/common.proto\x1a\x1fspark/connect/expressions.proto\x1a\x1dspark/connect/relations.proto\x1a\x19spark/connect/types.proto"\xcc\x30\n\x0fPipelineCommand\x12h\n\x15\x63reate_dataflow_graph\x18\x01 \x01(\x0b\x32\x32.spark.connect.PipelineCommand.CreateDataflowGraphH\x00R\x13\x63reateDataflowGraph\x12R\n\rdefine_output\x18\x02 \x01(\x0b\x32 [...]
+    b'\n\x1dspark/connect/pipelines.proto\x12\rspark.connect\x1a\x19google/protobuf/any.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1aspark/connect/common.proto\x1a\x1fspark/connect/expressions.proto\x1a\x1dspark/connect/relations.proto\x1a\x19spark/connect/types.proto"\x99\x31\n\x0fPipelineCommand\x12h\n\x15\x63reate_dataflow_graph\x18\x01 \x01(\x0b\x32\x32.spark.connect.PipelineCommand.CreateDataflowGraphH\x00R\x13\x63reateDataflowGraph\x12R\n\rdefine_output\x18\x02 \x01(\x0b\x32 [...]
 )
 
 _globals = globals()
@@ -87,10 +87,10 @@ if not _descriptor._USE_C_DESCRIPTORS:
     _globals["_PIPELINEANALYSISCONTEXT"].fields_by_name[
         "flow_name"
     ]._serialized_options = b"\030\001"
-    _globals["_OUTPUTTYPE"]._serialized_start = 8167
-    _globals["_OUTPUTTYPE"]._serialized_end = 8272
+    _globals["_OUTPUTTYPE"]._serialized_start = 8244
+    _globals["_OUTPUTTYPE"]._serialized_end = 8349
     _globals["_PIPELINECOMMAND"]._serialized_start = 228
-    _globals["_PIPELINECOMMAND"]._serialized_end = 6448
+    _globals["_PIPELINECOMMAND"]._serialized_end = 6525
     _globals["_PIPELINECOMMAND_CREATEDATAFLOWGRAPH"]._serialized_start = 1265
     _globals["_PIPELINECOMMAND_CREATEDATAFLOWGRAPH"]._serialized_end = 1573
     _globals["_PIPELINECOMMAND_CREATEDATAFLOWGRAPH_SQLCONFENTRY"]._serialized_start = 1474
@@ -112,43 +112,43 @@ if not _descriptor._USE_C_DESCRIPTORS:
     _globals["_PIPELINECOMMAND_DEFINEOUTPUT_SINKDETAILS_OPTIONSENTRY"]._serialized_start = 2795
     _globals["_PIPELINECOMMAND_DEFINEOUTPUT_SINKDETAILS_OPTIONSENTRY"]._serialized_end = 2853
     _globals["_PIPELINECOMMAND_DEFINEFLOW"]._serialized_start = 2969
-    _globals["_PIPELINECOMMAND_DEFINEFLOW"]._serialized_end = 5056
+    _globals["_PIPELINECOMMAND_DEFINEFLOW"]._serialized_end = 5133
     _globals["_PIPELINECOMMAND_DEFINEFLOW_SQLCONFENTRY"]._serialized_start = 1474
     _globals["_PIPELINECOMMAND_DEFINEFLOW_SQLCONFENTRY"]._serialized_end = 1532
     _globals["_PIPELINECOMMAND_DEFINEFLOW_WRITERELATIONFLOWDETAILS"]._serialized_start = 3703
     _globals["_PIPELINECOMMAND_DEFINEFLOW_WRITERELATIONFLOWDETAILS"]._serialized_end = 3800
     _globals["_PIPELINECOMMAND_DEFINEFLOW_AUTOCDCFLOWDETAILS"]._serialized_start = 3803
-    _globals["_PIPELINECOMMAND_DEFINEFLOW_AUTOCDCFLOWDETAILS"]._serialized_end = 4808
-    _globals["_PIPELINECOMMAND_DEFINEFLOW_RESPONSE"]._serialized_start = 4810
-    _globals["_PIPELINECOMMAND_DEFINEFLOW_RESPONSE"]._serialized_end = 4868
-    _globals["_PIPELINECOMMAND_DEFINEFLOW_SCDTYPE"]._serialized_start = 4870
-    _globals["_PIPELINECOMMAND_DEFINEFLOW_SCDTYPE"]._serialized_end = 4937
-    _globals["_PIPELINECOMMAND_EXECUTEOUTPUTFLOWS"]._serialized_start = 5059
-    _globals["_PIPELINECOMMAND_EXECUTEOUTPUTFLOWS"]._serialized_end = 5415
-    _globals["_PIPELINECOMMAND_STARTRUN"]._serialized_start = 5418
-    _globals["_PIPELINECOMMAND_STARTRUN"]._serialized_end = 5740
-    _globals["_PIPELINECOMMAND_DEFINESQLGRAPHELEMENTS"]._serialized_start = 5743
-    _globals["_PIPELINECOMMAND_DEFINESQLGRAPHELEMENTS"]._serialized_end = 5942
-    _globals["_PIPELINECOMMAND_GETQUERYFUNCTIONEXECUTIONSIGNALSTREAM"]._serialized_start = 5945
-    _globals["_PIPELINECOMMAND_GETQUERYFUNCTIONEXECUTIONSIGNALSTREAM"]._serialized_end = 6103
-    _globals["_PIPELINECOMMAND_DEFINEFLOWQUERYFUNCTIONRESULT"]._serialized_start = 6106
-    _globals["_PIPELINECOMMAND_DEFINEFLOWQUERYFUNCTIONRESULT"]._serialized_end = 6432
-    _globals["_PIPELINECOMMANDRESULT"]._serialized_start = 6451
-    _globals["_PIPELINECOMMANDRESULT"]._serialized_end = 7203
-    _globals["_PIPELINECOMMANDRESULT_CREATEDATAFLOWGRAPHRESULT"]._serialized_start = 6820
-    _globals["_PIPELINECOMMANDRESULT_CREATEDATAFLOWGRAPHRESULT"]._serialized_end = 6918
-    _globals["_PIPELINECOMMANDRESULT_DEFINEOUTPUTRESULT"]._serialized_start = 6921
-    _globals["_PIPELINECOMMANDRESULT_DEFINEOUTPUTRESULT"]._serialized_end = 7054
-    _globals["_PIPELINECOMMANDRESULT_DEFINEFLOWRESULT"]._serialized_start = 7057
-    _globals["_PIPELINECOMMANDRESULT_DEFINEFLOWRESULT"]._serialized_end = 7188
-    _globals["_PIPELINEEVENTRESULT"]._serialized_start = 7205
-    _globals["_PIPELINEEVENTRESULT"]._serialized_end = 7278
-    _globals["_PIPELINEEVENT"]._serialized_start = 7280
-    _globals["_PIPELINEEVENT"]._serialized_end = 7396
-    _globals["_SOURCECODELOCATION"]._serialized_start = 7399
-    _globals["_SOURCECODELOCATION"]._serialized_end = 7640
-    _globals["_PIPELINEQUERYFUNCTIONEXECUTIONSIGNAL"]._serialized_start = 7643
-    _globals["_PIPELINEQUERYFUNCTIONEXECUTIONSIGNAL"]._serialized_end = 7794
-    _globals["_PIPELINEANALYSISCONTEXT"]._serialized_start = 7797
-    _globals["_PIPELINEANALYSISCONTEXT"]._serialized_end = 8165
+    _globals["_PIPELINECOMMAND_DEFINEFLOW_AUTOCDCFLOWDETAILS"]._serialized_end = 4885
+    _globals["_PIPELINECOMMAND_DEFINEFLOW_RESPONSE"]._serialized_start = 4887
+    _globals["_PIPELINECOMMAND_DEFINEFLOW_RESPONSE"]._serialized_end = 4945
+    _globals["_PIPELINECOMMAND_DEFINEFLOW_SCDTYPE"]._serialized_start = 4947
+    _globals["_PIPELINECOMMAND_DEFINEFLOW_SCDTYPE"]._serialized_end = 5014
+    _globals["_PIPELINECOMMAND_EXECUTEOUTPUTFLOWS"]._serialized_start = 5136
+    _globals["_PIPELINECOMMAND_EXECUTEOUTPUTFLOWS"]._serialized_end = 5492
+    _globals["_PIPELINECOMMAND_STARTRUN"]._serialized_start = 5495
+    _globals["_PIPELINECOMMAND_STARTRUN"]._serialized_end = 5817
+    _globals["_PIPELINECOMMAND_DEFINESQLGRAPHELEMENTS"]._serialized_start = 5820
+    _globals["_PIPELINECOMMAND_DEFINESQLGRAPHELEMENTS"]._serialized_end = 6019
+    _globals["_PIPELINECOMMAND_GETQUERYFUNCTIONEXECUTIONSIGNALSTREAM"]._serialized_start = 6022
+    _globals["_PIPELINECOMMAND_GETQUERYFUNCTIONEXECUTIONSIGNALSTREAM"]._serialized_end = 6180
+    _globals["_PIPELINECOMMAND_DEFINEFLOWQUERYFUNCTIONRESULT"]._serialized_start = 6183
+    _globals["_PIPELINECOMMAND_DEFINEFLOWQUERYFUNCTIONRESULT"]._serialized_end = 6509
+    _globals["_PIPELINECOMMANDRESULT"]._serialized_start = 6528
+    _globals["_PIPELINECOMMANDRESULT"]._serialized_end = 7280
+    _globals["_PIPELINECOMMANDRESULT_CREATEDATAFLOWGRAPHRESULT"]._serialized_start = 6897
+    _globals["_PIPELINECOMMANDRESULT_CREATEDATAFLOWGRAPHRESULT"]._serialized_end = 6995
+    _globals["_PIPELINECOMMANDRESULT_DEFINEOUTPUTRESULT"]._serialized_start = 6998
+    _globals["_PIPELINECOMMANDRESULT_DEFINEOUTPUTRESULT"]._serialized_end = 7131
+    _globals["_PIPELINECOMMANDRESULT_DEFINEFLOWRESULT"]._serialized_start = 7134
+    _globals["_PIPELINECOMMANDRESULT_DEFINEFLOWRESULT"]._serialized_end = 7265
+    _globals["_PIPELINEEVENTRESULT"]._serialized_start = 7282
+    _globals["_PIPELINEEVENTRESULT"]._serialized_end = 7355
+    _globals["_PIPELINEEVENT"]._serialized_start = 7357
+    _globals["_PIPELINEEVENT"]._serialized_end = 7473
+    _globals["_SOURCECODELOCATION"]._serialized_start = 7476
+    _globals["_SOURCECODELOCATION"]._serialized_end = 7717
+    _globals["_PIPELINEQUERYFUNCTIONEXECUTIONSIGNAL"]._serialized_start = 7720
+    _globals["_PIPELINEQUERYFUNCTIONEXECUTIONSIGNAL"]._serialized_end = 7871
+    _globals["_PIPELINEANALYSISCONTEXT"]._serialized_start = 7874
+    _globals["_PIPELINEANALYSISCONTEXT"]._serialized_end = 8242
 # @@protoc_insertion_point(module_scope)
diff --git a/python/pyspark/sql/connect/proto/pipelines_pb2.pyi b/python/pyspark/sql/connect/proto/pipelines_pb2.pyi
index 70e04ad85042..869efdab1d6b 100644
--- a/python/pyspark/sql/connect/proto/pipelines_pb2.pyi
+++ b/python/pyspark/sql/connect/proto/pipelines_pb2.pyi
@@ -594,6 +594,7 @@ class PipelineCommand(google.protobuf.message.Message):
             TRACK_HISTORY_EXCEPT_COLUMN_LIST_FIELD_NUMBER: builtins.int
             IGNORE_NULL_UPDATES_COLUMN_LIST_FIELD_NUMBER: builtins.int
             IGNORE_NULL_UPDATES_EXCEPT_COLUMN_LIST_FIELD_NUMBER: builtins.int
+            IGNORE_NULL_UPDATES_FIELD_NUMBER: builtins.int
             source: builtins.str
             """The name of the CDC source to stream from."""
             @property
@@ -660,6 +661,10 @@ class PipelineCommand(google.protobuf.message.Message):
                 pyspark.sql.connect.proto.expressions_pb2.Expression
             ]:
                 """Subset of columns excluded from ignoring null in updates."""
+            ignore_null_updates: builtins.bool
+            """When true, ignore null updates on all columns of the target table. Mutually exclusive
+            with ignore_null_updates_column_list and ignore_null_updates_except_column_list.
+            """
             def __init__(
                 self,
                 *,
@@ -695,6 +700,7 @@ class PipelineCommand(google.protobuf.message.Message):
                     pyspark.sql.connect.proto.expressions_pb2.Expression
                 ]
                 | None = ...,
+                ignore_null_updates: builtins.bool | None = ...,
             ) -> None: ...
             def HasField(
                 self,
@@ -703,6 +709,8 @@ class PipelineCommand(google.protobuf.message.Message):
                     b"_apply_as_deletes",
                     "_apply_as_truncates",
                     b"_apply_as_truncates",
+                    "_ignore_null_updates",
+                    b"_ignore_null_updates",
                     "_sequence_by",
                     b"_sequence_by",
                     "_source",
@@ -711,6 +719,8 @@ class PipelineCommand(google.protobuf.message.Message):
                     b"apply_as_deletes",
                     "apply_as_truncates",
                     b"apply_as_truncates",
+                    "ignore_null_updates",
+                    b"ignore_null_updates",
                     "sequence_by",
                     b"sequence_by",
                     "source",
@@ -724,6 +734,8 @@ class PipelineCommand(google.protobuf.message.Message):
                     b"_apply_as_deletes",
                     "_apply_as_truncates",
                     b"_apply_as_truncates",
+                    "_ignore_null_updates",
+                    b"_ignore_null_updates",
                     "_sequence_by",
                     b"_sequence_by",
                     "_source",
@@ -736,6 +748,8 @@ class PipelineCommand(google.protobuf.message.Message):
                     b"column_list",
                     "except_column_list",
                     b"except_column_list",
+                    "ignore_null_updates",
+                    b"ignore_null_updates",
                     "ignore_null_updates_column_list",
                     b"ignore_null_updates_column_list",
                     "ignore_null_updates_except_column_list",
@@ -767,6 +781,13 @@ class PipelineCommand(google.protobuf.message.Message):
                 ],
             ) -> typing_extensions.Literal["apply_as_truncates"] | None: ...
             @typing.overload
+            def WhichOneof(
+                self,
+                oneof_group: typing_extensions.Literal[
+                    "_ignore_null_updates", b"_ignore_null_updates"
+                ],
+            ) -> typing_extensions.Literal["ignore_null_updates"] | None: ...
+            @typing.overload
             def WhichOneof(
                 self, oneof_group: typing_extensions.Literal["_sequence_by", b"_sequence_by"]
             ) -> typing_extensions.Literal["sequence_by"] | None: ...
diff --git a/sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto b/sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto
index 7bfc5ca39826..56fe5ccb1f0f 100644
--- a/sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto
+++ b/sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto
@@ -196,6 +196,10 @@ message PipelineCommand {
       // Subset of columns excluded from ignoring null in updates.
       repeated Expression ignore_null_updates_except_column_list = 15;
 
+      // When true, ignore null updates on all columns of the target table. Mutually exclusive
+      // with ignore_null_updates_column_list and ignore_null_updates_except_column_list.
+      optional bool ignore_null_updates = 16;
+
     }
 
     // SCD Type for Auto CDC target tables.
diff --git a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
index 718d1670bf8c..eb888287dace 100644
--- a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
+++ b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
@@ -430,9 +430,6 @@ private[connect] object PipelinesHandler extends Logging {
       transformExpressionFunc: proto.Expression => Expression): AutoCdcFlow = {
     // TODO(SPARK-57092): apply_as_truncates is declared on AutoCdcFlowDetails but is not yet
     //   honored by the engine; wire it through once SCD1 truncate support lands.
-    // TODO(SPARK-57093): ignore_null_updates_column_list and ignore_null_updates_except_column_list
-    //   are declared on AutoCdcFlowDetails but are not yet honored by the engine; wire them
-    //   through once SCD1 ignore-null support lands.
 
     if (!autoCdcDetails.hasSource) {
       throw new AnalysisException("AUTOCDC_MISSING_SOURCE", Map.empty)
@@ -512,6 +509,29 @@ private[connect] object PipelinesHandler extends Logging {
       throw new AnalysisException("AUTOCDC_TRACK_HISTORY_REQUIRES_SCD2", Map.empty)
     }
 
+    // Ignore-null (partial update) selection. The three inputs are mutually exclusive:
+    // ignore_null_updates (all columns), ignore_null_updates_column_list (an include list), and
+    // ignore_null_updates_except_column_list (an except list). "All columns" is represented as an
+    // ExcludeColumns selection with an empty list; this cannot be expressed via the repeated
+    // except field alone (an empty repeated field is indistinguishable from an unset one), which
+    // is why the dedicated boolean exists.
+    val ignoreNullSelection: Option[ColumnSelection] = {
+      val all = autoCdcDetails.getIgnoreNullUpdates
+      val included = autoCdcDetails.getIgnoreNullUpdatesColumnListList.asScala.toSeq
+      val excluded = autoCdcDetails.getIgnoreNullUpdatesExceptColumnListList.asScala.toSeq
+      if (Seq(all, included.nonEmpty, excluded.nonEmpty).count(identity) > 1) {
+        throw new AnalysisException("AUTOCDC_CONFLICTING_IGNORE_NULL_UPDATES_OPTIONS", Map.empty)
+      } else if (all) {
+        Some(ColumnSelection.ExcludeColumns(Seq.empty))
+      } else if (included.nonEmpty) {
+        Some(ColumnSelection.IncludeColumns(included.map(asUnqualifiedColumnName)))
+      } else if (excluded.nonEmpty) {
+        Some(ColumnSelection.ExcludeColumns(excluded.map(asUnqualifiedColumnName)))
+      } else {
+        None
+      }
+    }
+
     val changeArgs = ChangeArgs(
       keys = keys,
       sequencing = toColumn(autoCdcDetails.getSequenceBy),
@@ -519,7 +539,8 @@ private[connect] object PipelinesHandler extends Logging {
       deleteCondition =
         Option.when(autoCdcDetails.hasApplyAsDeletes)(toColumn(autoCdcDetails.getApplyAsDeletes)),
       columnSelection = columnSelection,
-      trackHistorySelection = trackHistorySelection)
+      trackHistorySelection = trackHistorySelection,
+      ignoreNullSelection = ignoreNullSelection)
 
     AutoCdcFlow(
       identifier = flowIdentifier,
diff --git a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/SparkDeclarativePipelinesServerSuite.scala b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/SparkDeclarativePipelinesServerSuite.scala
index ec06d15ad8db..4ada35c7629e 100644
--- a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/SparkDeclarativePipelinesServerSuite.scala
+++ b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/SparkDeclarativePipelinesServerSuite.scala
@@ -26,6 +26,8 @@ import org.apache.spark.connect.proto.{Expression, OutputType, PipelineCommand,
 import org.apache.spark.connect.proto.PipelineCommand.{DefineFlow, DefineOutput}
 import org.apache.spark.internal.Logging
 import org.apache.spark.sql.connect.service.{SessionKey, SparkConnectService}
+import org.apache.spark.sql.pipelines.autocdc.{ColumnSelection, UnqualifiedColumnName}
+import org.apache.spark.sql.pipelines.graph.AutoCdcFlow
 
 class SparkDeclarativePipelinesServerSuite
     extends SparkDeclarativePipelinesServerTest
@@ -133,7 +135,10 @@ class SparkDeclarativePipelinesServerSuite
       columns: Seq[String] = Seq.empty,
       exceptColumns: Seq[String] = Seq.empty,
       trackHistoryColumns: Seq[String] = Seq.empty,
-      trackHistoryExceptColumns: Seq[String] = Seq.empty): DefineFlow = {
+      trackHistoryExceptColumns: Seq[String] = Seq.empty,
+      ignoreNullUpdates: Boolean = false,
+      ignoreNullUpdatesColumns: Seq[String] = Seq.empty,
+      ignoreNullUpdatesExceptColumns: Seq[String] = Seq.empty): DefineFlow = {
     val autoCdcDetails = DefineFlow.AutoCdcFlowDetails
       .newBuilder()
       .setSource("src")
@@ -146,6 +151,11 @@ class SparkDeclarativePipelinesServerSuite
       autoCdcDetails.addTrackHistoryColumnList(unresolvedColumn(c)))
     trackHistoryExceptColumns.foreach(c =>
       autoCdcDetails.addTrackHistoryExceptColumnList(unresolvedColumn(c)))
+    if (ignoreNullUpdates) autoCdcDetails.setIgnoreNullUpdates(true)
+    ignoreNullUpdatesColumns.foreach(c =>
+      autoCdcDetails.addIgnoreNullUpdatesColumnList(unresolvedColumn(c)))
+    ignoreNullUpdatesExceptColumns.foreach(c =>
+      autoCdcDetails.addIgnoreNullUpdatesExceptColumnList(unresolvedColumn(c)))
     DefineFlow
       .newBuilder()
       .setDataflowGraphId(graphId)
@@ -214,6 +224,117 @@ class SparkDeclarativePipelinesServerSuite
       ex.getMessage.contains("AUTOCDC_BOTH_TRACK_HISTORY_COLUMN_LIST_AND_EXCEPT_COLUMN_LIST"))
   }
 
+  // Retrieves the (unresolved) AutoCdcFlow registered for the given graph so its ChangeArgs can be
+  // inspected. The graph must be non-empty (its target table defined), so build() succeeds.
+  private def registeredAutoCdcFlow(graphId: String): AutoCdcFlow =
+    getDefaultSessionHolder.dataflowGraphRegistry
+      .getDataflowGraphOrThrow(graphId)
+      .toDataflowGraph
+      .flows
+      .collectFirst { case f: AutoCdcFlow => f }
+      .getOrElse(fail("Expected an AutoCdcFlow in the graph"))
+
+  // Defines the "target" streaming table the AutoCDC flows below write into, so the graph is not
+  // empty when inspected.
+  private def defineTargetTable(graphId: String)(implicit
+      stub: proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub): Unit =
+    sendPlan(
+      buildPlanFromPipelineCommand(
+        PipelineCommand
+          .newBuilder()
+          .setDefineOutput(
+            DefineOutput
+              .newBuilder()
+              .setDataflowGraphId(graphId)
+              .setOutputName("target")
+              .setOutputType(OutputType.TABLE))
+          .build()))
+
+  test("AutoCDC: ignore_null_updates flag maps to an all-columns ignore-null selection") {
+    withRawBlockingStub { implicit stub =>
+      val graphId = createDataflowGraph
+      defineTargetTable(graphId)
+      sendPlan(
+        buildPlanFromPipelineCommand(
+          PipelineCommand
+            .newBuilder()
+            .setDefineFlow(
+              autoCdcDefineFlow(graphId, DefineFlow.SCDType.SCD_TYPE_1, ignoreNullUpdates = true))
+            .build()))
+      // "All columns" is an ExcludeColumns selection with an empty list.
+      assert(
+        registeredAutoCdcFlow(graphId).changeArgs.ignoreNullSelection
+          .contains(ColumnSelection.ExcludeColumns(Seq.empty)))
+    }
+  }
+
+  test("AutoCDC: ignore_null_updates_column_list maps to an include ignore-null selection") {
+    withRawBlockingStub { implicit stub =>
+      val graphId = createDataflowGraph
+      defineTargetTable(graphId)
+      sendPlan(
+        buildPlanFromPipelineCommand(
+          PipelineCommand
+            .newBuilder()
+            .setDefineFlow(autoCdcDefineFlow(
+              graphId,
+              DefineFlow.SCDType.SCD_TYPE_1,
+              ignoreNullUpdatesColumns = Seq("val")))
+            .build()))
+      assert(
+        registeredAutoCdcFlow(graphId).changeArgs.ignoreNullSelection
+          .contains(ColumnSelection.IncludeColumns(Seq(UnqualifiedColumnName("val")))))
+    }
+  }
+
+  test(
+    "AutoCDC: ignore_null_updates_except_column_list maps to an except ignore-null selection") {
+    withRawBlockingStub { implicit stub =>
+      val graphId = createDataflowGraph
+      defineTargetTable(graphId)
+      sendPlan(
+        buildPlanFromPipelineCommand(
+          PipelineCommand
+            .newBuilder()
+            .setDefineFlow(autoCdcDefineFlow(
+              graphId,
+              DefineFlow.SCDType.SCD_TYPE_1,
+              ignoreNullUpdatesExceptColumns = Seq("ts")))
+            .build()))
+      assert(
+        registeredAutoCdcFlow(graphId).changeArgs.ignoreNullSelection
+          .contains(ColumnSelection.ExcludeColumns(Seq(UnqualifiedColumnName("ts")))))
+    }
+  }
+
+  // Every combination of more than one of the three mutually-exclusive ignore-null options
+  // (all-columns flag, include list, except list) must be rejected server-side.
+  namedGridTest("AutoCDC: conflicting ignore-null options are rejected server-side")(
+    Map(
+      "(all + column_list)" -> ((true, Seq("val"), Seq.empty[String])),
+      "(all + except_column_list)" -> ((true, Seq.empty[String], Seq("ts"))),
+      "(column_list + except_column_list)" -> ((false, Seq("val"), Seq("ts"))),
+      "(all + column_list + except_column_list)" -> ((true, Seq("val"), Seq("ts"))))) {
+    case (ignoreNullUpdates, columns, exceptColumns) =>
+      val ex = intercept[Exception] {
+        withRawBlockingStub { implicit stub =>
+          val graphId = createDataflowGraph
+          sendPlan(
+            buildPlanFromPipelineCommand(
+              PipelineCommand
+                .newBuilder()
+                .setDefineFlow(autoCdcDefineFlow(
+                  graphId,
+                  DefineFlow.SCDType.SCD_TYPE_1,
+                  ignoreNullUpdates = ignoreNullUpdates,
+                  ignoreNullUpdatesColumns = columns,
+                  ignoreNullUpdatesExceptColumns = exceptColumns))
+                .build()))
+        }
+      }
+      assert(ex.getMessage.contains("AUTOCDC_CONFLICTING_IGNORE_NULL_UPDATES_OPTIONS"))
+  }
+
   test(
     "Cross dependency between SQL dataset and non-SQL dataset is valid and can be registered") {
     withRawBlockingStub { implicit stub =>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org