You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sz...@apache.org on 2026/09/15 19:55:21 UTC
(spark) branch master updated: [SPARK-59446][CONNECT] Support ignore-null updates in Auto CDC flows over Spark Connect
This is an automated email from the ASF dual-hosted git repository.
szehon-ho pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6077852d08a9 [SPARK-59446][CONNECT] Support ignore-null updates in Auto CDC flows over Spark Connect
6077852d08a9 is described below
commit 6077852d08a90ea0907abe868aa3e21d318e08f6
Author: Andreas Neumann <an...@apache.org>
AuthorDate: Tue Sep 15 12:55:08 2026 -0700
[SPARK-59446][CONNECT] Support ignore-null updates in Auto CDC flows over Spark Connect
### What changes were proposed in this pull request?
Adds partial-update ("ignore null updates") support to Auto CDC flows defined over Spark Connect. Partial updates keep the existing target value for a column when an incoming CDC upsert carries `NULL` for it, instead of overwriting with `NULL`.
- Adds an `optional bool ignore_null_updates` field (field 16) to the `PipelineCommand.DefineFlow.AutoCdcFlowDetails` message, and regenerates the Python protobuf stubs.
- Wires all three ignore-null inputs through the server handler (`PipelinesHandler.buildAutoCdcFlow`) into `ChangeArgs.ignoreNullSelection`, mirroring the existing `track_history_*` handling:
- `ignore_null_updates` (all columns) -> `ColumnSelection.ExcludeColumns(Seq.empty)`
- `ignore_null_updates_column_list` -> `ColumnSelection.IncludeColumns`
- `ignore_null_updates_except_column_list` -> `ColumnSelection.ExcludeColumns`
- The three inputs are mutually exclusive; specifying more than one raises the new error condition `AUTOCDC_CONFLICTING_IGNORE_NULL_UPDATES_OPTIONS`.
This removes the `TODO(SPARK-57093)` in the handler, which had left the two previously-declared column-list fields (14/15) unread.
### Why are the changes needed?
Auto CDC "ignore null updates" has three user-visible states: off, a column subset, and all columns. The engine already models these as `ChangeArgs.ignoreNullSelection: Option[ColumnSelection]`, where "all columns" is `ExcludeColumns(Seq.empty)`, but the Connect handler discarded the two existing column-list fields behind a TODO, so none of it was reachable from a Connect client.
The two repeated fields can express the subset cases but not "all columns": an empty repeated field is indistinguishable from an unset one on the wire, so an empty except-list collapses to "unset" (off). Expressing "ignore nulls on all columns" therefore requires an explicit scalar flag, `ignore_null_updates`.
### Does this PR introduce _any_ user-facing change?
No. Declarative pipeline flows are defined via Spark Connect; the Python API surface that sets these fields is added in a follow-up. The engine behavior itself is unchanged.
### How was this patch tested?
New tests in `SparkDeclarativePipelinesServerSuite` drive the raw Connect stub and assert that the all-columns flag and an include list map to the expected `ignoreNullSelection`, and that conflicting options are rejected server-side. `SparkThrowableSuite` validates the new error condition.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Opus 4.8
Closes #58784 from anew/autocdc-ignore-null-updates-all-proto.
Authored-by: Andreas Neumann <an...@apache.org>
Signed-off-by: Szehon Ho <sz...@gmail.com>
---
.../src/main/resources/error/error-conditions.json | 6 +
python/pyspark/sql/connect/proto/pipelines_pb2.py | 76 ++++++-------
python/pyspark/sql/connect/proto/pipelines_pb2.pyi | 21 ++++
.../main/protobuf/spark/connect/pipelines.proto | 4 +
.../sql/connect/pipelines/PipelinesHandler.scala | 29 ++++-
.../SparkDeclarativePipelinesServerSuite.scala | 123 ++++++++++++++++++++-
6 files changed, 216 insertions(+), 43 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json
index f52a28b00866..f355f13b8f2e 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -233,6 +233,12 @@
],
"sqlState" : "42703"
},
+ "AUTOCDC_CONFLICTING_IGNORE_NULL_UPDATES_OPTIONS" : {
+ "message" : [
+ "AutoCDC flow specifies more than one of `ignore_null_updates`, `ignore_null_updates_column_list`, and `ignore_null_updates_except_column_list`; at most one may be provided."
+ ],
+ "sqlState" : "42613"
+ },
"AUTOCDC_EMPTY_KEYS" : {
"message" : [
"AutoCDC requires at least one key column to identify rows, but received an empty key set."
diff --git a/python/pyspark/sql/connect/proto/pipelines_pb2.py b/python/pyspark/sql/connect/proto/pipelines_pb2.py
index c32f27f26cdc..d68abadac519 100644
--- a/python/pyspark/sql/connect/proto/pipelines_pb2.py
+++ b/python/pyspark/sql/connect/proto/pipelines_pb2.py
@@ -44,7 +44,7 @@ from pyspark.sql.connect.proto import types_pb2 as spark_dot_connect_dot_types__
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
- b'\n\x1dspark/connect/pipelines.proto\x12\rspark.connect\x1a\x19google/protobuf/any.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1aspark/connect/common.proto\x1a\x1fspark/connect/expressions.proto\x1a\x1dspark/connect/relations.proto\x1a\x19spark/connect/types.proto"\xcc\x30\n\x0fPipelineCommand\x12h\n\x15\x63reate_dataflow_graph\x18\x01 \x01(\x0b\x32\x32.spark.connect.PipelineCommand.CreateDataflowGraphH\x00R\x13\x63reateDataflowGraph\x12R\n\rdefine_output\x18\x02 \x01(\x0b\x32 [...]
+ b'\n\x1dspark/connect/pipelines.proto\x12\rspark.connect\x1a\x19google/protobuf/any.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1aspark/connect/common.proto\x1a\x1fspark/connect/expressions.proto\x1a\x1dspark/connect/relations.proto\x1a\x19spark/connect/types.proto"\x99\x31\n\x0fPipelineCommand\x12h\n\x15\x63reate_dataflow_graph\x18\x01 \x01(\x0b\x32\x32.spark.connect.PipelineCommand.CreateDataflowGraphH\x00R\x13\x63reateDataflowGraph\x12R\n\rdefine_output\x18\x02 \x01(\x0b\x32 [...]
)
_globals = globals()
@@ -87,10 +87,10 @@ if not _descriptor._USE_C_DESCRIPTORS:
_globals["_PIPELINEANALYSISCONTEXT"].fields_by_name[
"flow_name"
]._serialized_options = b"\030\001"
- _globals["_OUTPUTTYPE"]._serialized_start = 8167
- _globals["_OUTPUTTYPE"]._serialized_end = 8272
+ _globals["_OUTPUTTYPE"]._serialized_start = 8244
+ _globals["_OUTPUTTYPE"]._serialized_end = 8349
_globals["_PIPELINECOMMAND"]._serialized_start = 228
- _globals["_PIPELINECOMMAND"]._serialized_end = 6448
+ _globals["_PIPELINECOMMAND"]._serialized_end = 6525
_globals["_PIPELINECOMMAND_CREATEDATAFLOWGRAPH"]._serialized_start = 1265
_globals["_PIPELINECOMMAND_CREATEDATAFLOWGRAPH"]._serialized_end = 1573
_globals["_PIPELINECOMMAND_CREATEDATAFLOWGRAPH_SQLCONFENTRY"]._serialized_start = 1474
@@ -112,43 +112,43 @@ if not _descriptor._USE_C_DESCRIPTORS:
_globals["_PIPELINECOMMAND_DEFINEOUTPUT_SINKDETAILS_OPTIONSENTRY"]._serialized_start = 2795
_globals["_PIPELINECOMMAND_DEFINEOUTPUT_SINKDETAILS_OPTIONSENTRY"]._serialized_end = 2853
_globals["_PIPELINECOMMAND_DEFINEFLOW"]._serialized_start = 2969
- _globals["_PIPELINECOMMAND_DEFINEFLOW"]._serialized_end = 5056
+ _globals["_PIPELINECOMMAND_DEFINEFLOW"]._serialized_end = 5133
_globals["_PIPELINECOMMAND_DEFINEFLOW_SQLCONFENTRY"]._serialized_start = 1474
_globals["_PIPELINECOMMAND_DEFINEFLOW_SQLCONFENTRY"]._serialized_end = 1532
_globals["_PIPELINECOMMAND_DEFINEFLOW_WRITERELATIONFLOWDETAILS"]._serialized_start = 3703
_globals["_PIPELINECOMMAND_DEFINEFLOW_WRITERELATIONFLOWDETAILS"]._serialized_end = 3800
_globals["_PIPELINECOMMAND_DEFINEFLOW_AUTOCDCFLOWDETAILS"]._serialized_start = 3803
- _globals["_PIPELINECOMMAND_DEFINEFLOW_AUTOCDCFLOWDETAILS"]._serialized_end = 4808
- _globals["_PIPELINECOMMAND_DEFINEFLOW_RESPONSE"]._serialized_start = 4810
- _globals["_PIPELINECOMMAND_DEFINEFLOW_RESPONSE"]._serialized_end = 4868
- _globals["_PIPELINECOMMAND_DEFINEFLOW_SCDTYPE"]._serialized_start = 4870
- _globals["_PIPELINECOMMAND_DEFINEFLOW_SCDTYPE"]._serialized_end = 4937
- _globals["_PIPELINECOMMAND_EXECUTEOUTPUTFLOWS"]._serialized_start = 5059
- _globals["_PIPELINECOMMAND_EXECUTEOUTPUTFLOWS"]._serialized_end = 5415
- _globals["_PIPELINECOMMAND_STARTRUN"]._serialized_start = 5418
- _globals["_PIPELINECOMMAND_STARTRUN"]._serialized_end = 5740
- _globals["_PIPELINECOMMAND_DEFINESQLGRAPHELEMENTS"]._serialized_start = 5743
- _globals["_PIPELINECOMMAND_DEFINESQLGRAPHELEMENTS"]._serialized_end = 5942
- _globals["_PIPELINECOMMAND_GETQUERYFUNCTIONEXECUTIONSIGNALSTREAM"]._serialized_start = 5945
- _globals["_PIPELINECOMMAND_GETQUERYFUNCTIONEXECUTIONSIGNALSTREAM"]._serialized_end = 6103
- _globals["_PIPELINECOMMAND_DEFINEFLOWQUERYFUNCTIONRESULT"]._serialized_start = 6106
- _globals["_PIPELINECOMMAND_DEFINEFLOWQUERYFUNCTIONRESULT"]._serialized_end = 6432
- _globals["_PIPELINECOMMANDRESULT"]._serialized_start = 6451
- _globals["_PIPELINECOMMANDRESULT"]._serialized_end = 7203
- _globals["_PIPELINECOMMANDRESULT_CREATEDATAFLOWGRAPHRESULT"]._serialized_start = 6820
- _globals["_PIPELINECOMMANDRESULT_CREATEDATAFLOWGRAPHRESULT"]._serialized_end = 6918
- _globals["_PIPELINECOMMANDRESULT_DEFINEOUTPUTRESULT"]._serialized_start = 6921
- _globals["_PIPELINECOMMANDRESULT_DEFINEOUTPUTRESULT"]._serialized_end = 7054
- _globals["_PIPELINECOMMANDRESULT_DEFINEFLOWRESULT"]._serialized_start = 7057
- _globals["_PIPELINECOMMANDRESULT_DEFINEFLOWRESULT"]._serialized_end = 7188
- _globals["_PIPELINEEVENTRESULT"]._serialized_start = 7205
- _globals["_PIPELINEEVENTRESULT"]._serialized_end = 7278
- _globals["_PIPELINEEVENT"]._serialized_start = 7280
- _globals["_PIPELINEEVENT"]._serialized_end = 7396
- _globals["_SOURCECODELOCATION"]._serialized_start = 7399
- _globals["_SOURCECODELOCATION"]._serialized_end = 7640
- _globals["_PIPELINEQUERYFUNCTIONEXECUTIONSIGNAL"]._serialized_start = 7643
- _globals["_PIPELINEQUERYFUNCTIONEXECUTIONSIGNAL"]._serialized_end = 7794
- _globals["_PIPELINEANALYSISCONTEXT"]._serialized_start = 7797
- _globals["_PIPELINEANALYSISCONTEXT"]._serialized_end = 8165
+ _globals["_PIPELINECOMMAND_DEFINEFLOW_AUTOCDCFLOWDETAILS"]._serialized_end = 4885
+ _globals["_PIPELINECOMMAND_DEFINEFLOW_RESPONSE"]._serialized_start = 4887
+ _globals["_PIPELINECOMMAND_DEFINEFLOW_RESPONSE"]._serialized_end = 4945
+ _globals["_PIPELINECOMMAND_DEFINEFLOW_SCDTYPE"]._serialized_start = 4947
+ _globals["_PIPELINECOMMAND_DEFINEFLOW_SCDTYPE"]._serialized_end = 5014
+ _globals["_PIPELINECOMMAND_EXECUTEOUTPUTFLOWS"]._serialized_start = 5136
+ _globals["_PIPELINECOMMAND_EXECUTEOUTPUTFLOWS"]._serialized_end = 5492
+ _globals["_PIPELINECOMMAND_STARTRUN"]._serialized_start = 5495
+ _globals["_PIPELINECOMMAND_STARTRUN"]._serialized_end = 5817
+ _globals["_PIPELINECOMMAND_DEFINESQLGRAPHELEMENTS"]._serialized_start = 5820
+ _globals["_PIPELINECOMMAND_DEFINESQLGRAPHELEMENTS"]._serialized_end = 6019
+ _globals["_PIPELINECOMMAND_GETQUERYFUNCTIONEXECUTIONSIGNALSTREAM"]._serialized_start = 6022
+ _globals["_PIPELINECOMMAND_GETQUERYFUNCTIONEXECUTIONSIGNALSTREAM"]._serialized_end = 6180
+ _globals["_PIPELINECOMMAND_DEFINEFLOWQUERYFUNCTIONRESULT"]._serialized_start = 6183
+ _globals["_PIPELINECOMMAND_DEFINEFLOWQUERYFUNCTIONRESULT"]._serialized_end = 6509
+ _globals["_PIPELINECOMMANDRESULT"]._serialized_start = 6528
+ _globals["_PIPELINECOMMANDRESULT"]._serialized_end = 7280
+ _globals["_PIPELINECOMMANDRESULT_CREATEDATAFLOWGRAPHRESULT"]._serialized_start = 6897
+ _globals["_PIPELINECOMMANDRESULT_CREATEDATAFLOWGRAPHRESULT"]._serialized_end = 6995
+ _globals["_PIPELINECOMMANDRESULT_DEFINEOUTPUTRESULT"]._serialized_start = 6998
+ _globals["_PIPELINECOMMANDRESULT_DEFINEOUTPUTRESULT"]._serialized_end = 7131
+ _globals["_PIPELINECOMMANDRESULT_DEFINEFLOWRESULT"]._serialized_start = 7134
+ _globals["_PIPELINECOMMANDRESULT_DEFINEFLOWRESULT"]._serialized_end = 7265
+ _globals["_PIPELINEEVENTRESULT"]._serialized_start = 7282
+ _globals["_PIPELINEEVENTRESULT"]._serialized_end = 7355
+ _globals["_PIPELINEEVENT"]._serialized_start = 7357
+ _globals["_PIPELINEEVENT"]._serialized_end = 7473
+ _globals["_SOURCECODELOCATION"]._serialized_start = 7476
+ _globals["_SOURCECODELOCATION"]._serialized_end = 7717
+ _globals["_PIPELINEQUERYFUNCTIONEXECUTIONSIGNAL"]._serialized_start = 7720
+ _globals["_PIPELINEQUERYFUNCTIONEXECUTIONSIGNAL"]._serialized_end = 7871
+ _globals["_PIPELINEANALYSISCONTEXT"]._serialized_start = 7874
+ _globals["_PIPELINEANALYSISCONTEXT"]._serialized_end = 8242
# @@protoc_insertion_point(module_scope)
diff --git a/python/pyspark/sql/connect/proto/pipelines_pb2.pyi b/python/pyspark/sql/connect/proto/pipelines_pb2.pyi
index 70e04ad85042..869efdab1d6b 100644
--- a/python/pyspark/sql/connect/proto/pipelines_pb2.pyi
+++ b/python/pyspark/sql/connect/proto/pipelines_pb2.pyi
@@ -594,6 +594,7 @@ class PipelineCommand(google.protobuf.message.Message):
TRACK_HISTORY_EXCEPT_COLUMN_LIST_FIELD_NUMBER: builtins.int
IGNORE_NULL_UPDATES_COLUMN_LIST_FIELD_NUMBER: builtins.int
IGNORE_NULL_UPDATES_EXCEPT_COLUMN_LIST_FIELD_NUMBER: builtins.int
+ IGNORE_NULL_UPDATES_FIELD_NUMBER: builtins.int
source: builtins.str
"""The name of the CDC source to stream from."""
@property
@@ -660,6 +661,10 @@ class PipelineCommand(google.protobuf.message.Message):
pyspark.sql.connect.proto.expressions_pb2.Expression
]:
"""Subset of columns excluded from ignoring null in updates."""
+ ignore_null_updates: builtins.bool
+ """When true, ignore null updates on all columns of the target table. Mutually exclusive
+ with ignore_null_updates_column_list and ignore_null_updates_except_column_list.
+ """
def __init__(
self,
*,
@@ -695,6 +700,7 @@ class PipelineCommand(google.protobuf.message.Message):
pyspark.sql.connect.proto.expressions_pb2.Expression
]
| None = ...,
+ ignore_null_updates: builtins.bool | None = ...,
) -> None: ...
def HasField(
self,
@@ -703,6 +709,8 @@ class PipelineCommand(google.protobuf.message.Message):
b"_apply_as_deletes",
"_apply_as_truncates",
b"_apply_as_truncates",
+ "_ignore_null_updates",
+ b"_ignore_null_updates",
"_sequence_by",
b"_sequence_by",
"_source",
@@ -711,6 +719,8 @@ class PipelineCommand(google.protobuf.message.Message):
b"apply_as_deletes",
"apply_as_truncates",
b"apply_as_truncates",
+ "ignore_null_updates",
+ b"ignore_null_updates",
"sequence_by",
b"sequence_by",
"source",
@@ -724,6 +734,8 @@ class PipelineCommand(google.protobuf.message.Message):
b"_apply_as_deletes",
"_apply_as_truncates",
b"_apply_as_truncates",
+ "_ignore_null_updates",
+ b"_ignore_null_updates",
"_sequence_by",
b"_sequence_by",
"_source",
@@ -736,6 +748,8 @@ class PipelineCommand(google.protobuf.message.Message):
b"column_list",
"except_column_list",
b"except_column_list",
+ "ignore_null_updates",
+ b"ignore_null_updates",
"ignore_null_updates_column_list",
b"ignore_null_updates_column_list",
"ignore_null_updates_except_column_list",
@@ -767,6 +781,13 @@ class PipelineCommand(google.protobuf.message.Message):
],
) -> typing_extensions.Literal["apply_as_truncates"] | None: ...
@typing.overload
+ def WhichOneof(
+ self,
+ oneof_group: typing_extensions.Literal[
+ "_ignore_null_updates", b"_ignore_null_updates"
+ ],
+ ) -> typing_extensions.Literal["ignore_null_updates"] | None: ...
+ @typing.overload
def WhichOneof(
self, oneof_group: typing_extensions.Literal["_sequence_by", b"_sequence_by"]
) -> typing_extensions.Literal["sequence_by"] | None: ...
diff --git a/sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto b/sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto
index 7bfc5ca39826..56fe5ccb1f0f 100644
--- a/sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto
+++ b/sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto
@@ -196,6 +196,10 @@ message PipelineCommand {
// Subset of columns excluded from ignoring null in updates.
repeated Expression ignore_null_updates_except_column_list = 15;
+ // When true, ignore null updates on all columns of the target table. Mutually exclusive
+ // with ignore_null_updates_column_list and ignore_null_updates_except_column_list.
+ optional bool ignore_null_updates = 16;
+
}
// SCD Type for Auto CDC target tables.
diff --git a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
index 718d1670bf8c..eb888287dace 100644
--- a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
+++ b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
@@ -430,9 +430,6 @@ private[connect] object PipelinesHandler extends Logging {
transformExpressionFunc: proto.Expression => Expression): AutoCdcFlow = {
// TODO(SPARK-57092): apply_as_truncates is declared on AutoCdcFlowDetails but is not yet
// honored by the engine; wire it through once SCD1 truncate support lands.
- // TODO(SPARK-57093): ignore_null_updates_column_list and ignore_null_updates_except_column_list
- // are declared on AutoCdcFlowDetails but are not yet honored by the engine; wire them
- // through once SCD1 ignore-null support lands.
if (!autoCdcDetails.hasSource) {
throw new AnalysisException("AUTOCDC_MISSING_SOURCE", Map.empty)
@@ -512,6 +509,29 @@ private[connect] object PipelinesHandler extends Logging {
throw new AnalysisException("AUTOCDC_TRACK_HISTORY_REQUIRES_SCD2", Map.empty)
}
+ // Ignore-null (partial update) selection. The three inputs are mutually exclusive:
+ // ignore_null_updates (all columns), ignore_null_updates_column_list (an include list), and
+ // ignore_null_updates_except_column_list (an except list). "All columns" is represented as an
+ // ExcludeColumns selection with an empty list; this cannot be expressed via the repeated
+ // except field alone (an empty repeated field is indistinguishable from an unset one), which
+ // is why the dedicated boolean exists.
+ val ignoreNullSelection: Option[ColumnSelection] = {
+ val all = autoCdcDetails.getIgnoreNullUpdates
+ val included = autoCdcDetails.getIgnoreNullUpdatesColumnListList.asScala.toSeq
+ val excluded = autoCdcDetails.getIgnoreNullUpdatesExceptColumnListList.asScala.toSeq
+ if (Seq(all, included.nonEmpty, excluded.nonEmpty).count(identity) > 1) {
+ throw new AnalysisException("AUTOCDC_CONFLICTING_IGNORE_NULL_UPDATES_OPTIONS", Map.empty)
+ } else if (all) {
+ Some(ColumnSelection.ExcludeColumns(Seq.empty))
+ } else if (included.nonEmpty) {
+ Some(ColumnSelection.IncludeColumns(included.map(asUnqualifiedColumnName)))
+ } else if (excluded.nonEmpty) {
+ Some(ColumnSelection.ExcludeColumns(excluded.map(asUnqualifiedColumnName)))
+ } else {
+ None
+ }
+ }
+
val changeArgs = ChangeArgs(
keys = keys,
sequencing = toColumn(autoCdcDetails.getSequenceBy),
@@ -519,7 +539,8 @@ private[connect] object PipelinesHandler extends Logging {
deleteCondition =
Option.when(autoCdcDetails.hasApplyAsDeletes)(toColumn(autoCdcDetails.getApplyAsDeletes)),
columnSelection = columnSelection,
- trackHistorySelection = trackHistorySelection)
+ trackHistorySelection = trackHistorySelection,
+ ignoreNullSelection = ignoreNullSelection)
AutoCdcFlow(
identifier = flowIdentifier,
diff --git a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/SparkDeclarativePipelinesServerSuite.scala b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/SparkDeclarativePipelinesServerSuite.scala
index ec06d15ad8db..4ada35c7629e 100644
--- a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/SparkDeclarativePipelinesServerSuite.scala
+++ b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/SparkDeclarativePipelinesServerSuite.scala
@@ -26,6 +26,8 @@ import org.apache.spark.connect.proto.{Expression, OutputType, PipelineCommand,
import org.apache.spark.connect.proto.PipelineCommand.{DefineFlow, DefineOutput}
import org.apache.spark.internal.Logging
import org.apache.spark.sql.connect.service.{SessionKey, SparkConnectService}
+import org.apache.spark.sql.pipelines.autocdc.{ColumnSelection, UnqualifiedColumnName}
+import org.apache.spark.sql.pipelines.graph.AutoCdcFlow
class SparkDeclarativePipelinesServerSuite
extends SparkDeclarativePipelinesServerTest
@@ -133,7 +135,10 @@ class SparkDeclarativePipelinesServerSuite
columns: Seq[String] = Seq.empty,
exceptColumns: Seq[String] = Seq.empty,
trackHistoryColumns: Seq[String] = Seq.empty,
- trackHistoryExceptColumns: Seq[String] = Seq.empty): DefineFlow = {
+ trackHistoryExceptColumns: Seq[String] = Seq.empty,
+ ignoreNullUpdates: Boolean = false,
+ ignoreNullUpdatesColumns: Seq[String] = Seq.empty,
+ ignoreNullUpdatesExceptColumns: Seq[String] = Seq.empty): DefineFlow = {
val autoCdcDetails = DefineFlow.AutoCdcFlowDetails
.newBuilder()
.setSource("src")
@@ -146,6 +151,11 @@ class SparkDeclarativePipelinesServerSuite
autoCdcDetails.addTrackHistoryColumnList(unresolvedColumn(c)))
trackHistoryExceptColumns.foreach(c =>
autoCdcDetails.addTrackHistoryExceptColumnList(unresolvedColumn(c)))
+ if (ignoreNullUpdates) autoCdcDetails.setIgnoreNullUpdates(true)
+ ignoreNullUpdatesColumns.foreach(c =>
+ autoCdcDetails.addIgnoreNullUpdatesColumnList(unresolvedColumn(c)))
+ ignoreNullUpdatesExceptColumns.foreach(c =>
+ autoCdcDetails.addIgnoreNullUpdatesExceptColumnList(unresolvedColumn(c)))
DefineFlow
.newBuilder()
.setDataflowGraphId(graphId)
@@ -214,6 +224,117 @@ class SparkDeclarativePipelinesServerSuite
ex.getMessage.contains("AUTOCDC_BOTH_TRACK_HISTORY_COLUMN_LIST_AND_EXCEPT_COLUMN_LIST"))
}
+ // Retrieves the (unresolved) AutoCdcFlow registered for the given graph so its ChangeArgs can be
+ // inspected. The graph must be non-empty (its target table defined), so build() succeeds.
+ private def registeredAutoCdcFlow(graphId: String): AutoCdcFlow =
+ getDefaultSessionHolder.dataflowGraphRegistry
+ .getDataflowGraphOrThrow(graphId)
+ .toDataflowGraph
+ .flows
+ .collectFirst { case f: AutoCdcFlow => f }
+ .getOrElse(fail("Expected an AutoCdcFlow in the graph"))
+
+ // Defines the "target" streaming table the AutoCDC flows below write into, so the graph is not
+ // empty when inspected.
+ private def defineTargetTable(graphId: String)(implicit
+ stub: proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub): Unit =
+ sendPlan(
+ buildPlanFromPipelineCommand(
+ PipelineCommand
+ .newBuilder()
+ .setDefineOutput(
+ DefineOutput
+ .newBuilder()
+ .setDataflowGraphId(graphId)
+ .setOutputName("target")
+ .setOutputType(OutputType.TABLE))
+ .build()))
+
+ test("AutoCDC: ignore_null_updates flag maps to an all-columns ignore-null selection") {
+ withRawBlockingStub { implicit stub =>
+ val graphId = createDataflowGraph
+ defineTargetTable(graphId)
+ sendPlan(
+ buildPlanFromPipelineCommand(
+ PipelineCommand
+ .newBuilder()
+ .setDefineFlow(
+ autoCdcDefineFlow(graphId, DefineFlow.SCDType.SCD_TYPE_1, ignoreNullUpdates = true))
+ .build()))
+ // "All columns" is an ExcludeColumns selection with an empty list.
+ assert(
+ registeredAutoCdcFlow(graphId).changeArgs.ignoreNullSelection
+ .contains(ColumnSelection.ExcludeColumns(Seq.empty)))
+ }
+ }
+
+ test("AutoCDC: ignore_null_updates_column_list maps to an include ignore-null selection") {
+ withRawBlockingStub { implicit stub =>
+ val graphId = createDataflowGraph
+ defineTargetTable(graphId)
+ sendPlan(
+ buildPlanFromPipelineCommand(
+ PipelineCommand
+ .newBuilder()
+ .setDefineFlow(autoCdcDefineFlow(
+ graphId,
+ DefineFlow.SCDType.SCD_TYPE_1,
+ ignoreNullUpdatesColumns = Seq("val")))
+ .build()))
+ assert(
+ registeredAutoCdcFlow(graphId).changeArgs.ignoreNullSelection
+ .contains(ColumnSelection.IncludeColumns(Seq(UnqualifiedColumnName("val")))))
+ }
+ }
+
+ test(
+ "AutoCDC: ignore_null_updates_except_column_list maps to an except ignore-null selection") {
+ withRawBlockingStub { implicit stub =>
+ val graphId = createDataflowGraph
+ defineTargetTable(graphId)
+ sendPlan(
+ buildPlanFromPipelineCommand(
+ PipelineCommand
+ .newBuilder()
+ .setDefineFlow(autoCdcDefineFlow(
+ graphId,
+ DefineFlow.SCDType.SCD_TYPE_1,
+ ignoreNullUpdatesExceptColumns = Seq("ts")))
+ .build()))
+ assert(
+ registeredAutoCdcFlow(graphId).changeArgs.ignoreNullSelection
+ .contains(ColumnSelection.ExcludeColumns(Seq(UnqualifiedColumnName("ts")))))
+ }
+ }
+
+ // Every combination of more than one of the three mutually-exclusive ignore-null options
+ // (all-columns flag, include list, except list) must be rejected server-side.
+ namedGridTest("AutoCDC: conflicting ignore-null options are rejected server-side")(
+ Map(
+ "(all + column_list)" -> ((true, Seq("val"), Seq.empty[String])),
+ "(all + except_column_list)" -> ((true, Seq.empty[String], Seq("ts"))),
+ "(column_list + except_column_list)" -> ((false, Seq("val"), Seq("ts"))),
+ "(all + column_list + except_column_list)" -> ((true, Seq("val"), Seq("ts"))))) {
+ case (ignoreNullUpdates, columns, exceptColumns) =>
+ val ex = intercept[Exception] {
+ withRawBlockingStub { implicit stub =>
+ val graphId = createDataflowGraph
+ sendPlan(
+ buildPlanFromPipelineCommand(
+ PipelineCommand
+ .newBuilder()
+ .setDefineFlow(autoCdcDefineFlow(
+ graphId,
+ DefineFlow.SCDType.SCD_TYPE_1,
+ ignoreNullUpdates = ignoreNullUpdates,
+ ignoreNullUpdatesColumns = columns,
+ ignoreNullUpdatesExceptColumns = exceptColumns))
+ .build()))
+ }
+ }
+ assert(ex.getMessage.contains("AUTOCDC_CONFLICTING_IGNORE_NULL_UPDATES_OPTIONS"))
+ }
+
test(
"Cross dependency between SQL dataset and non-SQL dataset is valid and can be registered") {
withRawBlockingStub { implicit stub =>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org