(spark) branch branch-4.x updated: [SPARK-56712][SQL][DOCS] Document pushdown contract for CDC ChangelogTable

[ge...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gengliangwang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 18100678905c [SPARK-56712][SQL][DOCS] Document pushdown contract for CDC ChangelogTable
18100678905c is described below

commit 18100678905c793cd21c4c57c91c708a9d424172
Author: Gengliang Wang <ge...@apache.org>
AuthorDate: Mon May 4 11:58:40 2026 -0700

    [SPARK-56712][SQL][DOCS] Document pushdown contract for CDC ChangelogTable
    
    ### What changes were proposed in this pull request?
    
    Document the predicate pushdown and column pruning contract for CDC `Changelog` connectors in the `Changelog` Javadoc.
    
    When any post-processing pass applies (carry-over removal, update detection, or netChanges):
    - Spark only pushes predicates that reference `_commit_version`, `_commit_timestamp`, or columns named by `rowId()` to the connector's `SupportsPushDownFilters` / `SupportsPushDownV2Filters`.
    - Predicates on `_change_type`, the `rowVersion()` column, or data columns are kept above the scan; pushing them would drop one half of a delete/insert pair within a row-identity group and silently break post-processing.
    
    When no post-processing pass applies, pushdown is unrestricted.
    
    `SupportsPushDownRequiredColumns` (column pruning) is unrestricted in either case — Spark's pruning already respects what the rewrite operators reference, so connectors should serve whatever required-column set Spark requests.
    
    The restriction is enforced by the rewrite shape itself: a `Window` / `Aggregate` / `TransformWithState` keyed on the safe columns sits between the relation and the user's filter, so Catalyst's predicate-pushdown rules naturally block unsafe pushes. Connectors do not need to code this restriction themselves but must not bypass it (e.g. by self-applying filters from connector-specific options).
    
    This is a sub-task of SPARK-55668.
    
    ### Why are the changes needed?
    
    The contract was implicit. A connector author reading the Javadoc could reasonably implement `SupportsPushDownFilters` and accept all predicates, including unsafe ones, expecting Spark to handle the rest. Spelling out which predicates the connector actually needs to handle (and why others are intentionally never delivered) prevents accidental misuse and explains the asymmetry to anyone debugging an unexpected post-scan filter. Documenting the column pruning side keeps the two related  [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Documentation only. No behavior change.
    
    ### How was this patch tested?
    
    `Xdoclint:html,syntax,accessibility` is clean on `Changelog.java`. No code changed; existing CDC test suites unaffected.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude opus-4-7
    
    Closes #55664 from gengliangwang/SPARK-56712-restrict-cdc-pushdown.
    
    Authored-by: Gengliang Wang <ge...@apache.org>
    Signed-off-by: Gengliang Wang <ge...@apache.org>
    (cherry picked from commit ebfdf8b6b448b3c880a732290bfa02b515e32027)
    Signed-off-by: Gengliang Wang <ge...@apache.org>
---
 .../apache/spark/sql/connector/catalog/Changelog.java  | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java
index 8e2384b6b133..2ef0846ab800 100644
--- a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java
+++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java
@@ -75,6 +75,24 @@ import org.apache.spark.sql.util.CaseInsensitiveStringMap;
  * row identities only touched in the latest observed commit are held back until either a
  * later commit (with strictly greater `_commit_timestamp`) advances the global watermark
  * past them, or the source terminates.
+ * <p>
+ * <b>Pushdown contract.</b> When any post-processing pass applies (carry-over
+ * removal, update detection, or netChanges), Spark only pushes predicates
+ * that reference {@code _commit_version}, {@code _commit_timestamp}, or
+ * columns named by {@link #rowId()} to the connector's
+ * {@link org.apache.spark.sql.connector.read.SupportsPushDownFilters} /
+ * {@link org.apache.spark.sql.connector.read.SupportsPushDownV2Filters}.
+ * Predicates on {@code _change_type}, the {@link #rowVersion()} column, or
+ * non-rowId data columns are kept above the scan: pushing them would drop
+ * one half of a delete/insert pair within a row-identity group and silently
+ * break post-processing. Catalyst's pushdown rules enforce this via the
+ * rewrite operators, so connectors do not need to code the restriction
+ * themselves -- but must not bypass it via connector-specific options. When
+ * no post-processing pass applies, Spark does not impose any CDC-specific
+ * predicate-pushdown restriction.
+ * {@link org.apache.spark.sql.connector.read.SupportsPushDownRequiredColumns}
+ * (column pruning) is unrestricted in either case: Spark's pruning already
+ * respects what the rewrite operators reference.
  *
  * @since 4.2.0
  */


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org