(spark) branch branch-4.x updated: [SPARK-59572][SQL][FOLLOWUP] Clarify PartitionPredicate migration guide entry

[sz...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

szehon-ho pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 02faef4d9fab [SPARK-59572][SQL][FOLLOWUP] Clarify PartitionPredicate migration guide entry
02faef4d9fab is described below

commit 02faef4d9fab6b6086f9ada1cd635f4f6b87d343
Author: Szehon Ho <sz...@gmail.com>
AuthorDate: Thu Sep 17 18:04:38 2026 -0700

    [SPARK-59572][SQL][FOLLOWUP] Clarify PartitionPredicate migration guide entry
    
    ### What changes were proposed in this pull request?
    
    This follow-up expands the Spark 4.3 and 4.2.1 SQL migration guide entries for
    SPARK-59572. It explains that `PartitionPredicate` is sent during the second pass of
    iterative pushdown and adds concrete `SELECT` and metadata-only `DELETE` examples.
    
    This follows up https://github.com/apache/spark/pull/58851.
    
    ### Why are the changes needed?
    
    The existing entries state that evaluation failures now propagate, but do not make clear why
    returning `true` was a correctness bug. Once a data source accepts a static
    `PartitionPredicate`, Spark does not retain another evaluator for the condition. The same is
    true for metadata-only `DELETE`. The expanded examples show how the old behavior could return
    rows rejected by a filter or delete a partition that never matched the delete condition.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a documentation-only clarification of the behavior fixed by SPARK-59572.
    
    ### How was this patch tested?
    
    Not run (documentation-only change). `git diff --check` passes.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6)
    
    Closes #58902 from szehon-ho/codex/partition-predicate-doc-fix.
    
    Authored-by: Szehon Ho <sz...@gmail.com>
    Signed-off-by: Szehon Ho <sz...@gmail.com>
    (cherry picked from commit c35e98fb487224d07a1051a35a78ef1f5efe2740)
    Signed-off-by: Szehon Ho <sz...@gmail.com>
---
 docs/sql-migration-guide.md | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 7b0d73ce7a0d..d46b7545574d 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -59,11 +59,17 @@ license: |
 - Since Spark 4.3, the Spark Connect session errors `INVALID_HANDLE.SESSION_CHANGED`/`SESSION_CLOSED`/`SESSION_NOT_FOUND` carry SQLSTATE `08003` instead of `HY000`; the condition names are unchanged. Code matching these errors on SQLSTATE should match `08003` or class `08`.
 - Since Spark 4.3, [Declarative Pipelines](declarative-pipelines-programming-guide.html) honors `spark.sql.caseSensitive` when inferring and evolving pipeline table schemas. Under case-insensitive resolution (the default), column names that differ only in case now identify the same column: flows writing to one table contribute a single column rather than one per spelling, and a column that differs only in case from one already persisted in the target is written to that column instead of  [...]
 - Since Spark 4.3, all flows writing to the same pipeline table must agree on the effective `spark.sql.caseSensitive`, which each flow takes from its own SQL configuration (a `SET` in pipeline source, which never reaches the session) and otherwise from the session. A disagreement fails the update with `CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY`, because that value decides whether names differing only in case identify the same column and would otherwise make the table's schema depend on  [...]
-- Since Spark 4.3, when a data source cannot evaluate a partition predicate that Spark pushed down, for example because an ANSI cast of a partition value fails, the error is raised instead of the partition being reported as a match. Previously the failure was ignored, so a scan could return rows the filter does not accept and a metadata-only `DELETE` could remove a partition its condition never matched. Queries that appeared to succeed may now fail with the same error they raise without  [...]
+- Since Spark 4.3, Spark fixes a correctness bug in the Data Source V2 `PartitionPredicate` introduced in Spark 4.2.0. For scans, `PartitionPredicate` is the additional predicate type Spark sends during the second pass of iterative pushdown. When a data source opts into iterative pushdown by returning `true` from `supportsIterativePushdown()`, Spark may call its pushdown API again with a `PartitionPredicate` that evaluates a filter against the full partition key described by `Table.parti [...]
+  - For a `SELECT` over partitions `dep = 'hr'` and `dep = '1'` with the filter `CAST(dep AS INT) = 1`, evaluating the predicate for `dep = 'hr'` fails. Returning `true` kept that partition, and because Spark had removed the post-scan filter, the query incorrectly returned its rows.
+  - For a metadata-only `DELETE` with the same condition, returning `true` could delete the `dep = 'hr'` partition even though the condition never matched it.
+  Spark now propagates the evaluation error in these cases. Queries that previously returned an incorrect result may therefore fail, and the error can surface while the scan is built rather than while rows are read. Runtime filters are unaffected because Spark still evaluates their condition after pruning.
 
 ## Upgrading from Spark SQL 4.2.0 to 4.2.1
 
-- Since Spark 4.2.1, when a data source cannot evaluate a partition predicate that Spark pushed down, for example because an ANSI cast of a partition value fails, the error is raised instead of the partition being reported as a match. Previously the failure was ignored, so a scan could return rows the filter does not accept and a metadata-only `DELETE` could remove a partition its condition never matched. Queries that appeared to succeed may now fail with the same error they raise withou [...]
+- Spark 4.2.0 contains a correctness bug in the Data Source V2 `PartitionPredicate`. For scans, `PartitionPredicate` is the additional predicate type Spark sends during the second pass of iterative pushdown. When a data source opts into iterative pushdown by returning `true` from `supportsIterativePushdown()`, Spark may call its pushdown API again with a `PartitionPredicate` that evaluates a filter against the full partition key described by `Table.partitioning()`. Spark also uses `Parti [...]
+  - For a `SELECT` over partitions `dep = 'hr'` and `dep = '1'` with the filter `CAST(dep AS INT) = 1`, evaluating the predicate for `dep = 'hr'` fails. Returning `true` keeps that partition, and because Spark removes the post-scan filter, the query incorrectly returns its rows.
+  - For a metadata-only `DELETE` with the same condition, returning `true` can delete the `dep = 'hr'` partition even though the condition never matches it.
+  Spark 4.2.1 propagates the evaluation error in these cases. Queries that previously returned an incorrect result may therefore fail, and the error can surface while the scan is built rather than while rows are read. Runtime filters are unaffected because Spark still evaluates their condition after pruning.
 
 ## Upgrading from Spark SQL 4.1 to 4.2
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org