You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2026/09/16 09:28:09 UTC
[PR] [SPARK-59572][SQL] PartitionPredicate evaluation failure must not be reported as a partition match [spark]
pan3793 opened a new pull request, #58851:
URL: https://github.com/apache/spark/pull/58851
### What changes were proposed in this pull request?
`PartitionPredicateImpl.eval` reported a partition as matching whenever it could not evaluate
the predicate, either because evaluating the expression threw or because the partition key width
did not match the schema. It now propagates that failure.
The old behavior is kept for runtime filters only, where Spark evaluates the filter again in the
post-scan `FilterExec`, so failing open costs a pruning opportunity and nothing else.
`createRuntimePartitionPredicates` is the only caller that asks for it.
This also documents the `eval` contract for connectors, and corrects a comment in the in-memory
test table that stated the old justification as a general rule.
### Why are the changes needed?
Failing open is sound only while something else still evaluates the filter. Two paths have not
met that condition since `PartitionPredicate` was introduced in 4.2.0:
- the static second pass, where an accepted `PartitionPredicate` is removed from the post-scan
filters;
- the metadata-only DELETE rewrite, which has no post-scan filter at all.
On both, the predicate is Spark's only evaluator, so reporting a match on failure is a wrong
answer. On a table partitioned by `dep STRING` holding `'hr'` and `'1'`, with a filter on `dep`
whose evaluation fails for `'hr'`, an ANSI cast error for example:
- `SELECT ... WHERE <filter>` returns the `'hr'` row, which the filter does not accept, and
reports no error;
- `DELETE FROM ... WHERE <filter>` deletes the `'hr'` partition, which the condition never
accepted.
The second is silent data loss.
The invariant was identified during the SPARK-58523 review, where `fullyPushedFilterAttributes`
was described as the first thing in Spark to remove the post-scan `FilterExec`. The two paths
above already removed it.
Keeping the pushed filter instead of propagating is not available here: by the time `eval` fails,
the connector has accepted the predicate and Spark has already dropped the filter, and Spark
never learns that an evaluation failed.
### Does this PR introduce _any_ user-facing change?
Yes. A query whose partition predicate cannot be evaluated now fails with the underlying error
instead of silently returning rows the filter rejects, or deleting partitions the condition never
accepted. The error is the one the same query produces without pushdown. Queries that appear to
succeed on 4.2.0 or 4.3.0 may start failing; that failure is the correct result and was
previously masked.
The same applies to a connector that passes a partition key whose width does not match
`Table.partitioning()`. That now raises an internal error rather than being silently treated as a
match.
Runtime filtering is unchanged.
### How was this patch tested?
New tests: a scan and a metadata-only DELETE whose partition predicate fails to evaluate must
surface the error rather than keep the partition, and the DELETE must leave both partitions in
place. Both fail without the fix. A unit test pins the split, asserting that the same predicate
propagates by default and reports a match when constructed for a runtime filter.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59572][SQL] PartitionPredicate evaluation failure must not be reported as a partition match [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #58851:
URL: https://github.com/apache/spark/pull/58851#discussion_r4031041084
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/PartitionPredicateImpl.scala:
##########
@@ -58,7 +78,8 @@ class PartitionPredicateImpl private (
try {
boundPredicate(partitionValues)
} catch {
- case e: Exception =>
+ // Propagated when this predicate is the only evaluator: see `failOpen`.
Review Comment:
nit: not sure what do you mean 'is the only evaluator'. do you mean :
```// When evaluation is pruning-only, keep the partition on failure;
// otherwise, let the exception propagate.```
if we rename the variable as i suggest, i think its more clear and we may not need comment?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/PartitionPredicateImpl.scala:
##########
@@ -26,10 +27,23 @@ import org.apache.spark.sql.connector.expressions.filter.PartitionPredicate
/**
* An implementation for [[PartitionPredicate]] that wraps a Catalyst Expression representing a
* partition filter.
+ *
+ * @param catalystExpr the partition filter this predicate evaluates.
+ * @param partitionFields one entry per transform of `Table.partitioning()`, in that order, so a
+ * bound ordinal matches the partition key a connector passes to [[eval]].
+ * @param failOpen what [[eval]] does when it cannot evaluate the expression for a partition.
Review Comment:
can we rename it for clarity? maybe 'keepPartitionOnEvaluationFailure'
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/PartitionPredicateImpl.scala:
##########
@@ -47,6 +61,12 @@ class PartitionPredicateImpl private (
override def eval(partitionValues: InternalRow): Boolean = {
if (partitionValues.numFields != partitionFields.length) {
+ if (!failOpen) {
+ throw SparkException.internalError(
Review Comment:
can we make the message once? like for example
```
val message =
log"Cannot evaluate partition predicate ${MDC(LogKeys.EXPR, catalystExpr.sql)}: " +
log"partition value field count " +
log"(${MDC(LogKeys.COUNT, partitionValues.numFields)}) does not match schema " +
log"(${MDC(LogKeys.NUM_PARTITIONS, partitionFields.length)})."
if (!keepPartitionOnEvaluationFailure) {
throw SparkException.internalError(message.message)
}
logWarning(message + log" Including partition in scan result.")
return true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59572][SQL] PartitionPredicate evaluation failure must not be reported as a partition match [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #58851:
URL: https://github.com/apache/spark/pull/58851#issuecomment-5695228253
cc @szehon-ho @peter-toth @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59572][SQL] PartitionPredicate evaluation failure must not be reported as a partition match [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #58851:
URL: https://github.com/apache/spark/pull/58851#issuecomment-5704255866
Thank you for the fix, @pan3793. The direction looks right to me. I'd like to raise two user-facing points that may be worth documenting in the PR description (or covering with a test).
1. **Queries that currently succeed may fail, depending on conjunct evaluation order.**
For example, `WHERE dep <> 'hr' AND to_int(dep) = 1`. Without pushdown, `FilterExec` evaluates the translated conjunct first (`prioritizeFilters`) and short-circuits, so `to_int` is never called on `'hr'` and the query succeeds. With pushdown, `dep <> 'hr'` (first pass) and `to_int(dep) = 1` (second-pass `PartitionPredicate`) are evaluated independently inside the connector, in whatever order the connector chooses. Before this PR, fail-open happened to give the correct answer here. After it, the query may fail, depending on the connector. Since Spark doesn't guarantee evaluation order, this is not a correctness issue, but the set of queries that "start failing" is wider than the description currently suggests.
2. **The error can now surface earlier, and on empty partitions.**
As the new test notes, the error is raised while the scan is built. A connector that evaluates partition predicates during scan building or planning will therefore make `EXPLAIN SELECT ...` fail with a data error. Also, the non-pushdown path fails only when a row in the partition is actually evaluated, whereas partition-metadata evaluation can fail even for a partition with no rows. Both seem acceptable, but it would be good to mention them in the user-facing change section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59572][SQL] PartitionPredicate evaluation failure must not be reported as a partition match [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #58851:
URL: https://github.com/apache/spark/pull/58851#discussion_r4034741382
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2EnhancedPartitionFilterSuite.scala:
##########
@@ -403,6 +404,26 @@ class DataSourceV2EnhancedPartitionFilterSuite
}
}
+ test("SPARK-59572: a failed second-pass PartitionPredicate must not keep the partition") {
+ withTable(partFilterTableName) {
+ sql(s"CREATE TABLE $partFilterTableName (part_col string, data string) USING $v2Source " +
+ "PARTITIONED BY (part_col)")
+ sql(s"INSERT INTO $partFilterTableName VALUES ('hr', 'x'), ('1', 'y')")
+
+ spark.udf.register("to_int", (s: String) => s.toInt)
+
+ // `to_int('hr')` throws, so the filter is never true for that partition. The filter is
+ // untranslatable, so it is pushed as a PartitionPredicate and accepted, which removes it
+ // from the post-scan filters. The source is then its only evaluator, so reporting a failed
+ // evaluation as a match returns the 'hr' row. The error surfaces while the scan is built.
+ val e = intercept[SparkException] {
Review Comment:
Confirmed by running it, and fixed in 38a683b. With `accept-partition-predicates` set to `false` the scan query throws the same `FAILED_EXECUTE_UDF` with the same cause, and the DELETE fallback throws and leaves both rows, so both tests passed on the path they were written to exclude.
The scan test now asserts against `queryExecution.optimizedPlan`, which raises on the pushed path and raises nothing on the fallback. For the DELETE I split `executeAndKeepPlan` into a `keepingPlan` helper that records the plan in `onFailure` too, so the test asserts `DeleteFromTableExec` around the intercept.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala:
##########
@@ -481,7 +485,10 @@ object PushDownUtils extends Logging {
partitionFields: Seq[PartitionPredicateField]): Seq[PartitionPredicateImpl] = {
val catalystExprs = runtimeFilters.flatMap(unwrapRuntimeFilterExpression)
val flattened = flattenNestedPartitionFilters(catalystExprs, partitionFields).keys
- createPartitionPredicates(flattened.toSeq, partitionFields)._1
+ // A runtime filter only prunes: its rows are filtered anyway, by the post-scan `FilterExec`
+ // for a scalar subquery filter and by the join it was derived from for a DPP filter. So a
Review Comment:
Right, and the third reason is the one that makes it safe. Fixed in 38a683b: both places that enumerate now name the row-level group filter and say a `ReplaceData` or `WriteDelta` rewrite re-applies its own condition. The `InMemoryBaseTable` comment needed no change, since it says "whose rows are filtered anyway" without listing the sources.
Thanks also for verifying the runtime half independently. That matches what I traced before opening the PR, so it now has two confirmations.
##########
docs/sql-migration-guide.md:
##########
@@ -55,6 +55,11 @@ license: |
- Since Spark 4.3, the Spark Connect session errors `INVALID_HANDLE.SESSION_CHANGED`/`SESSION_CLOSED`/`SESSION_NOT_FOUND` carry SQLSTATE `08003` instead of `HY000`; the condition names are unchanged. Code matching these errors on SQLSTATE should match `08003` or class `08`.
- Since Spark 4.3, [Declarative Pipelines](declarative-pipelines-programming-guide.html) honors `spark.sql.caseSensitive` when inferring and evolving pipeline table schemas. Under case-insensitive resolution (the default), column names that differ only in case now identify the same column: flows writing to one table contribute a single column rather than one per spelling, and a column that differs only in case from one already persisted in the target is written to that column instead of being added alongside it. Previously such names were always treated as distinct, producing a table schema that Spark's own resolver could not disambiguate and that could fail later with errors such as `COLUMN_ALREADY_EXISTS` or `AMBIGUOUS_REFERENCE`. When two flows' columns fold together but their types are incompatible, the update now fails at validation with `UNABLE_TO_INFER_PIPELINE_TABLE_SCHEMA`. Where the spellings differ, the surviving one comes from the flow with the lowest identifier. An expl
icitly declared table schema keeps its spelling over the inferred one. Incremental streaming tables keep the persisted spelling of an existing column; materialized views re-infer the schema on every update. Set `spark.sql.caseSensitive` to `true` to keep names differing only in case distinct, as before.
- Since Spark 4.3, all flows writing to the same pipeline table must agree on the effective `spark.sql.caseSensitive`, which each flow takes from its own SQL configuration (a `SET` in pipeline source, which never reaches the session) and otherwise from the session. A disagreement fails the update with `CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY`, because that value decides whether names differing only in case identify the same column and would otherwise make the table's schema depend on the order the flows are evaluated in. Set `spark.sql.caseSensitive` to the same value for every flow writing to the table, remembering that a flow which does not set it inherits the session's value.
+- Since Spark 4.3, when a data source cannot evaluate a partition predicate that Spark pushed down, for example because an ANSI cast of a partition value fails, the error is raised instead of the partition being reported as a match. Previously the failure was ignored and the partition was kept, so a scan could return rows the filter does not accept and a metadata-only `DELETE` could remove a partition its condition never matched. Queries that appeared to succeed may now fail with the same error the query raises without pushdown, and whether a given query fails depends on the order in which the data source evaluates the predicates it accepted, which Spark does not define. The error can also surface earlier than before, while the scan is built rather than while rows are read, so `EXPLAIN` may report it in place of a plan and a partition holding no rows can raise it. Runtime filters are unaffected: a partition they cannot evaluate is still kept, because their rows are filtered after th
e scan or by the join the filter came from.
Review Comment:
Added to both entries in 38a683b. I also shortened them while there, since they had grown well past their neighbours.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59572][SQL] PartitionPredicate evaluation failure must not be reported as a partition match [spark]
Posted by "tdcmeehan (via GitHub)" <gi...@apache.org>.
tdcmeehan commented on PR #58851:
URL: https://github.com/apache/spark/pull/58851#issuecomment-5704442494
This change overall LGTM, but does this require an update to `docs/sql-migration-guide.md`? Per the [contributing guide](https://spark.apache.org/contributing.html).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59572][SQL] PartitionPredicate evaluation failure must not be reported as a partition match [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #58851:
URL: https://github.com/apache/spark/pull/58851#issuecomment-5712855019
The Kafka flaky test failure is irrelevant, thanks all for the review, I'm going to merge this to unblock subsequent PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59572][SQL] PartitionPredicate evaluation failure must not be reported as a partition match [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #58851: [SPARK-59572][SQL] PartitionPredicate evaluation failure must not be reported as a partition match
URL: https://github.com/apache/spark/pull/58851
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59572][SQL] PartitionPredicate evaluation failure must not be reported as a partition match [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #58851:
URL: https://github.com/apache/spark/pull/58851#issuecomment-5713051626
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/8974c2ce653c79e828ed827e5667904b745254d6
- merged into branch-4.x https://github.com/apache/spark/commit/a55618ba3cee17f99c3c24d3c9664803c81f520b
- merged into branch-4.3 https://github.com/apache/spark/commit/7f4c297ffdb9950c3d87e86678e492120d20c6a1
- merged into branch-4.2 https://github.com/apache/spark/commit/5ef96075f26f55da613ece03e1a53ba81c92328e
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59572][SQL] PartitionPredicate evaluation failure must not be reported as a partition match [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #58851:
URL: https://github.com/apache/spark/pull/58851#discussion_r4032572802
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/PartitionPredicateImpl.scala:
##########
@@ -26,10 +27,23 @@ import org.apache.spark.sql.connector.expressions.filter.PartitionPredicate
/**
* An implementation for [[PartitionPredicate]] that wraps a Catalyst Expression representing a
* partition filter.
+ *
+ * @param catalystExpr the partition filter this predicate evaluates.
+ * @param partitionFields one entry per transform of `Table.partitioning()`, in that order, so a
+ * bound ordinal matches the partition key a connector passes to [[eval]].
+ * @param failOpen what [[eval]] does when it cannot evaluate the expression for a partition.
Review Comment:
Renamed to `keepOnEvalFailure` in 9b8d884. Kept it shorter than the suggestion since the class is already `PartitionPredicateImpl`, so "Partition" reads as redundant.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/PartitionPredicateImpl.scala:
##########
@@ -47,6 +61,12 @@ class PartitionPredicateImpl private (
override def eval(partitionValues: InternalRow): Boolean = {
if (partitionValues.numFields != partitionFields.length) {
+ if (!failOpen) {
+ throw SparkException.internalError(
Review Comment:
Applied in 9b8d884, as suggested. The exception text is unchanged, so the assertion that pins it still matches.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/PartitionPredicateImpl.scala:
##########
@@ -58,7 +78,8 @@ class PartitionPredicateImpl private (
try {
boundPredicate(partitionValues)
} catch {
- case e: Exception =>
+ // Propagated when this predicate is the only evaluator: see `failOpen`.
Review Comment:
Dropped it in 9b8d884. You were right that the rename carries it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59572][SQL] PartitionPredicate evaluation failure must not be reported as a partition match [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #58851:
URL: https://github.com/apache/spark/pull/58851#issuecomment-5707587835
@dongjoon-hyun both points are real, and both are now in the description and the migration guide.
1. Confirmed, and connector-dependent as you say. I measured your example: `dep <> 'hr' AND to_int(dep) = 1` still succeeds against the in-memory source, because its predicate loop short-circuits before reaching the UDF, while `data <> 'x' AND to_int(dep) = 1` throws, since the data-column conjunct stays post-scan and cannot shield the partition predicate. Spark defines no order here, so the guide says the outcome depends on the source.
2. Confirmed with one correction worth recording: `EXPLAIN` does not fail. It prints `Error occurred during query planning:` followed by the error in place of the plan. The empty-partition case follows from evaluating partition metadata rather than rows, and is noted too.
@tdcmeehan added in 9b8d884. Since this targets 4.3.0 and 4.2.1, there are two entries: one under "Upgrading from Spark SQL 4.2 to 4.3" and one under a new "Upgrading from Spark SQL 4.2.0 to 4.2.1" section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59572][SQL] PartitionPredicate evaluation failure must not be reported as a partition match [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #58851:
URL: https://github.com/apache/spark/pull/58851#discussion_r4034497348
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2EnhancedPartitionFilterSuite.scala:
##########
@@ -403,6 +404,26 @@ class DataSourceV2EnhancedPartitionFilterSuite
}
}
+ test("SPARK-59572: a failed second-pass PartitionPredicate must not keep the partition") {
+ withTable(partFilterTableName) {
+ sql(s"CREATE TABLE $partFilterTableName (part_col string, data string) USING $v2Source " +
+ "PARTITIONED BY (part_col)")
+ sql(s"INSERT INTO $partFilterTableName VALUES ('hr', 'x'), ('1', 'y')")
+
+ spark.udf.register("to_int", (s: String) => s.toInt)
+
+ // `to_int('hr')` throws, so the filter is never true for that partition. The filter is
+ // untranslatable, so it is pushed as a PartitionPredicate and accepted, which removes it
+ // from the post-scan filters. The source is then its only evaluator, so reporting a failed
+ // evaluation as a match returns the 'hr' row. The error surfaces while the scan is built.
+ val e = intercept[SparkException] {
Review Comment:
**Finding 1.** This test passes whether or not the `PartitionPredicate` is the evaluator, so it does not pin the path its comment describes.
Any path that leaves `to_int(part_col) = 1` in the post-scan filters raises the same error. The `FilterExec` above the scan evaluates it on the `'hr'` row and gets `FAILED_EXECUTE_UDF` with a `NumberFormatException` cause, so both assertions hold. Setting `accept-partition-predicates` to `false` on the fixture is the cheapest way to see it: `pushPredicates` returns the predicate, `pushPartitionPredicates` restores the original filter, and the test still goes green. It does fail on master today, so the fix is proven. What is missing is that a later change which stops pushing this filter leaves the test green and silent.
The comment already names the discriminator: on the pushed path the error comes out of planning, on the post-scan path only out of execution. Asserting that is a one-line change:
```scala
val e = intercept[SparkException] {
sql(s"SELECT * FROM $partFilterTableName WHERE to_int(part_col) = 1")
.queryExecution.optimizedPlan
}
```
The DELETE test at `sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2EnhancedDeleteFilterSuite.scala:255` has the same gap and needs a different fix. If the metadata-only rewrite does not fire, the row-level DELETE evaluates the same UDF, throws the same error and writes nothing, so `checkAnswer` passes too. `executeAndKeepPlan` cannot capture the plan there because its `onFailure` is a no-op; recording `qe.executedPlan` in `onFailure` as well lets the test assert `DeleteFromTableExec` around the `intercept`.
Both of these are derived from reading, not from running them.
##########
docs/sql-migration-guide.md:
##########
@@ -55,6 +55,11 @@ license: |
- Since Spark 4.3, the Spark Connect session errors `INVALID_HANDLE.SESSION_CHANGED`/`SESSION_CLOSED`/`SESSION_NOT_FOUND` carry SQLSTATE `08003` instead of `HY000`; the condition names are unchanged. Code matching these errors on SQLSTATE should match `08003` or class `08`.
- Since Spark 4.3, [Declarative Pipelines](declarative-pipelines-programming-guide.html) honors `spark.sql.caseSensitive` when inferring and evolving pipeline table schemas. Under case-insensitive resolution (the default), column names that differ only in case now identify the same column: flows writing to one table contribute a single column rather than one per spelling, and a column that differs only in case from one already persisted in the target is written to that column instead of being added alongside it. Previously such names were always treated as distinct, producing a table schema that Spark's own resolver could not disambiguate and that could fail later with errors such as `COLUMN_ALREADY_EXISTS` or `AMBIGUOUS_REFERENCE`. When two flows' columns fold together but their types are incompatible, the update now fails at validation with `UNABLE_TO_INFER_PIPELINE_TABLE_SCHEMA`. Where the spellings differ, the surviving one comes from the flow with the lowest identifier. An expl
icitly declared table schema keeps its spelling over the inferred one. Incremental streaming tables keep the persisted spelling of an existing column; materialized views re-infer the schema on every update. Set `spark.sql.caseSensitive` to `true` to keep names differing only in case distinct, as before.
- Since Spark 4.3, all flows writing to the same pipeline table must agree on the effective `spark.sql.caseSensitive`, which each flow takes from its own SQL configuration (a `SET` in pipeline source, which never reaches the session) and otherwise from the session. A disagreement fails the update with `CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY`, because that value decides whether names differing only in case identify the same column and would otherwise make the table's schema depend on the order the flows are evaluated in. Set `spark.sql.caseSensitive` to the same value for every flow writing to the table, remembering that a flow which does not set it inherits the session's value.
+- Since Spark 4.3, when a data source cannot evaluate a partition predicate that Spark pushed down, for example because an ANSI cast of a partition value fails, the error is raised instead of the partition being reported as a match. Previously the failure was ignored and the partition was kept, so a scan could return rows the filter does not accept and a metadata-only `DELETE` could remove a partition its condition never matched. Queries that appeared to succeed may now fail with the same error the query raises without pushdown, and whether a given query fails depends on the order in which the data source evaluates the predicates it accepted, which Spark does not define. The error can also surface earlier than before, while the scan is built rather than while rows are read, so `EXPLAIN` may report it in place of a plan and a partition holding no rows can raise it. Runtime filters are unaffected: a partition they cannot evaluate is still kept, because their rows are filtered after th
e scan or by the join the filter came from.
Review Comment:
**Finding 3.** The description's user-facing section lists two changes and says both are recorded here, but only the first is.
"A partition key whose width does not match `Table.partitioning()` now raises an internal error" appears in neither this entry nor the 4.2.1 one. For a connector that passes a subset key it turns a warning plus a kept partition into a failed query, so a user of such a connector sees a query that used to work start failing.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala:
##########
@@ -481,7 +485,10 @@ object PushDownUtils extends Logging {
partitionFields: Seq[PartitionPredicateField]): Seq[PartitionPredicateImpl] = {
val catalystExprs = runtimeFilters.flatMap(unwrapRuntimeFilterExpression)
val flattened = flattenNestedPartitionFilters(catalystExprs, partitionFields).keys
- createPartitionPredicates(flattened.toSeq, partitionFields)._1
+ // A runtime filter only prunes: its rows are filtered anyway, by the post-scan `FilterExec`
+ // for a scalar subquery filter and by the join it was derived from for a DPP filter. So a
Review Comment:
**Finding 2.** A third source of runtime `PartitionPredicate`s is missing from this list, and neither reason given covers it.
`RowLevelOperationRuntimeGroupFiltering.buildDynamicPruningCond` injects `DynamicPruningExpression(InSubquery(pruningKeys, ...))`, where `pruningKeys` resolve from the scan's own `filterAttributes()` (`sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/RowLevelOperationRuntimeGroupFiltering.scala:146`). A partition column there reaches `createRuntimePartitionPredicates` like any other DPP filter. `DataSourceV2Strategy` partitions it out as a `DynamicPruning`, so it has no post-scan `FilterExec`, and it was not derived from a join: it comes from the row-level operation's own condition.
Keeping the partition is still safe there, for a third reason. A `ReplaceData` or `WriteDelta` rewrite re-applies that condition to the rows it reads, so an extra group is rewritten unchanged rather than mis-written. Worth spelling out, because this same two-case list is the whole safety argument in the `keepOnEvalFailure` scaladoc and in the `InMemoryBaseTable` comment, and the next reader will check it against this path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org