You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "szehon-ho (via GitHub)" <gi...@apache.org> on 2026/08/27 19:05:01 UTC
[PR] [SPARK-59012][SQL][4.3] Improve SupportsRuntimeCatalystFiltering test coverage and reject nested filter attributes [spark]
szehon-ho opened a new pull request, #58356:
URL: https://github.com/apache/spark/pull/58356
### What changes were proposed in this pull request?
This backports #58296 to `branch-4.3`.
The patch improves test coverage for `SupportsRuntimeCatalystFiltering` in dynamic partition
pruning and storage-partitioned joins, and rejects nested runtime-filter attributes that violate
the connector contract.
The cherry-pick had one conflict in `KeyGroupedPartitioningSuite.scala`, resolved to preserve the
4.3 branch's existing test organization and tagging while adding the new runtime-filter suites.
### Why are the changes needed?
Spark 4.3 contains `SupportsRuntimeCatalystFiltering`, but lacks the coverage and contract
validation added by #58296. Backporting the change prevents regressions in Catalyst-expression
runtime filtering and rejects invalid nested filter attributes consistently.
### Does this PR introduce _any_ user-facing change?
No new user-facing change beyond #58296. For connector authors, a scan that declares a nested
runtime-filter attribute is rejected with an internal error instead of silently widening it to the
enclosing top-level column.
### How was this patch tested?
Run locally with `SPARK_LOCAL_IP=127.0.0.1`:
```text
build/sbt 'sql/testOnly org.apache.spark.sql.connector.DataSourceV2CatalystRuntimeFilterSuite org.apache.spark.sql.DynamicPartitionPruningV2FilterSuiteAEOff org.apache.spark.sql.DynamicPartitionPruningV2FilterSuiteAEOn org.apache.spark.sql.DynamicPartitionPruningV2CatalystFilterSuiteAEOff org.apache.spark.sql.DynamicPartitionPruningV2CatalystFilterSuiteAEOn org.apache.spark.sql.connector.KeyGroupedPartitioningSuite org.apache.spark.sql.connector.KeyGroupedPartitioningRuntimeFilterSuite org.apache.spark.sql.connector.KeyGroupedPartitioningCatalystRuntimeFilterSuite'
```
277 tests succeeded, 0 failed, and 2 were ignored.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor with Claude Opus 5 for the original patch.
Generated-by: OpenAI Codex with GPT-5 for the backport and conflict resolution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59012][SQL][4.3] Improve SupportsRuntimeCatalystFiltering test coverage and reject nested filter attributes [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #58356:
URL: https://github.com/apache/spark/pull/58356#issuecomment-5444086492
@cloud-fan @HeartSaVioR can we get this in for 4.3? the public DSV2 API is added in 4.3 and we changed a little behavior. But somehow due to conflicts I couldn't merge https://github.com/apache/spark/pull/58296 directly to 4.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59012][SQL][4.3] Improve SupportsRuntimeCatalystFiltering test coverage and reject nested filter attributes [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #58356:
URL: https://github.com/apache/spark/pull/58356#issuecomment-5447295474
actually found a breaking issue, fixing it first: https://github.com/apache/spark/pull/58370
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59012][SQL][4.3] Improve SupportsRuntimeCatalystFiltering test coverage and reject nested filter attributes [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #58356:
URL: https://github.com/apache/spark/pull/58356#issuecomment-5446529124
holding on this for the moment, looking more deep if the extra validation make sense or not
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59012][SQL][4.3] Improve SupportsRuntimeCatalystFiltering test coverage and reject nested filter attributes [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho closed pull request #58356: [SPARK-59012][SQL][4.3] Improve SupportsRuntimeCatalystFiltering test coverage and reject nested filter attributes
URL: https://github.com/apache/spark/pull/58356
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org