You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "gengliangwang (via GitHub)" <gi...@apache.org> on 2026/10/19 23:43:49 UTC

[PR] [SQL] Push down local limits below unions and cross joins [spark]

gengliangwang opened a new pull request, #59618:
URL: https://github.com/apache/spark/pull/59618

   ### What changes were proposed in this pull request?
   
   Push redundant local limits into DataSource V2 scans below `UNION ALL` and conditionless inner/cross joins when an enclosing global limit provides the same total row bound.
   
   The new path requires an existing local limit with the same bound, a scan with only deterministic projections above it, and no residual filters or pushed offset. It retains both the global and local limit operators. Connector attempts are tracked by builder identity for each rule invocation, including rejected attempts, so offset/limit fallback does not repeat the new pushdown.
   
   Existing complete limit, Top-N, and offset pushdown behavior is preserved. Arbitrary standalone local limits are not pushed because their per-partition bound is not equivalent to a global source limit.
   
   ### Why are the changes needed?
   
   Catalyst already adds local limits to union and join inputs, but V2 scan pushdown only handles the adjacent global/local limit pair. For queries such as `SELECT * FROM a CROSS JOIN b LIMIT 5`, the scans therefore miss the opportunity to apply their existing input limits at the source.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. Eligible V2 sources can receive limit pushdown for these union and join inputs, reducing source work and rows transferred to Spark. Query result semantics are unchanged.
   
   ### How was this patch tested?
   
   Added 25 focused optimizer tests covering supported branches, unsupported standalone limits, barriers, partial/rejected requests, callback counts, and existing limit/Top-N/offset behavior. Added four JDBC integration cases covering unions and cross joins with one and two partitions, each checking plain limits and both offset/limit orders.
   
   Local validation:
   
   - `sql/testOnly *V2ScanRelationPushDownSuite`: 25 tests passed.
   - `sql/testOnly *JDBCV2Suite -- -z "local limit" -z "simple scan with" -z "scan with aggregate push-down, top N push-down and offset push-down"`: 11 tests passed.
   - `sql/scalastyle` and `sql/Test/scalastyle`: passed with no errors or warnings.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Codex (GPT-6)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org