You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "david-mollitor-db (via GitHub)" <gi...@apache.org> on 2026/09/17 12:55:19 UTC
[PR] [SPARK-59614][SQL] Compile the AS-OF join as-of/residual conditions to a Predicate [spark]
david-mollitor-db opened a new pull request, #58886:
URL: https://github.com/apache/spark/pull/58886
### What changes were proposed in this pull request?
`SortMergeAsOfJoinScanner`
(`sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala`)
evaluated its as-of and residual conditions as bound `Expression`s, interpreted per right-buffer
row (`boundAsOfCond.eval(joinedRow)` and `boundResidualCond.forall(...)`). This compiles them to
`BasePredicate` via `Predicate.create(...)` and evaluates them through the primitive-returning
`eval(InternalRow): Boolean`:
- `boundAsOfCond` becomes a `BasePredicate`; `boundResidualCond` becomes `Option[BasePredicate]`.
- Each predicate is initialized per partition (`initialize(partitionIndex)`, index from
`TaskContext.getPartitionId()`), matching the `CartesianProductExec` idiom.
- The call sites in `findBestBackwardForward`/`findBestForwardNearest` use
`if (boundAsOfCond.eval(joinedRow))` plus a small `residualHolds` helper that avoids the
`Option.forall` closure and its boxed result.
- The `orderExpression` (a distance value, not a boolean) intentionally stays a bound `Expression`.
### Why are the changes needed?
`SortMergeAsOfJoinScanner` has no whole-stage codegen, so its inner scan is always interpreted.
Interpreted `Expression.eval` walks the expression tree and `BoundReference.eval` boxes every
operand it reads into non-cached `java.lang.Double`/`java.lang.Long` objects. JFR profiling of
`AsOfJoinBenchmark` (sort-merge cases isolated) showed this dominated the scan: ~24% of CPU
(`boxToInteger` 17.6% + `Double.valueOf` 4.1% + `Long.valueOf` 2.0%) and ~33% of allocation
(`Double` 20% + `Long` 13%). Compiling the conditions removes both the tree-walk and the boxing;
the profile's boxing/interpreted-eval frames are replaced by a single generated
`SpecificPredicate.eval`, and a no-equi-key `AsOfJoinBenchmark` run improved ~14% (929ms -> 800ms
best time).
### Does this PR introduce _any_ user-facing change?
No. `Predicate.eval` returns `false` for a null/false result, exactly matching the previous
`x != null && x.asInstanceOf[Boolean]` guard, and the binding schema is unchanged.
### How was this patch tested?
Existing tests pass: `SortMergeAsOfJoinSuite`, `DataFrameAsOfJoinSuite`, `AsOfJoinSQLSuite`, and
`AsOfJoinSortMergeSQLSuite` (89 tests). Before/after JFR on `AsOfJoinBenchmark` confirms the
boxing and interpreted-eval frames are eliminated.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59614][SQL] Compile the AS-OF join as-of/residual conditions to a Predicate [spark]
Posted by "david-mollitor-db (via GitHub) via reviews" <re...@spark.apache.org>.
david-mollitor-db closed pull request #58886: [SPARK-59614][SQL] Compile the AS-OF join as-of/residual conditions to a Predicate
URL: https://github.com/apache/spark/pull/58886
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59614][SQL] Compile the AS-OF join as-of/residual conditions to a Predicate [spark]
Posted by "LukaZdravic (via GitHub) via reviews" <re...@spark.apache.org>.
LukaZdravic commented on PR #58886:
URL: https://github.com/apache/spark/pull/58886#issuecomment-6582757960
Thanks @david-mollitor-db for this effort, as per offline discussion this work will be finished in this PR https://github.com/apache/spark/pull/60029.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59614][SQL] Compile the AS-OF join as-of/residual conditions to a Predicate [spark]
Posted by "david-mollitor-db (via GitHub) via reviews" <re...@spark.apache.org>.
david-mollitor-db commented on PR #58886:
URL: https://github.com/apache/spark/pull/58886#issuecomment-6583009692
Thanks @LukaZdravic for incorporating the change! Closing this ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org