You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "David Mollitor (Jira)" <ji...@apache.org> on 2026/09/17 13:25:00 UTC
[jira] [Created] (SPARK-59616) Avoid copying every candidate row in the AS-OF join best-match scan
David Mollitor created SPARK-59616:
--------------------------------------
Summary: Avoid copying every candidate row in the AS-OF join best-match scan
Key: SPARK-59616
URL: https://issues.apache.org/jira/browse/SPARK-59616
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h2. Problem
{{SortMergeAsOfJoinScanner }}({{{}sql/core/.../execution/joins/SortMergeAsOfJoinExec.scala{}}}) has no
whole-stage codegen, so its inner right-buffer scan is always interpreted. For each left row it scans the buffered right group and materializes the best match with {{{}bestMatch = rightRow.copy(){}}}. Under a backward (last-match-wins) join this copies *every* as-of-satisfying candidate and discards all but the last; forward/nearest copies on each improvement.
JFR profiling of {{AsOfJoinBenchmark}} (sort-merge cases isolated) showed {{UnsafeRow.copy()}} at ~18% of CPU and ~61% of allocation ({{{}UnsafeRow{}}} 36% + {{byte[]}} 25%): each {{copy()}} allocates a fresh {{UnsafeRow}} and {{{}byte[]{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org