You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "David Mollitor (Jira)" <ji...@apache.org> on 2026/09/17 13:25:00 UTC

[jira] [Created] (SPARK-59616) Avoid copying every candidate row in the AS-OF join best-match scan

David Mollitor created SPARK-59616:
--------------------------------------

             Summary: Avoid copying every candidate row in the AS-OF join best-match scan
                 Key: SPARK-59616
                 URL: https://issues.apache.org/jira/browse/SPARK-59616
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 4.1.0
            Reporter: David Mollitor


h2. Problem

{{SortMergeAsOfJoinScanner }}({{{}sql/core/.../execution/joins/SortMergeAsOfJoinExec.scala{}}}) has no
whole-stage codegen, so its inner right-buffer scan is always interpreted. For each left row it scans the buffered right group and materializes the best match with {{{}bestMatch = rightRow.copy(){}}}. Under a backward (last-match-wins) join this copies *every* as-of-satisfying candidate and discards all but the last; forward/nearest copies on each improvement.

JFR profiling of {{AsOfJoinBenchmark}} (sort-merge cases isolated) showed {{UnsafeRow.copy()}} at ~18% of CPU and ~61% of allocation ({{{}UnsafeRow{}}} 36% + {{byte[]}} 25%): each {{copy()}} allocates a fresh {{UnsafeRow}} and {{{}byte[]{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org