[jira] [Resolved] (SPARK-59004) Use vectorized Arrays.compare for ComparableByteArray.compareTo

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun resolved SPARK-59004.
-----------------------------------
    Fix Version/s: 4.4.0
       Resolution: Fixed

Issue resolved by pull request 58285
[https://github.com/apache/spark/pull/58285]

> Use vectorized Arrays.compare for ComparableByteArray.compareTo
> ---------------------------------------------------------------
>
>                 Key: SPARK-59004
>                 URL: https://issues.apache.org/jira/browse/SPARK-59004
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 4.1.0
>            Reporter: David Mollitor
>            Assignee: David Mollitor
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 4.4.0
>
>
> {{ArrayWrappers.ComparableByteArray.compareTo}} (in {{{}common/kvstore{}}}, used to order {{byte[]}} keys in the KVStore that backs the Spark status/UI store) implements a
> hand-written scalar loop that compares bytes one at a time. This replaces that loop
> with {{{}java.util.Arrays.compare(byte[], byte[]){}}}.
> {{Arrays.compare}} performs the identical comparison – signed, lexicographic, comparing
> the common prefix first and ordering the shorter array first on a tie – but it is a
> HotSpot intrinsic backed by a vectorized (SIMD) mismatch scan, so it is faster while
> preserving the exact ordering. It is also simpler than the hand-written loop.
> *Semantics are unchanged.* The existing byte comparison is signed 
> ({{{}array[i] - other.array[i]{}}} promotes each {{byte}} to a sign-extended {{{}int{}}}), which
> matches {{{}Arrays.compare{}}}'s {{Byte.compare}} semantics; the single-element byte difference cannot overflow, so behavior is unchanged.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org