You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/18 06:25:31 UTC

[GitHub] [spark] ulysses-you opened a new pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

ulysses-you opened a new pull request #34310:
URL: https://github.com/apache/spark/pull/34310


   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'core/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   Unify the compare function of `UTF8String` and `ByteArray`.
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   `BinaryType` use `TypeUtils.compareBinary` to compare two byte array, however it's slow since it compares byte array using unsigned int comparison byte by bye.
   
   We can compare them using `Platform.getLong` with unsigned long comparison if they have more than 8 bytes. And here is some histroy about this `TODO` https://github.com/apache/spark/pull/6755/files#r32197461
   
   The benchmark result should be same with `UTF8String`, can be found in https://github.com/apache/spark/pull/19180 (https://github.com/apache/spark/pull/19180#issuecomment-329467838)
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   Move test from `TypeUtilsSuite` to `ByteArraySuite`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945606694


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48837/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945706303


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48842/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-946443508






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945408735


   **[Test build #144352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144352/testReport)** for PR 34310 at commit [`65a8e64`](https://github.com/apache/spark/commit/65a8e6413f6d6d2265bd4c1ae64edf4be199d67b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ulysses-you commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945704888


   cc @xkrogen @srowen @kiszk @cloud-fan @JoshRosen if you have time to take another look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945408735


   **[Test build #144352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144352/testReport)** for PR 34310 at commit [`65a8e64`](https://github.com/apache/spark/commit/65a8e6413f6d6d2265bd4c1ae64edf4be199d67b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-948240904


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48955/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ulysses-you commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r735087285



##########
File path: sql/core/benchmarks/ByteArrayBenchmark-results.txt
##########
@@ -0,0 +1,16 @@
+================================================================================================

Review comment:
       I compared the two code path within one [patch](https://github.com/ulysses-you/spark/blob/benchmark-test/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ByteArrayBenchmark.scala), and here is the  result.
   
   [JDK8](https://github.com/ulysses-you/spark/actions/runs/1377336453):
   ```
   ================================================================================================
   byte array comparisons
   ================================================================================================
   
   OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 5.8.0-1042-azure
   Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
   Byte Array compare offHeap:               Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   2-7 byte                                            636            661          14        103.0           9.7       1.0X
   8-16 byte                                          1067           1112          21         61.4          16.3       0.6X
   16-32 byte                                         1226           1352          98         53.4          18.7       0.5X
   512-1024 byte                                      1803           1916          46         36.3          27.5       0.4X
   512 byte slow                                      4343           4662         171         15.1          66.3       0.1X
   2-7 byte                                           1075           1119          26         61.0          16.4       0.6X
   
   OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 5.8.0-1042-azure
   Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
   Byte Array compare onHeap:                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   2-7 byte                                           1511           1570          30         43.4          23.1       1.0X
   8-16 byte                                          1522           1564          27         43.1          23.2       1.0X
   16-32 byte                                         1426           1554          36         46.0          21.8       1.1X
   512-1024 byte                                      2080           2198          86         31.5          31.7       0.7X
   512 byte slow                                     28498          29222         410          2.3         434.9       0.1X
   2-7 byte                                           1382           1485          61         47.4          21.1       1.1X
   ```
   
   [JDK11](https://github.com/ulysses-you/spark/actions/runs/1377337612)
   ```
   ================================================================================================
   byte array comparisons
   ================================================================================================
   
   OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.8.0-1042-azure
   Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
   Byte Array compare offHeap:               Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   2-7 byte                                            720            777          21         91.0          11.0       1.0X
   8-16 byte                                          1077           1138          32         60.8          16.4       0.7X
   16-32 byte                                         1347           1463          84         48.7          20.5       0.5X
   512-1024 byte                                      1898           1989          40         34.5          29.0       0.4X
   512 byte slow                                      4621           4878         168         14.2          70.5       0.2X
   2-7 byte                                           1062           1133          28         61.7          16.2       0.7X
   
   OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.8.0-1042-azure
   Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
   Byte Array compare onHeap:                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   2-7 byte                                           1377           1471          37         47.6          21.0       1.0X
   8-16 byte                                          1398           1475          38         46.9          21.3       1.0X
   16-32 byte                                         1452           1547          47         45.2          22.1       0.9X
   512-1024 byte                                      1826           1953          55         35.9          27.9       0.8X
   512 byte slow                                     45883          47146         NaN          1.4         700.1       0.0X
   2-7 byte                                           1401           1484          39         46.8          21.4       1.0X
   
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945630944


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48837/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-946443508






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945676651


   **[Test build #144367 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144367/testReport)** for PR 34310 at commit [`a0b1c56`](https://github.com/apache/spark/commit/a0b1c5675fddd9695b3eeb24f311c0301ae4a788).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-948204042


   **[Test build #144483 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144483/testReport)** for PR 34310 at commit [`f9f11d6`](https://github.com/apache/spark/commit/f9f11d6c06cc9d63e7d4f342b010c70477b29cfa).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ulysses-you commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r734281197



##########
File path: sql/core/benchmarks/ByteArrayBenchmark-results.txt
##########
@@ -0,0 +1,16 @@
+================================================================================================

Review comment:
       Here is the old code path benchmark result:
   
   JDK8
   ```
   ================================================================================================
   byte array comparisons
   ================================================================================================
   
   OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 5.8.0-1042-azure
   Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
   Byte Array compareTo:                     Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   2-7 byte                                            799            836          24         82.0          12.2       1.0X
   8-16 byte                                           832            906          32         78.8          12.7       1.0X
   16-32 byte                                          812            854          28         80.7          12.4       1.0X
   512-1024 byte                                      1057           1088          20         62.0          16.1       0.8X
   512 byte slow                                     24628          26054         NaN          2.7         375.8       0.0X
   2-7 byte                                            811            849          23         80.8          12.4       1.0X
   ```
   
   JDK11
   ```
   ================================================================================================
   byte array comparisons
   ================================================================================================
   
   OpenJDK 64-Bit Server VM 11.0.13+8-LTS on Linux 5.8.0-1042-azure
   Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
   Byte Array compareTo:                     Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   2-7 byte                                            771            812          28         85.0          11.8       1.0X
   8-16 byte                                           839            857          13         78.1          12.8       0.9X
   16-32 byte                                          898            926          17         73.0          13.7       0.9X
   512-1024 byte                                      1141           1189          23         57.4          17.4       0.7X
   512 byte slow                                     40124          40689         495          1.6         612.2       0.0X
   2-7 byte                                            827            847          14         79.3          12.6       0.9X
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945643899


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144362/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945839574


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144367/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r731137255



##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java
##########
@@ -75,6 +75,42 @@ static long getPrefix(Object base, long offset, int numBytes) {
     return (IS_LITTLE_ENDIAN ? java.lang.Long.reverseBytes(p) : p) & ~mask;
   }
 
+  public static int compareBinary(byte[] leftBase, byte[] rightBase) {
+    return compareBinary(leftBase, Platform.BYTE_ARRAY_OFFSET, leftBase.length,

Review comment:
       I'm only wondering if this ends up being slower - you already have byte arrays, and now have to go through platform methods to read them?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945570648


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48837/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945643899


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144362/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-946360301


   **[Test build #144394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144394/testReport)** for PR 34310 at commit [`15b8efe`](https://github.com/apache/spark/commit/15b8efe7bb2726f7b8662b9f68e1cd0c63ee32e2).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945533091


   **[Test build #144362 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144362/testReport)** for PR 34310 at commit [`a0b1c56`](https://github.com/apache/spark/commit/a0b1c5675fddd9695b3eeb24f311c0301ae4a788).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ulysses-you commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r731483712



##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java
##########
@@ -75,6 +75,42 @@ static long getPrefix(Object base, long offset, int numBytes) {
     return (IS_LITTLE_ENDIAN ? java.lang.Long.reverseBytes(p) : p) & ~mask;
   }
 
+  public static int compareBinary(byte[] leftBase, byte[] rightBase) {
+    return compareBinary(leftBase, Platform.BYTE_ARRAY_OFFSET, leftBase.length,

Review comment:
       thank you @srowen and @JoshRosen for point out the difference. I follow the linked benchmark but add a new 512 byte slow benchmark which the first 511 bytes are same. The benchmark result shows it has no regression after this PR and has big benifits if the byte arrays have many same prefix.
   
   Before this PR:
   ```
   Java HotSpot(TM) 64-Bit Server VM 1.8.0_271-b09 on Mac OS X 10.16
   Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
   Byte Array compareTo:                     Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   2-7 byte                                            800            861          70         81.9          12.2       1.0X
   8-16 byte                                           810            878          59         80.9          12.4       1.0X
   16-32 byte                                          804            887          40         81.5          12.3       1.0X
   512-1024 byte                                      1050           1181          43         62.4          16.0       0.8X
   512 byte slow                                     23593          23698         311          2.8         360.0       0.0X
   2-7 byte                                            778            784           5         84.2          11.9       1.0X
   ```
   
   After this PR:
   ```
   Java HotSpot(TM) 64-Bit Server VM 1.8.0_271-b09 on Mac OS X 10.16
   Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
   Byte Array compareTo:                     Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   2-7 byte                                            425            471          24        154.2           6.5       1.0X
   8-16 byte                                           751            814          40         87.2          11.5       0.5X
   16-32 byte                                          789            842          42         83.1          12.0       0.5X
   512-1024 byte                                      1038           1175         193         63.1          15.8       0.4X
   512 byte slow                                      3419           3924         NaN         19.2          52.2       0.1X
   2-7 byte                                            421            424           2        155.6           6.4       1.0X
   
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-946422362


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48868/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-948280948


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144483/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-948276121


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48955/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-946360301


   **[Test build #144394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144394/testReport)** for PR 34310 at commit [`15b8efe`](https://github.com/apache/spark/commit/15b8efe7bb2726f7b8662b9f68e1cd0c63ee32e2).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-946434718


   **[Test build #144394 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144394/testReport)** for PR 34310 at commit [`15b8efe`](https://github.com/apache/spark/commit/15b8efe7bb2726f7b8662b9f68e1cd0c63ee32e2).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945530369


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48830/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945500934


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48830/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945642847


   **[Test build #144362 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144362/testReport)** for PR 34310 at commit [`a0b1c56`](https://github.com/apache/spark/commit/a0b1c5675fddd9695b3eeb24f311c0301ae4a788).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] kiszk commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-951043099


   late LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ulysses-you commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r733251337



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ByteArrayBenchmark.scala
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import scala.util.Random
+
+import org.apache.spark.benchmark.{Benchmark, BenchmarkBase}
+import org.apache.spark.unsafe.types.ByteArray
+
+/**
+ * Benchmark to measure performance for byte array comparisons.
+ * {{{
+ *   To run this benchmark:
+ *   1. without sbt:
+ *      bin/spark-submit --class <this class> --jars <spark core test jar> <sql core test jar>
+ *   2. build/sbt "sql/test:runMain <this class>"
+ *   3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain <this class>"
+ *      Results will be written to "benchmarks/<this class>-results.txt".
+ * }}}
+ */
+object ByteArrayBenchmark extends BenchmarkBase {
+
+  def byteArrayComparisons(iters: Long): Unit = {
+    val chars = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+    val random = new Random(0)
+    def randomBytes(min: Int, max: Int): Array[Byte] = {
+      val len = random.nextInt(max - min) + min
+      val bytes = new Array[Byte](len)
+      var i = 0
+      while (i < len) {
+        bytes(i) = chars.charAt(random.nextInt(chars.length())).toByte
+        i += 1
+      }
+      bytes
+    }
+
+    val count = 16 * 1000
+    val dataTiny = Seq.fill(count)(randomBytes(2, 7)).toArray
+    val dataSmall = Seq.fill(count)(randomBytes(8, 16)).toArray
+    val dataMedium = Seq.fill(count)(randomBytes(16, 32)).toArray
+    val dataLarge = Seq.fill(count)(randomBytes(512, 1024)).toArray
+    val dataLargeSlow = Seq.fill(count)(
+      Array.tabulate(512) {i => if (i < 511) 0.toByte else 1.toByte}).toArray
+
+    def compareBinary(data: Array[Array[Byte]]) = { _: Int =>
+      var sum = 0L
+      for (_ <- 0L until iters) {
+        var i = 0
+        while (i < count) {
+          sum += ByteArray.compareBinary(data(i), data((i + 1) % count))
+          i += 1
+        }
+      }
+    }
+
+    val benchmark = new Benchmark("Byte Array compareTo", count * iters, 25, output = output)
+    benchmark.addCase("2-7 byte")(compareBinary(dataTiny))
+    benchmark.addCase("8-16 byte")(compareBinary(dataSmall))
+    benchmark.addCase("16-32 byte")(compareBinary(dataMedium))
+    benchmark.addCase("512-1024 byte")(compareBinary(dataLarge))
+    benchmark.addCase("512 byte slow")(compareBinary(dataLargeSlow))
+    benchmark.addCase("2-7 byte")(compareBinary(dataTiny))

Review comment:
       The first benchmark case may run slower than the latter due to the JIT optimization and this case has small size which can be done in a short time that would be more likely affected.
   
   So I also keep it running twice in case this issue.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r734547370



##########
File path: sql/core/benchmarks/ByteArrayBenchmark-results.txt
##########
@@ -0,0 +1,16 @@
+================================================================================================

Review comment:
       I'm inclined to believe it is a win based on your first benchmark. Is there any easy way to run before/after on these Xeons, or is that hard?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945533091


   **[Test build #144362 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144362/testReport)** for PR 34310 at commit [`a0b1c56`](https://github.com/apache/spark/commit/a0b1c5675fddd9695b3eeb24f311c0301ae4a788).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945837296


   **[Test build #144367 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144367/testReport)** for PR 34310 at commit [`a0b1c56`](https://github.com/apache/spark/commit/a0b1c5675fddd9695b3eeb24f311c0301ae4a788).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] JoshRosen commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
JoshRosen commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r731210073



##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java
##########
@@ -75,6 +75,42 @@ static long getPrefix(Object base, long offset, int numBytes) {
     return (IS_LITTLE_ENDIAN ? java.lang.Long.reverseBytes(p) : p) & ~mask;
   }
 
+  public static int compareBinary(byte[] leftBase, byte[] rightBase) {
+    return compareBinary(leftBase, Platform.BYTE_ARRAY_OFFSET, leftBase.length,

Review comment:
       +1. 
   
   It seems plausible that the new version will be faster, but it's probably a good idea to run a quick benchmark to confirm. There's a `UTF8StringBenchmark` linked from https://github.com/apache/spark/pull/19180#issuecomment-329467838 : maybe we could adapt that and do a quick before-and-after comparison to just to double check?

##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java
##########
@@ -75,6 +75,42 @@ static long getPrefix(Object base, long offset, int numBytes) {
     return (IS_LITTLE_ENDIAN ? java.lang.Long.reverseBytes(p) : p) & ~mask;
   }
 
+  public static int compareBinary(byte[] leftBase, byte[] rightBase) {
+    return compareBinary(leftBase, Platform.BYTE_ARRAY_OFFSET, leftBase.length,

Review comment:
       +1. 
   
   It seems plausible that the new version will be faster, but it's probably a good idea to run a quick benchmark to confirm. There's a `UTF8StringBenchmark` linked from https://github.com/apache/spark/pull/19180#issuecomment-329467838 : maybe we could adapt that to work on byte arrays and do a quick before-and-after comparison to just to double check?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] JoshRosen commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
JoshRosen commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r731210073



##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java
##########
@@ -75,6 +75,42 @@ static long getPrefix(Object base, long offset, int numBytes) {
     return (IS_LITTLE_ENDIAN ? java.lang.Long.reverseBytes(p) : p) & ~mask;
   }
 
+  public static int compareBinary(byte[] leftBase, byte[] rightBase) {
+    return compareBinary(leftBase, Platform.BYTE_ARRAY_OFFSET, leftBase.length,

Review comment:
       +1. 
   
   It seems plausible that the new version will be faster, but it's probably a good idea to run a quick benchmark to confirm. There's a `UTF8StringBenchmark` linked from https://github.com/apache/spark/pull/19180#issuecomment-329467838 : maybe we could adapt that to work on byte arrays and do a quick before-and-after comparison to just to double check?
   
   **Edit**: just to clarify: I noticed that this benchmark is also linked in the PR description. As Sean points out, I think the key difference in this PR is whether we're using `getByte()` versus directly accessing the on-heap byte array (in the linked UTF8String benchmark, both the old and new code were using `getByte()`).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ulysses-you commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r734285699



##########
File path: sql/core/benchmarks/ByteArrayBenchmark-results.txt
##########
@@ -0,0 +1,16 @@
+================================================================================================

Review comment:
       ah, I just notice the env of GA is still different.  The two benchmark result based on:
   ```
   Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
   ```
   
   ```
   Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-948276121


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48955/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-948280948


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144483/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-948204042


   **[Test build #144483 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144483/testReport)** for PR 34310 at commit [`f9f11d6`](https://github.com/apache/spark/commit/f9f11d6c06cc9d63e7d4f342b010c70477b29cfa).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ulysses-you commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r733279529



##########
File path: sql/core/benchmarks/ByteArrayBenchmark-jdk11-results.txt
##########
@@ -0,0 +1,16 @@
+================================================================================================
+byte array comparisons
+================================================================================================
+
+Java HotSpot(TM) 64-Bit Server VM 11.0.12+8-LTS-237 on Mac OS X 11.5

Review comment:
       I also add the benchmark tool guide in pull request template, https://github.com/apache/spark/pull/34349




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945772115


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48842/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945676651


   **[Test build #144367 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144367/testReport)** for PR 34310 at commit [`a0b1c56`](https://github.com/apache/spark/commit/a0b1c5675fddd9695b3eeb24f311c0301ae4a788).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945839574


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144367/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ulysses-you commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r733265565



##########
File path: sql/core/benchmarks/ByteArrayBenchmark-jdk11-results.txt
##########
@@ -0,0 +1,16 @@
+================================================================================================
+byte array comparisons
+================================================================================================
+
+Java HotSpot(TM) 64-Bit Server VM 11.0.12+8-LTS-237 on Mac OS X 11.5

Review comment:
       thank you @JoshRosen , it's really a good tool for me. Updated the benchmark result from GA.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r734103844



##########
File path: sql/core/benchmarks/ByteArrayBenchmark-results.txt
##########
@@ -0,0 +1,16 @@
+================================================================================================

Review comment:
       Do we have 'before' numbers for these? you don't need to include them just want to verify that it also seemed to show an improvement like your local laptop one did




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ulysses-you commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-950444953


   thank you @srowen and @JoshRosen !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-946401904


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48868/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945530369


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48830/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945540647


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144352/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945772115


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48842/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] JoshRosen commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
JoshRosen commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r731210073



##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java
##########
@@ -75,6 +75,42 @@ static long getPrefix(Object base, long offset, int numBytes) {
     return (IS_LITTLE_ENDIAN ? java.lang.Long.reverseBytes(p) : p) & ~mask;
   }
 
+  public static int compareBinary(byte[] leftBase, byte[] rightBase) {
+    return compareBinary(leftBase, Platform.BYTE_ARRAY_OFFSET, leftBase.length,

Review comment:
       +1. 
   
   It seems plausible that the new version will be faster, but it's probably a good idea to run a quick benchmark to confirm. There's a `UTF8StringBenchmark` linked from https://github.com/apache/spark/pull/19180#issuecomment-329467838 : maybe we could adapt that to work on byte arrays and do a quick before-and-after comparison to just to double check?
   
   **Edit**: just to clarify: I noticed that this benchmark is also linked in the PR description. As Sean points out, I think the key difference in this PR is whether we're using `getByte()` versus directly accessing the on-heap byte array (in the linked UTF8String benchmark, both the old and new code were using `getByte()`.

##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java
##########
@@ -75,6 +75,42 @@ static long getPrefix(Object base, long offset, int numBytes) {
     return (IS_LITTLE_ENDIAN ? java.lang.Long.reverseBytes(p) : p) & ~mask;
   }
 
+  public static int compareBinary(byte[] leftBase, byte[] rightBase) {
+    return compareBinary(leftBase, Platform.BYTE_ARRAY_OFFSET, leftBase.length,

Review comment:
       +1. 
   
   It seems plausible that the new version will be faster, but it's probably a good idea to run a quick benchmark to confirm. There's a `UTF8StringBenchmark` linked from https://github.com/apache/spark/pull/19180#issuecomment-329467838 : maybe we could adapt that to work on byte arrays and do a quick before-and-after comparison to just to double check?
   
   **Edit**: just to clarify: I noticed that this benchmark is also linked in the PR description. As Sean points out, I think the key difference in this PR is whether we're using `getByte()` versus directly accessing the on-heap byte array (in the linked UTF8String benchmark, both the old and new code were using `getByte()`0.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] JoshRosen commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
JoshRosen commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r733028830



##########
File path: sql/core/benchmarks/ByteArrayBenchmark-jdk11-results.txt
##########
@@ -0,0 +1,16 @@
+================================================================================================
+byte array comparisons
+================================================================================================
+
+Java HotSpot(TM) 64-Bit Server VM 11.0.12+8-LTS-237 on Mac OS X 11.5

Review comment:
       Can you run the benchmarks in GitHub Actions according to the instructions at https://spark.apache.org/developer-tools.html#github-workflow-benchmarks and then include those results in this PR in place of these ones? This helps to ensure that checked-in benchmark results come from a consistent environment.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] JoshRosen commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
JoshRosen commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r733023154



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ByteArrayBenchmark.scala
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import scala.util.Random
+
+import org.apache.spark.benchmark.{Benchmark, BenchmarkBase}
+import org.apache.spark.unsafe.types.ByteArray
+
+/**
+ * Benchmark to measure performance for byte array comparisons.
+ * {{{
+ *   To run this benchmark:
+ *   1. without sbt:
+ *      bin/spark-submit --class <this class> --jars <spark core test jar> <sql core test jar>
+ *   2. build/sbt "sql/test:runMain <this class>"
+ *   3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain <this class>"
+ *      Results will be written to "benchmarks/<this class>-results.txt".
+ * }}}
+ */
+object ByteArrayBenchmark extends BenchmarkBase {
+
+  def byteArrayComparisons(iters: Long): Unit = {
+    val chars = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+    val random = new Random(0)
+    def randomBytes(min: Int, max: Int): Array[Byte] = {
+      val len = random.nextInt(max - min) + min
+      val bytes = new Array[Byte](len)
+      var i = 0
+      while (i < len) {
+        bytes(i) = chars.charAt(random.nextInt(chars.length())).toByte
+        i += 1
+      }
+      bytes
+    }
+
+    val count = 16 * 1000
+    val dataTiny = Seq.fill(count)(randomBytes(2, 7)).toArray
+    val dataSmall = Seq.fill(count)(randomBytes(8, 16)).toArray
+    val dataMedium = Seq.fill(count)(randomBytes(16, 32)).toArray
+    val dataLarge = Seq.fill(count)(randomBytes(512, 1024)).toArray
+    val dataLargeSlow = Seq.fill(count)(
+      Array.tabulate(512) {i => if (i < 511) 0.toByte else 1.toByte}).toArray
+
+    def compareBinary(data: Array[Array[Byte]]) = { _: Int =>
+      var sum = 0L
+      for (_ <- 0L until iters) {
+        var i = 0
+        while (i < count) {
+          sum += ByteArray.compareBinary(data(i), data((i + 1) % count))
+          i += 1
+        }
+      }
+    }
+
+    val benchmark = new Benchmark("Byte Array compareTo", count * iters, 25, output = output)
+    benchmark.addCase("2-7 byte")(compareBinary(dataTiny))
+    benchmark.addCase("8-16 byte")(compareBinary(dataSmall))
+    benchmark.addCase("16-32 byte")(compareBinary(dataMedium))
+    benchmark.addCase("512-1024 byte")(compareBinary(dataLarge))
+    benchmark.addCase("512 byte slow")(compareBinary(dataLargeSlow))
+    benchmark.addCase("2-7 byte")(compareBinary(dataTiny))

Review comment:
       It looks like this this case is listed twice. Maybe drop this line?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-948280043


   **[Test build #144483 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144483/testReport)** for PR 34310 at commit [`f9f11d6`](https://github.com/apache/spark/commit/f9f11d6c06cc9d63e7d4f342b010c70477b29cfa).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ulysses-you commented on a change in pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #34310:
URL: https://github.com/apache/spark/pull/34310#discussion_r734282012



##########
File path: sql/core/benchmarks/ByteArrayBenchmark-results.txt
##########
@@ -0,0 +1,16 @@
+================================================================================================

Review comment:
       It shows we still have the benefits with GA env.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-948260529


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48955/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen closed pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
srowen closed pull request #34310:
URL: https://github.com/apache/spark/pull/34310


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-950318211


   Merged to master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945539462


   **[Test build #144352 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144352/testReport)** for PR 34310 at commit [`65a8e64`](https://github.com/apache/spark/commit/65a8e6413f6d6d2265bd4c1ae64edf4be199d67b).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945540647


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144352/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945438149


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48830/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945630944


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48837/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ulysses-you commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945653354


   retest this please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34310: [SPARK-37037][SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34310:
URL: https://github.com/apache/spark/pull/34310#issuecomment-945744302


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48842/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org