You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "panbingkun (via GitHub)" <gi...@apache.org> on 2024/10/31 11:34:35 UTC
[PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
panbingkun opened a new pull request, #48721:
URL: https://github.com/apache/spark/pull/48721
### What changes were proposed in this pull request?
The pr aims to upgrade `ICU4J` from `75.1` to `76.1`.
### Why are the changes needed?
We need to keep the version up-to-date.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1824827328
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -429,7 +429,7 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
}
- test("invalid collationId") {
+ ignore("invalid collationId") {
Review Comment:
+1 for the above question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1826904536
##########
sql/core/benchmarks/CollationBenchmark-results.txt:
##########
@@ -1,54 +1,54 @@
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
--------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1372 1372 1 0.1 13718.5 1.0X
-UTF8_LCASE 3115 3116 1 0.0 31154.4 2.3X
-UNICODE 19813 19820 9 0.0 198132.2 14.4X
-UNICODE_CI 19669 19686 24 0.0 196694.2 14.3X
+UTF8_BINARY 1374 1375 1 0.1 13741.7 1.0X
+UTF8_LCASE 3163 3166 5 0.0 31630.7 2.3X
+UNICODE 19230 19242 16 0.0 192304.5 14.0X
+UNICODE_CI 18920 18930 15 0.0 189197.8 13.8X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
---------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1727 1728 1 0.1 17271.3 1.0X
-UTF8_LCASE 3034 3035 1 0.0 30337.2 1.8X
-UNICODE 19230 19243 18 0.0 192301.2 11.1X
-UNICODE_CI 19080 19082 3 0.0 190802.0 11.0X
+UTF8_BINARY 2674 2675 1 0.0 26744.0 1.0X
+UTF8_LCASE 4595 4598 5 0.0 45951.3 1.7X
+UNICODE 20327 20334 11 0.0 203265.8 7.6X
+UNICODE_CI 20219 20229 14 0.0 202188.0 7.6X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 3080 3080 0 0.0 30796.4 1.0X
-UTF8_LCASE 6436 6454 25 0.0 64360.0 2.1X
-UNICODE 68095 68167 101 0.0 680951.3 22.1X
-UNICODE_CI 62122 62123 2 0.0 621215.8 20.2X
+UTF8_BINARY 3106 3109 4 0.0 31059.9 1.0X
+UTF8_LCASE 6451 6464 19 0.0 64505.1 2.1X
+UNICODE 67033 67078 63 0.0 670329.8 21.6X
+UNICODE_CI 52301 52314 18 0.0 523012.5 16.8X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 8260 8261 1 0.0 82604.0 1.0X
-UTF8_LCASE 23629 23629 0 0.0 236286.4 2.9X
-UNICODE 364843 366078 1747 0.0 3648427.9 44.2X
-UNICODE_CI 425728 426449 1020 0.0 4257275.1 51.5X
+UTF8_BINARY 12370 12383 19 0.0 123697.8 1.0X
+UTF8_LCASE 28164 28166 3 0.0 281639.6 2.3X
+UNICODE 375367 375482 162 0.0 3753668.4 30.3X
+UNICODE_CI 418811 419177 517 0.0 4188111.1 33.9X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 6844 6848 5 0.0 68440.4 1.0X
-UTF8_LCASE 21849 21870 30 0.0 218486.3 3.2X
-UNICODE 363474 363811 476 0.0 3634738.4 53.1X
-UNICODE_CI 427563 428029 659 0.0 4275629.8 62.5X
+UTF8_BINARY 11261 11271 15 0.0 112607.7 1.0X
+UTF8_LCASE 26357 26366 12 0.0 263570.5 2.3X
+UNICODE 363306 364001 983 0.0 3633059.7 32.3X
+UNICODE_CI 420866 421292 603 0.0 4208656.7 37.4X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 6904 6907 4 0.0 69039.3 1.0X
-UTF8_LCASE 22007 22009 3 0.0 220067.8 3.2X
-UNICODE 376402 377858 2060 0.0 3764015.4 54.5X
-UNICODE_CI 444485 444809 458 0.0 4444850.8 64.4X
+UTF8_BINARY 12000 12018 26 0.0 120000.9 1.0X
Review Comment:
Relatively, `UTF8_BINARY` becomes slower than before. Do you happen to know any reason?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #48721:
URL: https://github.com/apache/spark/pull/48721#issuecomment-2453692814
- `org.apache.spark.sql.execution.benchmark.CollationBenchmark`
JDK 17:
JDK 21:
- `org.apache.spark.sql.execution.benchmark.CollationNonASCIIBenchmark`
JDK 17:
JDK 21:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1825593552
##########
sql/core/src/test/scala/org/apache/spark/sql/CollationSQLExpressionsSuite.scala:
##########
Review Comment:
`CollationKeys#writeSortKeyUpToQuaternary`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1825695456
##########
sql/core/src/test/scala/org/apache/spark/sql/CollationSQLExpressionsSuite.scala:
##########
Review Comment:
- Let's use the following code to reproduce it (collation: `UNICODE`)
```scala
object CollationKeySuite {
def main(args: Array[String]): Unit = {
val builder = new ULocale.Builder
builder.setLocale(ULocale.ROOT)
builder.setUnicodeLocaleKeyword("ks", "level3")
val resultLocale = builder.build
val collator = Collator.getInstance(resultLocale)
collator.freeze
val s = UTF8String.fromString("aa")
val hash = collator.getCollationKey(s.toValidString).hashCode()
println(hash)
}
}
```
- ICU4j 76.1, result:
```shell
10628395
```
- ICU4j 75.1, result:
```shell
10381418
```
- Through debugging, it was found that different versions of `icu4j` load different versions of underlying data resource files, such as `nfc.nrm`
A.`ICU4j 76.1` -> `15.1.0.0`
B. `ICU4j 75.1` -> `16.0.0.0`
- I guess it should be related to the PR below
https://github.com/unicode-org/icu/pull/2930
https://github.com/unicode-org/icu/pull/3028
https://github.com/unicode-org/icu/pull/3110
https://github.com/unicode-org/icu/pull/3130
https://github.com/unicode-org/icu/pull/3000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #48721:
URL: https://github.com/apache/spark/pull/48721#issuecomment-2449640295
let me update the benchmark result of `CollationBenchmark` and `CollationNonASCIIBenchmark`
- `CollationBenchmark`
JDK 17:
JDK 21:
- `CollationNonASCIIBenchmark`
JDK 17:
JDK 21:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1824629495
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -429,7 +429,7 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
}
- test("invalid collationId") {
+ ignore("invalid collationId") {
Review Comment:
why are we making this change in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1827391898
##########
sql/core/benchmarks/CollationNonASCIIBenchmark-results.txt:
##########
@@ -1,54 +1,54 @@
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
--------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 133 133 1 0.3 3317.1 1.0X
-UTF8_LCASE 7092 7097 6 0.0 177310.9 53.5X
-UNICODE 5946 5966 29 0.0 148638.1 44.8X
-UNICODE_CI 5715 5717 2 0.0 142885.1 43.1X
+UTF8_BINARY 125 128 6 0.3 3119.4 1.0X
+UTF8_LCASE 6635 6701 93 0.0 165887.1 53.2X
+UNICODE 5195 5217 31 0.0 129878.9 41.6X
+UNICODE_CI 5240 5263 33 0.0 131003.7 42.0X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
---------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 433 435 2 0.1 10816.6 1.0X
-UTF8_LCASE 7365 7369 5 0.0 184135.4 17.0X
-UNICODE 5785 5790 7 0.0 144616.9 13.4X
-UNICODE_CI 5742 5744 3 0.0 143557.1 13.3X
+UTF8_BINARY 416 419 2 0.1 10410.2 1.0X
+UTF8_LCASE 6909 6910 2 0.0 172713.2 16.6X
+UNICODE 5444 5480 50 0.0 136112.1 13.1X
+UNICODE_CI 5444 5470 37 0.0 136087.6 13.1X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 410 411 1 0.1 10246.1 1.0X
-UTF8_LCASE 3588 3589 1 0.0 89698.8 8.8X
-UNICODE 15788 15802 20 0.0 394702.8 38.5X
-UNICODE_CI 12179 12192 19 0.0 304466.6 29.7X
+UTF8_BINARY 380 386 4 0.1 9510.3 1.0X
+UTF8_LCASE 3390 3398 10 0.0 84756.7 8.9X
+UNICODE 13224 13243 27 0.0 330604.1 34.8X
+UNICODE_CI 10524 10635 157 0.0 263095.0 27.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1367 1370 4 0.0 34182.9 1.0X
-UTF8_LCASE 9644 9645 1 0.0 241101.2 7.1X
-UNICODE 67169 67171 3 0.0 1679230.1 49.1X
-UNICODE_CI 79077 79209 188 0.0 1976919.1 57.8X
+UTF8_BINARY 1740 1758 26 0.0 43489.5 1.0X
+UTF8_LCASE 10697 10708 15 0.0 267435.0 6.1X
+UNICODE 61284 61521 336 0.0 1532092.7 35.2X
+UNICODE_CI 70030 70051 29 0.0 1750749.7 40.3X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1064 1067 3 0.0 26608.1 1.0X
-UTF8_LCASE 6487 6491 4 0.0 162186.5 6.1X
-UNICODE 68473 68523 71 0.0 1711818.5 64.3X
-UNICODE_CI 79374 79419 64 0.0 1984338.0 74.6X
+UTF8_BINARY 1663 1675 18 0.0 41563.3 1.0X
+UTF8_LCASE 6786 6787 2 0.0 169640.2 4.1X
+UNICODE 60580 60668 124 0.0 1514498.6 36.4X
+UNICODE_CI 71018 71018 0 0.0 1775444.7 42.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1002 1004 2 0.0 25061.8 1.0X
-UTF8_LCASE 6052 6052 0 0.0 151298.7 6.0X
-UNICODE 74506 74551 64 0.0 1862644.2 74.3X
-UNICODE_CI 83607 83756 211 0.0 2090164.5 83.4X
+UTF8_BINARY 1778 1779 2 0.0 44450.1 1.0X
Review Comment:
The regression ±70% seems significant. Could you figure out this is because of upgrade of ICU4J or just the bechmark is outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][BUILD][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #48721:
URL: https://github.com/apache/spark/pull/48721#issuecomment-2455976216
> To @MaxGekk and all.
>
> * [SPARK-50216](https://issues.apache.org/jira/browse/SPARK-50216) (`Investigate UTF8_BINARY regression`) is filed as a blocker issue for Apache Spark 4.0.0.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1825543236
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollationExpressionSuite.scala:
##########
@@ -168,6 +168,7 @@ class CollationExpressionSuite extends SparkFunSuite with ExpressionEvalHelper {
}
test("CollationKey generates correct collation key for collated string") {
+ val b: Byte = 0x2B
Review Comment:
This is the same, the `CollationKey` is different.
- In version `75.1`, its value is `0x2A` (`42`),
- while in version `76.1`, its value is `0x2B` (`43`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #48721:
URL: https://github.com/apache/spark/pull/48721#issuecomment-2451740948
@uros-db @dongjoon-hyun @stefankandic @MaxGekk
The detailed explanation has been updated, this PR is ready for review.
Thank you very much for the review, if you has free time. ❤️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1828062021
##########
sql/core/benchmarks/CollationNonASCIIBenchmark-results.txt:
##########
@@ -1,54 +1,54 @@
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
--------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 133 133 1 0.3 3317.1 1.0X
-UTF8_LCASE 7092 7097 6 0.0 177310.9 53.5X
-UNICODE 5946 5966 29 0.0 148638.1 44.8X
-UNICODE_CI 5715 5717 2 0.0 142885.1 43.1X
+UTF8_BINARY 125 128 6 0.3 3119.4 1.0X
+UTF8_LCASE 6635 6701 93 0.0 165887.1 53.2X
+UNICODE 5195 5217 31 0.0 129878.9 41.6X
+UNICODE_CI 5240 5263 33 0.0 131003.7 42.0X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
---------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 433 435 2 0.1 10816.6 1.0X
-UTF8_LCASE 7365 7369 5 0.0 184135.4 17.0X
-UNICODE 5785 5790 7 0.0 144616.9 13.4X
-UNICODE_CI 5742 5744 3 0.0 143557.1 13.3X
+UTF8_BINARY 416 419 2 0.1 10410.2 1.0X
+UTF8_LCASE 6909 6910 2 0.0 172713.2 16.6X
+UNICODE 5444 5480 50 0.0 136112.1 13.1X
+UNICODE_CI 5444 5470 37 0.0 136087.6 13.1X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 410 411 1 0.1 10246.1 1.0X
-UTF8_LCASE 3588 3589 1 0.0 89698.8 8.8X
-UNICODE 15788 15802 20 0.0 394702.8 38.5X
-UNICODE_CI 12179 12192 19 0.0 304466.6 29.7X
+UTF8_BINARY 380 386 4 0.1 9510.3 1.0X
+UTF8_LCASE 3390 3398 10 0.0 84756.7 8.9X
+UNICODE 13224 13243 27 0.0 330604.1 34.8X
+UNICODE_CI 10524 10635 157 0.0 263095.0 27.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1367 1370 4 0.0 34182.9 1.0X
-UTF8_LCASE 9644 9645 1 0.0 241101.2 7.1X
-UNICODE 67169 67171 3 0.0 1679230.1 49.1X
-UNICODE_CI 79077 79209 188 0.0 1976919.1 57.8X
+UTF8_BINARY 1740 1758 26 0.0 43489.5 1.0X
+UTF8_LCASE 10697 10708 15 0.0 267435.0 6.1X
+UNICODE 61284 61521 336 0.0 1532092.7 35.2X
+UNICODE_CI 70030 70051 29 0.0 1750749.7 40.3X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1064 1067 3 0.0 26608.1 1.0X
-UTF8_LCASE 6487 6491 4 0.0 162186.5 6.1X
-UNICODE 68473 68523 71 0.0 1711818.5 64.3X
-UNICODE_CI 79374 79419 64 0.0 1984338.0 74.6X
+UTF8_BINARY 1663 1675 18 0.0 41563.3 1.0X
+UTF8_LCASE 6786 6787 2 0.0 169640.2 4.1X
+UNICODE 60580 60668 124 0.0 1514498.6 36.4X
+UNICODE_CI 71018 71018 0 0.0 1775444.7 42.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1002 1004 2 0.0 25061.8 1.0X
-UTF8_LCASE 6052 6052 0 0.0 151298.7 6.0X
-UNICODE 74506 74551 64 0.0 1862644.2 74.3X
-UNICODE_CI 83607 83756 211 0.0 2090164.5 83.4X
+UTF8_BINARY 1778 1779 2 0.0 44450.1 1.0X
Review Comment:
BTW, that new JIRA issue will not be a blocker for this dependency change PR, right, @MaxGekk ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1828098979
##########
sql/core/benchmarks/CollationNonASCIIBenchmark-results.txt:
##########
@@ -1,54 +1,54 @@
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
--------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 133 133 1 0.3 3317.1 1.0X
-UTF8_LCASE 7092 7097 6 0.0 177310.9 53.5X
-UNICODE 5946 5966 29 0.0 148638.1 44.8X
-UNICODE_CI 5715 5717 2 0.0 142885.1 43.1X
+UTF8_BINARY 125 128 6 0.3 3119.4 1.0X
+UTF8_LCASE 6635 6701 93 0.0 165887.1 53.2X
+UNICODE 5195 5217 31 0.0 129878.9 41.6X
+UNICODE_CI 5240 5263 33 0.0 131003.7 42.0X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
---------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 433 435 2 0.1 10816.6 1.0X
-UTF8_LCASE 7365 7369 5 0.0 184135.4 17.0X
-UNICODE 5785 5790 7 0.0 144616.9 13.4X
-UNICODE_CI 5742 5744 3 0.0 143557.1 13.3X
+UTF8_BINARY 416 419 2 0.1 10410.2 1.0X
+UTF8_LCASE 6909 6910 2 0.0 172713.2 16.6X
+UNICODE 5444 5480 50 0.0 136112.1 13.1X
+UNICODE_CI 5444 5470 37 0.0 136087.6 13.1X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 410 411 1 0.1 10246.1 1.0X
-UTF8_LCASE 3588 3589 1 0.0 89698.8 8.8X
-UNICODE 15788 15802 20 0.0 394702.8 38.5X
-UNICODE_CI 12179 12192 19 0.0 304466.6 29.7X
+UTF8_BINARY 380 386 4 0.1 9510.3 1.0X
+UTF8_LCASE 3390 3398 10 0.0 84756.7 8.9X
+UNICODE 13224 13243 27 0.0 330604.1 34.8X
+UNICODE_CI 10524 10635 157 0.0 263095.0 27.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1367 1370 4 0.0 34182.9 1.0X
-UTF8_LCASE 9644 9645 1 0.0 241101.2 7.1X
-UNICODE 67169 67171 3 0.0 1679230.1 49.1X
-UNICODE_CI 79077 79209 188 0.0 1976919.1 57.8X
+UTF8_BINARY 1740 1758 26 0.0 43489.5 1.0X
+UTF8_LCASE 10697 10708 15 0.0 267435.0 6.1X
+UNICODE 61284 61521 336 0.0 1532092.7 35.2X
+UNICODE_CI 70030 70051 29 0.0 1750749.7 40.3X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1064 1067 3 0.0 26608.1 1.0X
-UTF8_LCASE 6487 6491 4 0.0 162186.5 6.1X
-UNICODE 68473 68523 71 0.0 1711818.5 64.3X
-UNICODE_CI 79374 79419 64 0.0 1984338.0 74.6X
+UTF8_BINARY 1663 1675 18 0.0 41563.3 1.0X
+UTF8_LCASE 6786 6787 2 0.0 169640.2 4.1X
+UNICODE 60580 60668 124 0.0 1514498.6 36.4X
+UNICODE_CI 71018 71018 0 0.0 1775444.7 42.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1002 1004 2 0.0 25061.8 1.0X
-UTF8_LCASE 6052 6052 0 0.0 151298.7 6.0X
-UNICODE 74506 74551 64 0.0 1862644.2 74.3X
-UNICODE_CI 83607 83756 211 0.0 2090164.5 83.4X
+UTF8_BINARY 1778 1779 2 0.0 44450.1 1.0X
Review Comment:
Thank you for the confirmation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1824629952
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollationExpressionSuite.scala:
##########
@@ -167,7 +167,7 @@ class CollationExpressionSuite extends SparkFunSuite with ExpressionEvalHelper {
}
}
- test("CollationKey generates correct collation key for collated string") {
+ ignore("CollationKey generates correct collation key for collated string") {
Review Comment:
why are we making this change in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1825543236
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollationExpressionSuite.scala:
##########
@@ -168,6 +168,7 @@ class CollationExpressionSuite extends SparkFunSuite with ExpressionEvalHelper {
}
test("CollationKey generates correct collation key for collated string") {
+ val b: Byte = 0x2B
Review Comment:
This is the same, the `CollationKey ( m_key_ )` is different.
- In version `75.1`, its value is `0x2A` (`42`),
- while in version `76.1`, its value is `0x2B` (`43`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1826904250
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollationExpressionSuite.scala:
##########
@@ -168,6 +168,7 @@ class CollationExpressionSuite extends SparkFunSuite with ExpressionEvalHelper {
}
test("CollationKey generates correct collation key for collated string") {
+ val b: Byte = 0x2B
Review Comment:
Could you add a comment before line 171, please, @panbingkun ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1827120077
##########
sql/core/benchmarks/CollationBenchmark-results.txt:
##########
@@ -1,54 +1,54 @@
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
--------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1372 1372 1 0.1 13718.5 1.0X
-UTF8_LCASE 3115 3116 1 0.0 31154.4 2.3X
-UNICODE 19813 19820 9 0.0 198132.2 14.4X
-UNICODE_CI 19669 19686 24 0.0 196694.2 14.3X
+UTF8_BINARY 1374 1375 1 0.1 13741.7 1.0X
+UTF8_LCASE 3163 3166 5 0.0 31630.7 2.3X
+UNICODE 19230 19242 16 0.0 192304.5 14.0X
+UNICODE_CI 18920 18930 15 0.0 189197.8 13.8X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
---------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1727 1728 1 0.1 17271.3 1.0X
-UTF8_LCASE 3034 3035 1 0.0 30337.2 1.8X
-UNICODE 19230 19243 18 0.0 192301.2 11.1X
-UNICODE_CI 19080 19082 3 0.0 190802.0 11.0X
+UTF8_BINARY 2674 2675 1 0.0 26744.0 1.0X
+UTF8_LCASE 4595 4598 5 0.0 45951.3 1.7X
+UNICODE 20327 20334 11 0.0 203265.8 7.6X
+UNICODE_CI 20219 20229 14 0.0 202188.0 7.6X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 3080 3080 0 0.0 30796.4 1.0X
-UTF8_LCASE 6436 6454 25 0.0 64360.0 2.1X
-UNICODE 68095 68167 101 0.0 680951.3 22.1X
-UNICODE_CI 62122 62123 2 0.0 621215.8 20.2X
+UTF8_BINARY 3106 3109 4 0.0 31059.9 1.0X
+UTF8_LCASE 6451 6464 19 0.0 64505.1 2.1X
+UNICODE 67033 67078 63 0.0 670329.8 21.6X
+UNICODE_CI 52301 52314 18 0.0 523012.5 16.8X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 8260 8261 1 0.0 82604.0 1.0X
-UTF8_LCASE 23629 23629 0 0.0 236286.4 2.9X
-UNICODE 364843 366078 1747 0.0 3648427.9 44.2X
-UNICODE_CI 425728 426449 1020 0.0 4257275.1 51.5X
+UTF8_BINARY 12370 12383 19 0.0 123697.8 1.0X
+UTF8_LCASE 28164 28166 3 0.0 281639.6 2.3X
+UNICODE 375367 375482 162 0.0 3753668.4 30.3X
+UNICODE_CI 418811 419177 517 0.0 4188111.1 33.9X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 6844 6848 5 0.0 68440.4 1.0X
-UTF8_LCASE 21849 21870 30 0.0 218486.3 3.2X
-UNICODE 363474 363811 476 0.0 3634738.4 53.1X
-UNICODE_CI 427563 428029 659 0.0 4275629.8 62.5X
+UTF8_BINARY 11261 11271 15 0.0 112607.7 1.0X
+UTF8_LCASE 26357 26366 12 0.0 263570.5 2.3X
+UNICODE 363306 364001 983 0.0 3633059.7 32.3X
+UNICODE_CI 420866 421292 603 0.0 4208656.7 37.4X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 6904 6907 4 0.0 69039.3 1.0X
-UTF8_LCASE 22007 22009 3 0.0 220067.8 3.2X
-UNICODE 376402 377858 2060 0.0 3764015.4 54.5X
-UNICODE_CI 444485 444809 458 0.0 4444850.8 64.4X
+UTF8_BINARY 12000 12018 26 0.0 120000.9 1.0X
Review Comment:
Let me run the benchmark a few more times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][BUILD][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #48721:
URL: https://github.com/apache/spark/pull/48721#issuecomment-2455291954
Merged to master for Apache Spark 4.0.0 on February 2025.
Thank you, @panbingkun , @MaxGekk , @uros-db .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1825695456
##########
sql/core/src/test/scala/org/apache/spark/sql/CollationSQLExpressionsSuite.scala:
##########
Review Comment:
- Let's use the following code to reproduce it (collation: `UNICODE`)
```scala
object CollationKeySuite {
def main(args: Array[String]): Unit = {
val builder = new ULocale.Builder
builder.setLocale(ULocale.ROOT)
builder.setUnicodeLocaleKeyword("ks", "level3")
val resultLocale = builder.build
val collator = Collator.getInstance(resultLocale)
collator.freeze
val s = UTF8String.fromString("aa")
val hash = collator.getCollationKey(s.toValidString).hashCode()
println(hash)
}
}
```
- ICU4j 76.1, result:
```shell
10628395
```
- ICU4j 75.1, result:
```shell
10381418
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1825695456
##########
sql/core/src/test/scala/org/apache/spark/sql/CollationSQLExpressionsSuite.scala:
##########
Review Comment:
- Let's use the following code to reproduce it (collation: `UNICODE`)
```scala
object CollationKeySuite {
def main(args: Array[String]): Unit = {
val builder = new ULocale.Builder
builder.setLocale(ULocale.ROOT)
builder.setUnicodeLocaleKeyword("ks", "level3")
val resultLocale = builder.build
val collator = Collator.getInstance(resultLocale)
collator.freeze
val s = UTF8String.fromString("aa")
val hash = collator.getCollationKey(s.toValidString).hashCode()
println(hash)
}
}
```
- ICU4j 76.1, result:
```shell
10628395
```
- ICU4j 75.1, result:
```shell
10381418
```
- Through debugging, it was found that different versions of `icu4j` load different versions of underlying data resource files, such as `nfc.nrm`
A. `ICU4j 76.1` -> `15.1.0.0`
B. `ICU4j 75.1` -> `16.0.0.0`
- I `guess` it should be related to the PR below (`Unicode 15.1` -> `Unicode 16`)
https://github.com/unicode-org/icu/pull/2930
https://github.com/unicode-org/icu/pull/3028
https://github.com/unicode-org/icu/pull/3110
https://github.com/unicode-org/icu/pull/3130
https://github.com/unicode-org/icu/pull/3000
- ref docs
https://unicode-org.github.io/icu/download/76.html#release-overview
<img width="743" alt="image" src="https://github.com/user-attachments/assets/db085e48-e8c4-4bfd-840b-3237e03b9c85">
https://unicode-org.github.io/icu/download/76.html#common-changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1827122567
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollationExpressionSuite.scala:
##########
@@ -168,6 +168,7 @@ class CollationExpressionSuite extends SparkFunSuite with ExpressionEvalHelper {
}
test("CollationKey generates correct collation key for collated string") {
+ val b: Byte = 0x2B
Review Comment:
Sure, done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1827597395
##########
sql/core/benchmarks/CollationNonASCIIBenchmark-results.txt:
##########
@@ -1,54 +1,54 @@
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
--------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 133 133 1 0.3 3317.1 1.0X
-UTF8_LCASE 7092 7097 6 0.0 177310.9 53.5X
-UNICODE 5946 5966 29 0.0 148638.1 44.8X
-UNICODE_CI 5715 5717 2 0.0 142885.1 43.1X
+UTF8_BINARY 125 128 6 0.3 3119.4 1.0X
+UTF8_LCASE 6635 6701 93 0.0 165887.1 53.2X
+UNICODE 5195 5217 31 0.0 129878.9 41.6X
+UNICODE_CI 5240 5263 33 0.0 131003.7 42.0X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
---------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 433 435 2 0.1 10816.6 1.0X
-UTF8_LCASE 7365 7369 5 0.0 184135.4 17.0X
-UNICODE 5785 5790 7 0.0 144616.9 13.4X
-UNICODE_CI 5742 5744 3 0.0 143557.1 13.3X
+UTF8_BINARY 416 419 2 0.1 10410.2 1.0X
+UTF8_LCASE 6909 6910 2 0.0 172713.2 16.6X
+UNICODE 5444 5480 50 0.0 136112.1 13.1X
+UNICODE_CI 5444 5470 37 0.0 136087.6 13.1X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 410 411 1 0.1 10246.1 1.0X
-UTF8_LCASE 3588 3589 1 0.0 89698.8 8.8X
-UNICODE 15788 15802 20 0.0 394702.8 38.5X
-UNICODE_CI 12179 12192 19 0.0 304466.6 29.7X
+UTF8_BINARY 380 386 4 0.1 9510.3 1.0X
+UTF8_LCASE 3390 3398 10 0.0 84756.7 8.9X
+UNICODE 13224 13243 27 0.0 330604.1 34.8X
+UNICODE_CI 10524 10635 157 0.0 263095.0 27.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1367 1370 4 0.0 34182.9 1.0X
-UTF8_LCASE 9644 9645 1 0.0 241101.2 7.1X
-UNICODE 67169 67171 3 0.0 1679230.1 49.1X
-UNICODE_CI 79077 79209 188 0.0 1976919.1 57.8X
+UTF8_BINARY 1740 1758 26 0.0 43489.5 1.0X
+UTF8_LCASE 10697 10708 15 0.0 267435.0 6.1X
+UNICODE 61284 61521 336 0.0 1532092.7 35.2X
+UNICODE_CI 70030 70051 29 0.0 1750749.7 40.3X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1064 1067 3 0.0 26608.1 1.0X
-UTF8_LCASE 6487 6491 4 0.0 162186.5 6.1X
-UNICODE 68473 68523 71 0.0 1711818.5 64.3X
-UNICODE_CI 79374 79419 64 0.0 1984338.0 74.6X
+UTF8_BINARY 1663 1675 18 0.0 41563.3 1.0X
+UTF8_LCASE 6786 6787 2 0.0 169640.2 4.1X
+UNICODE 60580 60668 124 0.0 1514498.6 36.4X
+UNICODE_CI 71018 71018 0 0.0 1775444.7 42.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1002 1004 2 0.0 25061.8 1.0X
-UTF8_LCASE 6052 6052 0 0.0 151298.7 6.0X
-UNICODE 74506 74551 64 0.0 1862644.2 74.3X
-UNICODE_CI 83607 83756 211 0.0 2090164.5 83.4X
+UTF8_BINARY 1778 1779 2 0.0 44450.1 1.0X
Review Comment:
From the testing, it seems that the benchmark is `outdated`, the data is as follows:
<img width="1352" alt="image" src="https://github.com/user-attachments/assets/b2af3562-2184-420f-80ce-8562fc2a5f25">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #48721:
URL: https://github.com/apache/spark/pull/48721#issuecomment-2454478755
<img width="1366" alt="image" src="https://github.com/user-attachments/assets/2be85af2-e5f4-4e74-a5bf-2d912a344a90">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1825539871
##########
sql/core/src/test/scala/org/apache/spark/sql/CollationSQLExpressionsSuite.scala:
##########
Review Comment:
The reason for this change is that the `CollationKey` returned by `Collator.getCollationKey(...)` are different. As for why they are different, I am investigating it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1825722866
##########
sql/core/src/test/scala/org/apache/spark/sql/CollationSQLExpressionsSuite.scala:
##########
@@ -200,10 +200,10 @@ class CollationSQLExpressionsSuite
Murmur3HashTestCase("Spark ", "UTF8_BINARY_RTRIM", 1779328737),
Murmur3HashTestCase("Spark", "UTF8_LCASE", -1928694360),
Murmur3HashTestCase("Spark ", "UTF8_LCASE_RTRIM", -1928694360),
- Murmur3HashTestCase("SQL", "UNICODE", -1923567940),
- Murmur3HashTestCase("SQL ", "UNICODE_RTRIM", -1923567940),
- Murmur3HashTestCase("SQL", "UNICODE_CI", 1029527950),
- Murmur3HashTestCase("SQL ", "UNICODE_CI_RTRIM", 1029527950)
+ Murmur3HashTestCase("SQL", "UNICODE", 1483684981),
Review Comment:
This is also related to `CollationKey`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1825722211
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollationExpressionSuite.scala:
##########
@@ -168,6 +168,7 @@ class CollationExpressionSuite extends SparkFunSuite with ExpressionEvalHelper {
}
test("CollationKey generates correct collation key for collated string") {
+ val b: Byte = 0x2B
Review Comment:
This is also related to `CollationKey`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1825326886
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollationExpressionSuite.scala:
##########
@@ -167,7 +167,7 @@ class CollationExpressionSuite extends SparkFunSuite with ExpressionEvalHelper {
}
}
- test("CollationKey generates correct collation key for collated string") {
+ ignore("CollationKey generates correct collation key for collated string") {
Review Comment:
It is not the final 'answer' to this PR, I am still investigating. 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1825695456
##########
sql/core/src/test/scala/org/apache/spark/sql/CollationSQLExpressionsSuite.scala:
##########
Review Comment:
- Let's use the following code to reproduce it (collation: `UNICODE`)
```scala
object CollationKeySuite {
def main(args: Array[String]): Unit = {
val builder = new ULocale.Builder
builder.setLocale(ULocale.ROOT)
builder.setUnicodeLocaleKeyword("ks", "level3")
val resultLocale = builder.build
val collator = Collator.getInstance(resultLocale)
collator.freeze
val s = UTF8String.fromString("aa")
val hash = collator.getCollationKey(s.toValidString).hashCode()
println(hash)
}
}
```
- ICU4j 76.1, result:
```shell
10628395
```
- ICU4j 75.1, result:
```shell
10381418
```
- Through debugging, it was found that different versions of `icu4j` load different versions of underlying data resource files, such as `nfc.nrm`
A.`ICU4j 76.1` -> `15.1.0.0`
B. `ICU4j 75.1` -> `16.0.0.0`
- I guess it should be related to the PR below (`Unicode 15.1` -> `Unicode 16`)
https://github.com/unicode-org/icu/pull/2930
https://github.com/unicode-org/icu/pull/3028
https://github.com/unicode-org/icu/pull/3110
https://github.com/unicode-org/icu/pull/3130
https://github.com/unicode-org/icu/pull/3000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1825695456
##########
sql/core/src/test/scala/org/apache/spark/sql/CollationSQLExpressionsSuite.scala:
##########
Review Comment:
- Let's use the following code to reproduce it (collation: `UNICODE`)
```scala
object CollationKeySuite {
def main(args: Array[String]): Unit = {
val builder = new ULocale.Builder
builder.setLocale(ULocale.ROOT)
builder.setUnicodeLocaleKeyword("ks", "level3")
val resultLocale = builder.build
val collator = Collator.getInstance(resultLocale)
collator.freeze
val s = UTF8String.fromString("aa")
val hash = collator.getCollationKey(s.toValidString).hashCode()
println(hash)
}
}
```
- ICU4j 76.1, result:
```shell
10628395
```
- ICU4j 75.1, result:
```shell
10381418
```
- Through debugging, it was found that different versions of `icu4j` load different versions of underlying data resource files, such as `nfc.nrm`
A. `ICU4j 76.1` -> `15.1.0.0`
B. `ICU4j 75.1` -> `16.0.0.0`
- I `guess` it should be related to the PR below (`Unicode 15.1` -> `Unicode 16`)
https://github.com/unicode-org/icu/pull/2930
https://github.com/unicode-org/icu/pull/3028
https://github.com/unicode-org/icu/pull/3110
https://github.com/unicode-org/icu/pull/3130
https://github.com/unicode-org/icu/pull/3000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1825695456
##########
sql/core/src/test/scala/org/apache/spark/sql/CollationSQLExpressionsSuite.scala:
##########
Review Comment:
- Let's use the following code to reproduce it (collation: `UNICODE`)
```scala
object CollationKeySuite {
def main(args: Array[String]): Unit = {
val builder = new ULocale.Builder
builder.setLocale(ULocale.ROOT)
builder.setUnicodeLocaleKeyword("ks", "level3")
val resultLocale = builder.build
val collator = Collator.getInstance(resultLocale)
collator.freeze
val s = UTF8String.fromString("aa")
val hash = collator.getCollationKey(s.toValidString).hashCode()
println(hash)
}
}
```
- ICU4j 76.1, result:
```shell
10628395
```
- ICU4j 75.1, result:
```shell
10381418
```
- Through debugging, it was found that different versions of `icu4j` load different versions of underlying data resource files, such as `nfc.nrm`
A.`ICU4j 76.1` -> `15.1.0.0`
B. `ICU4j 75.1` -> `16.0.0.0`
-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1827597395
##########
sql/core/benchmarks/CollationNonASCIIBenchmark-results.txt:
##########
@@ -1,54 +1,54 @@
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
--------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 133 133 1 0.3 3317.1 1.0X
-UTF8_LCASE 7092 7097 6 0.0 177310.9 53.5X
-UNICODE 5946 5966 29 0.0 148638.1 44.8X
-UNICODE_CI 5715 5717 2 0.0 142885.1 43.1X
+UTF8_BINARY 125 128 6 0.3 3119.4 1.0X
+UTF8_LCASE 6635 6701 93 0.0 165887.1 53.2X
+UNICODE 5195 5217 31 0.0 129878.9 41.6X
+UNICODE_CI 5240 5263 33 0.0 131003.7 42.0X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
---------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 433 435 2 0.1 10816.6 1.0X
-UTF8_LCASE 7365 7369 5 0.0 184135.4 17.0X
-UNICODE 5785 5790 7 0.0 144616.9 13.4X
-UNICODE_CI 5742 5744 3 0.0 143557.1 13.3X
+UTF8_BINARY 416 419 2 0.1 10410.2 1.0X
+UTF8_LCASE 6909 6910 2 0.0 172713.2 16.6X
+UNICODE 5444 5480 50 0.0 136112.1 13.1X
+UNICODE_CI 5444 5470 37 0.0 136087.6 13.1X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 410 411 1 0.1 10246.1 1.0X
-UTF8_LCASE 3588 3589 1 0.0 89698.8 8.8X
-UNICODE 15788 15802 20 0.0 394702.8 38.5X
-UNICODE_CI 12179 12192 19 0.0 304466.6 29.7X
+UTF8_BINARY 380 386 4 0.1 9510.3 1.0X
+UTF8_LCASE 3390 3398 10 0.0 84756.7 8.9X
+UNICODE 13224 13243 27 0.0 330604.1 34.8X
+UNICODE_CI 10524 10635 157 0.0 263095.0 27.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1367 1370 4 0.0 34182.9 1.0X
-UTF8_LCASE 9644 9645 1 0.0 241101.2 7.1X
-UNICODE 67169 67171 3 0.0 1679230.1 49.1X
-UNICODE_CI 79077 79209 188 0.0 1976919.1 57.8X
+UTF8_BINARY 1740 1758 26 0.0 43489.5 1.0X
+UTF8_LCASE 10697 10708 15 0.0 267435.0 6.1X
+UNICODE 61284 61521 336 0.0 1532092.7 35.2X
+UNICODE_CI 70030 70051 29 0.0 1750749.7 40.3X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1064 1067 3 0.0 26608.1 1.0X
-UTF8_LCASE 6487 6491 4 0.0 162186.5 6.1X
-UNICODE 68473 68523 71 0.0 1711818.5 64.3X
-UNICODE_CI 79374 79419 64 0.0 1984338.0 74.6X
+UTF8_BINARY 1663 1675 18 0.0 41563.3 1.0X
+UTF8_LCASE 6786 6787 2 0.0 169640.2 4.1X
+UNICODE 60580 60668 124 0.0 1514498.6 36.4X
+UNICODE_CI 71018 71018 0 0.0 1775444.7 42.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1002 1004 2 0.0 25061.8 1.0X
-UTF8_LCASE 6052 6052 0 0.0 151298.7 6.0X
-UNICODE 74506 74551 64 0.0 1862644.2 74.3X
-UNICODE_CI 83607 83756 211 0.0 2090164.5 83.4X
+UTF8_BINARY 1778 1779 2 0.0 44450.1 1.0X
Review Comment:
From the testing, it seems that the benchmark is `outdated`, the data is as follows:
<img width="1375" alt="image" src="https://github.com/user-attachments/assets/4dc34d54-6561-4a1f-80e5-88ea836b3983">
<img width="1352" alt="image" src="https://github.com/user-attachments/assets/b2af3562-2184-420f-80ce-8562fc2a5f25">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1827122567
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollationExpressionSuite.scala:
##########
@@ -168,6 +168,7 @@ class CollationExpressionSuite extends SparkFunSuite with ExpressionEvalHelper {
}
test("CollationKey generates correct collation key for collated string") {
+ val b: Byte = 0x2B
Review Comment:
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1827986403
##########
sql/core/benchmarks/CollationNonASCIIBenchmark-results.txt:
##########
@@ -1,54 +1,54 @@
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
--------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 133 133 1 0.3 3317.1 1.0X
-UTF8_LCASE 7092 7097 6 0.0 177310.9 53.5X
-UNICODE 5946 5966 29 0.0 148638.1 44.8X
-UNICODE_CI 5715 5717 2 0.0 142885.1 43.1X
+UTF8_BINARY 125 128 6 0.3 3119.4 1.0X
+UTF8_LCASE 6635 6701 93 0.0 165887.1 53.2X
+UNICODE 5195 5217 31 0.0 129878.9 41.6X
+UNICODE_CI 5240 5263 33 0.0 131003.7 42.0X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
---------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 433 435 2 0.1 10816.6 1.0X
-UTF8_LCASE 7365 7369 5 0.0 184135.4 17.0X
-UNICODE 5785 5790 7 0.0 144616.9 13.4X
-UNICODE_CI 5742 5744 3 0.0 143557.1 13.3X
+UTF8_BINARY 416 419 2 0.1 10410.2 1.0X
+UTF8_LCASE 6909 6910 2 0.0 172713.2 16.6X
+UNICODE 5444 5480 50 0.0 136112.1 13.1X
+UNICODE_CI 5444 5470 37 0.0 136087.6 13.1X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 410 411 1 0.1 10246.1 1.0X
-UTF8_LCASE 3588 3589 1 0.0 89698.8 8.8X
-UNICODE 15788 15802 20 0.0 394702.8 38.5X
-UNICODE_CI 12179 12192 19 0.0 304466.6 29.7X
+UTF8_BINARY 380 386 4 0.1 9510.3 1.0X
+UTF8_LCASE 3390 3398 10 0.0 84756.7 8.9X
+UNICODE 13224 13243 27 0.0 330604.1 34.8X
+UNICODE_CI 10524 10635 157 0.0 263095.0 27.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1367 1370 4 0.0 34182.9 1.0X
-UTF8_LCASE 9644 9645 1 0.0 241101.2 7.1X
-UNICODE 67169 67171 3 0.0 1679230.1 49.1X
-UNICODE_CI 79077 79209 188 0.0 1976919.1 57.8X
+UTF8_BINARY 1740 1758 26 0.0 43489.5 1.0X
+UTF8_LCASE 10697 10708 15 0.0 267435.0 6.1X
+UNICODE 61284 61521 336 0.0 1532092.7 35.2X
+UNICODE_CI 70030 70051 29 0.0 1750749.7 40.3X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1064 1067 3 0.0 26608.1 1.0X
-UTF8_LCASE 6487 6491 4 0.0 162186.5 6.1X
-UNICODE 68473 68523 71 0.0 1711818.5 64.3X
-UNICODE_CI 79374 79419 64 0.0 1984338.0 74.6X
+UTF8_BINARY 1663 1675 18 0.0 41563.3 1.0X
+UTF8_LCASE 6786 6787 2 0.0 169640.2 4.1X
+UNICODE 60580 60668 124 0.0 1514498.6 36.4X
+UNICODE_CI 71018 71018 0 0.0 1775444.7 42.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1002 1004 2 0.0 25061.8 1.0X
-UTF8_LCASE 6052 6052 0 0.0 151298.7 6.0X
-UNICODE 74506 74551 64 0.0 1862644.2 74.3X
-UNICODE_CI 83607 83756 211 0.0 2090164.5 83.4X
+UTF8_BINARY 1778 1779 2 0.0 44450.1 1.0X
Review Comment:
@panbingkun Thank you for checking this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1828031983
##########
sql/core/benchmarks/CollationNonASCIIBenchmark-results.txt:
##########
@@ -1,54 +1,54 @@
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
--------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 133 133 1 0.3 3317.1 1.0X
-UTF8_LCASE 7092 7097 6 0.0 177310.9 53.5X
-UNICODE 5946 5966 29 0.0 148638.1 44.8X
-UNICODE_CI 5715 5717 2 0.0 142885.1 43.1X
+UTF8_BINARY 125 128 6 0.3 3119.4 1.0X
+UTF8_LCASE 6635 6701 93 0.0 165887.1 53.2X
+UNICODE 5195 5217 31 0.0 129878.9 41.6X
+UNICODE_CI 5240 5263 33 0.0 131003.7 42.0X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
---------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 433 435 2 0.1 10816.6 1.0X
-UTF8_LCASE 7365 7369 5 0.0 184135.4 17.0X
-UNICODE 5785 5790 7 0.0 144616.9 13.4X
-UNICODE_CI 5742 5744 3 0.0 143557.1 13.3X
+UTF8_BINARY 416 419 2 0.1 10410.2 1.0X
+UTF8_LCASE 6909 6910 2 0.0 172713.2 16.6X
+UNICODE 5444 5480 50 0.0 136112.1 13.1X
+UNICODE_CI 5444 5470 37 0.0 136087.6 13.1X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 410 411 1 0.1 10246.1 1.0X
-UTF8_LCASE 3588 3589 1 0.0 89698.8 8.8X
-UNICODE 15788 15802 20 0.0 394702.8 38.5X
-UNICODE_CI 12179 12192 19 0.0 304466.6 29.7X
+UTF8_BINARY 380 386 4 0.1 9510.3 1.0X
+UTF8_LCASE 3390 3398 10 0.0 84756.7 8.9X
+UNICODE 13224 13243 27 0.0 330604.1 34.8X
+UNICODE_CI 10524 10635 157 0.0 263095.0 27.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1367 1370 4 0.0 34182.9 1.0X
-UTF8_LCASE 9644 9645 1 0.0 241101.2 7.1X
-UNICODE 67169 67171 3 0.0 1679230.1 49.1X
-UNICODE_CI 79077 79209 188 0.0 1976919.1 57.8X
+UTF8_BINARY 1740 1758 26 0.0 43489.5 1.0X
+UTF8_LCASE 10697 10708 15 0.0 267435.0 6.1X
+UNICODE 61284 61521 336 0.0 1532092.7 35.2X
+UNICODE_CI 70030 70051 29 0.0 1750749.7 40.3X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1064 1067 3 0.0 26608.1 1.0X
-UTF8_LCASE 6487 6491 4 0.0 162186.5 6.1X
-UNICODE 68473 68523 71 0.0 1711818.5 64.3X
-UNICODE_CI 79374 79419 64 0.0 1984338.0 74.6X
+UTF8_BINARY 1663 1675 18 0.0 41563.3 1.0X
+UTF8_LCASE 6786 6787 2 0.0 169640.2 4.1X
+UNICODE 60580 60668 124 0.0 1514498.6 36.4X
+UNICODE_CI 71018 71018 0 0.0 1775444.7 42.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1002 1004 2 0.0 25061.8 1.0X
-UTF8_LCASE 6052 6052 0 0.0 151298.7 6.0X
-UNICODE 74506 74551 64 0.0 1862644.2 74.3X
-UNICODE_CI 83607 83756 211 0.0 2090164.5 83.4X
+UTF8_BINARY 1778 1779 2 0.0 44450.1 1.0X
Review Comment:
It would be nice to identify which commit causes the regression. @panbingkun Could you open an JIRA for investigation, please. Also if you have time, please, do the investigation. Seems the regression was introduced recently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1824632521
##########
sql/core/src/test/scala/org/apache/spark/sql/CollationSQLExpressionsSuite.scala:
##########
Review Comment:
I wouldn't expect to see any changes here. Do we know why these hashes have changed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1828077161
##########
sql/core/benchmarks/CollationNonASCIIBenchmark-results.txt:
##########
@@ -1,54 +1,54 @@
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
--------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 133 133 1 0.3 3317.1 1.0X
-UTF8_LCASE 7092 7097 6 0.0 177310.9 53.5X
-UNICODE 5946 5966 29 0.0 148638.1 44.8X
-UNICODE_CI 5715 5717 2 0.0 142885.1 43.1X
+UTF8_BINARY 125 128 6 0.3 3119.4 1.0X
+UTF8_LCASE 6635 6701 93 0.0 165887.1 53.2X
+UNICODE 5195 5217 31 0.0 129878.9 41.6X
+UNICODE_CI 5240 5263 33 0.0 131003.7 42.0X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
---------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 433 435 2 0.1 10816.6 1.0X
-UTF8_LCASE 7365 7369 5 0.0 184135.4 17.0X
-UNICODE 5785 5790 7 0.0 144616.9 13.4X
-UNICODE_CI 5742 5744 3 0.0 143557.1 13.3X
+UTF8_BINARY 416 419 2 0.1 10410.2 1.0X
+UTF8_LCASE 6909 6910 2 0.0 172713.2 16.6X
+UNICODE 5444 5480 50 0.0 136112.1 13.1X
+UNICODE_CI 5444 5470 37 0.0 136087.6 13.1X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 410 411 1 0.1 10246.1 1.0X
-UTF8_LCASE 3588 3589 1 0.0 89698.8 8.8X
-UNICODE 15788 15802 20 0.0 394702.8 38.5X
-UNICODE_CI 12179 12192 19 0.0 304466.6 29.7X
+UTF8_BINARY 380 386 4 0.1 9510.3 1.0X
+UTF8_LCASE 3390 3398 10 0.0 84756.7 8.9X
+UNICODE 13224 13243 27 0.0 330604.1 34.8X
+UNICODE_CI 10524 10635 157 0.0 263095.0 27.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1367 1370 4 0.0 34182.9 1.0X
-UTF8_LCASE 9644 9645 1 0.0 241101.2 7.1X
-UNICODE 67169 67171 3 0.0 1679230.1 49.1X
-UNICODE_CI 79077 79209 188 0.0 1976919.1 57.8X
+UTF8_BINARY 1740 1758 26 0.0 43489.5 1.0X
+UTF8_LCASE 10697 10708 15 0.0 267435.0 6.1X
+UNICODE 61284 61521 336 0.0 1532092.7 35.2X
+UNICODE_CI 70030 70051 29 0.0 1750749.7 40.3X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1064 1067 3 0.0 26608.1 1.0X
-UTF8_LCASE 6487 6491 4 0.0 162186.5 6.1X
-UNICODE 68473 68523 71 0.0 1711818.5 64.3X
-UNICODE_CI 79374 79419 64 0.0 1984338.0 74.6X
+UTF8_BINARY 1663 1675 18 0.0 41563.3 1.0X
+UTF8_LCASE 6786 6787 2 0.0 169640.2 4.1X
+UNICODE 60580 60668 124 0.0 1514498.6 36.4X
+UNICODE_CI 71018 71018 0 0.0 1775444.7 42.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1002 1004 2 0.0 25061.8 1.0X
-UTF8_LCASE 6052 6052 0 0.0 151298.7 6.0X
-UNICODE 74506 74551 64 0.0 1862644.2 74.3X
-UNICODE_CI 83607 83756 211 0.0 2090164.5 83.4X
+UTF8_BINARY 1778 1779 2 0.0 44450.1 1.0X
Review Comment:
right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1826904605
##########
sql/core/src/test/scala/org/apache/spark/sql/CollationSuite.scala:
##########
@@ -1979,62 +1979,63 @@ class CollationSuite extends DatasourceV2SQLBase with AdaptiveSparkPlanHelper {
// verify that the output ordering is as expected (UTF8_BINARY, UTF8_LCASE, etc.)
val df = sql("SELECT * FROM collations() limit 10")
+ val icvVersion = "76.1.0.0"
Review Comment:
Thank you for adding this variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #48721:
URL: https://github.com/apache/spark/pull/48721#discussion_r1827398656
##########
sql/core/benchmarks/CollationNonASCIIBenchmark-results.txt:
##########
@@ -1,54 +1,54 @@
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - equalsFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
--------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 133 133 1 0.3 3317.1 1.0X
-UTF8_LCASE 7092 7097 6 0.0 177310.9 53.5X
-UNICODE 5946 5966 29 0.0 148638.1 44.8X
-UNICODE_CI 5715 5717 2 0.0 142885.1 43.1X
+UTF8_BINARY 125 128 6 0.3 3119.4 1.0X
+UTF8_LCASE 6635 6701 93 0.0 165887.1 53.2X
+UNICODE 5195 5217 31 0.0 129878.9 41.6X
+UNICODE_CI 5240 5263 33 0.0 131003.7 42.0X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - compareFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
---------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 433 435 2 0.1 10816.6 1.0X
-UTF8_LCASE 7365 7369 5 0.0 184135.4 17.0X
-UNICODE 5785 5790 7 0.0 144616.9 13.4X
-UNICODE_CI 5742 5744 3 0.0 143557.1 13.3X
+UTF8_BINARY 416 419 2 0.1 10410.2 1.0X
+UTF8_LCASE 6909 6910 2 0.0 172713.2 16.6X
+UNICODE 5444 5480 50 0.0 136112.1 13.1X
+UNICODE_CI 5444 5470 37 0.0 136087.6 13.1X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - hashFunction: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 410 411 1 0.1 10246.1 1.0X
-UTF8_LCASE 3588 3589 1 0.0 89698.8 8.8X
-UNICODE 15788 15802 20 0.0 394702.8 38.5X
-UNICODE_CI 12179 12192 19 0.0 304466.6 29.7X
+UTF8_BINARY 380 386 4 0.1 9510.3 1.0X
+UTF8_LCASE 3390 3398 10 0.0 84756.7 8.9X
+UNICODE 13224 13243 27 0.0 330604.1 34.8X
+UNICODE_CI 10524 10635 157 0.0 263095.0 27.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - contains: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1367 1370 4 0.0 34182.9 1.0X
-UTF8_LCASE 9644 9645 1 0.0 241101.2 7.1X
-UNICODE 67169 67171 3 0.0 1679230.1 49.1X
-UNICODE_CI 79077 79209 188 0.0 1976919.1 57.8X
+UTF8_BINARY 1740 1758 26 0.0 43489.5 1.0X
+UTF8_LCASE 10697 10708 15 0.0 267435.0 6.1X
+UNICODE 61284 61521 336 0.0 1532092.7 35.2X
+UNICODE_CI 70030 70051 29 0.0 1750749.7 40.3X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - startsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1064 1067 3 0.0 26608.1 1.0X
-UTF8_LCASE 6487 6491 4 0.0 162186.5 6.1X
-UNICODE 68473 68523 71 0.0 1711818.5 64.3X
-UNICODE_CI 79374 79419 64 0.0 1984338.0 74.6X
+UTF8_BINARY 1663 1675 18 0.0 41563.3 1.0X
+UTF8_LCASE 6786 6787 2 0.0 169640.2 4.1X
+UNICODE 60580 60668 124 0.0 1514498.6 36.4X
+UNICODE_CI 71018 71018 0 0.0 1775444.7 42.7X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
collation unit benchmarks - endsWith: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative time
------------------------------------------------------------------------------------------------------------------------
-UTF8_BINARY 1002 1004 2 0.0 25061.8 1.0X
-UTF8_LCASE 6052 6052 0 0.0 151298.7 6.0X
-UNICODE 74506 74551 64 0.0 1862644.2 74.3X
-UNICODE_CI 83607 83756 211 0.0 2090164.5 83.4X
+UTF8_BINARY 1778 1779 2 0.0 44450.1 1.0X
Review Comment:
Yes, I have already run the benchmark on the `master`. Let me take a look. Wait a moment. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #48721:
URL: https://github.com/apache/spark/pull/48721#issuecomment-2455285757
To @MaxGekk and all.
- SPARK-50216 (`Investigate UTF8_BINARY regression`) is filed as a blocker issue for Apache Spark 4.0.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50189][BUILD][SQL] Upgrade ICU4J to `76.1` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #48721: [SPARK-50189][BUILD][SQL] Upgrade ICU4J to `76.1`
URL: https://github.com/apache/spark/pull/48721
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org