You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2024/09/24 04:39:38 UTC
(spark) branch master updated: [SPARK-49753][BUILD] Upgrade ZSTD-JNI to 1.5.6-6
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 438a6e7782ec [SPARK-49753][BUILD] Upgrade ZSTD-JNI to 1.5.6-6
438a6e7782ec is described below
commit 438a6e7782ece23492928cfbb2d01e14104dfd9a
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Mon Sep 23 21:39:27 2024 -0700
[SPARK-49753][BUILD] Upgrade ZSTD-JNI to 1.5.6-6
### What changes were proposed in this pull request?
The pr aims to upgrade `zstd-jni` from `1.5.6-5` to `1.5.6-6`.
### Why are the changes needed?
The new version allow including compression level when training a dictionary: https://github.com/luben/zstd-jni/commit/3ca26eed6c84fb09c382854ead527188e643e206#diff-bd5c0f62db7cb85cac88c7b6cfad1c0e5e2f433ba45097761654829627b7a31c
All changes in the new version are as follows:
- https://github.com/luben/zstd-jni/compare/v1.5.6-5...v1.5.6-6
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GitHub Actions
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #48204 from LuciferYang/zstd-jni-1.5.6-6.
Lead-authored-by: yangjie01 <ya...@baidu.com>
Co-authored-by: YangJie <ya...@baidu.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../ZStandardBenchmark-jdk21-results.txt | 56 +++++++++++-----------
core/benchmarks/ZStandardBenchmark-results.txt | 56 +++++++++++-----------
dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +-
pom.xml | 2 +-
4 files changed, 58 insertions(+), 58 deletions(-)
diff --git a/core/benchmarks/ZStandardBenchmark-jdk21-results.txt b/core/benchmarks/ZStandardBenchmark-jdk21-results.txt
index b3bffea826e5..f6bd681451d5 100644
--- a/core/benchmarks/ZStandardBenchmark-jdk21-results.txt
+++ b/core/benchmarks/ZStandardBenchmark-jdk21-results.txt
@@ -2,48 +2,48 @@
Benchmark ZStandardCompressionCodec
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------
-Compression 10000 times at level 1 without buffer pool 657 670 14 0.0 65699.2 1.0X
-Compression 10000 times at level 2 without buffer pool 697 697 1 0.0 69673.4 0.9X
-Compression 10000 times at level 3 without buffer pool 799 802 3 0.0 79855.2 0.8X
-Compression 10000 times at level 1 with buffer pool 593 595 1 0.0 59326.9 1.1X
-Compression 10000 times at level 2 with buffer pool 622 624 3 0.0 62194.1 1.1X
-Compression 10000 times at level 3 with buffer pool 732 733 1 0.0 73178.6 0.9X
+Compression 10000 times at level 1 without buffer pool 659 676 16 0.0 65860.7 1.0X
+Compression 10000 times at level 2 without buffer pool 721 723 2 0.0 72135.5 0.9X
+Compression 10000 times at level 3 without buffer pool 815 816 1 0.0 81500.6 0.8X
+Compression 10000 times at level 1 with buffer pool 608 609 0 0.0 60846.6 1.1X
+Compression 10000 times at level 2 with buffer pool 645 647 3 0.0 64476.3 1.0X
+Compression 10000 times at level 3 with buffer pool 746 746 1 0.0 74584.0 0.9X
-OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------
-Decompression 10000 times from level 1 without buffer pool 813 820 11 0.0 81273.2 1.0X
-Decompression 10000 times from level 2 without buffer pool 810 813 3 0.0 80986.2 1.0X
-Decompression 10000 times from level 3 without buffer pool 812 813 2 0.0 81183.1 1.0X
-Decompression 10000 times from level 1 with buffer pool 746 747 2 0.0 74568.7 1.1X
-Decompression 10000 times from level 2 with buffer pool 744 746 2 0.0 74414.5 1.1X
-Decompression 10000 times from level 3 with buffer pool 745 746 1 0.0 74538.6 1.1X
+Decompression 10000 times from level 1 without buffer pool 828 829 1 0.0 82822.6 1.0X
+Decompression 10000 times from level 2 without buffer pool 829 829 1 0.0 82900.7 1.0X
+Decompression 10000 times from level 3 without buffer pool 828 833 8 0.0 82784.4 1.0X
+Decompression 10000 times from level 1 with buffer pool 758 760 2 0.0 75756.5 1.1X
+Decompression 10000 times from level 2 with buffer pool 758 758 1 0.0 75772.3 1.1X
+Decompression 10000 times from level 3 with buffer pool 759 759 0 0.0 75852.7 1.1X
-OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Parallel Compression at level 3: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Parallel Compression with 0 workers 48 49 1 0.0 374256.1 1.0X
-Parallel Compression with 1 workers 34 36 3 0.0 267557.3 1.4X
-Parallel Compression with 2 workers 34 38 2 0.0 263684.3 1.4X
-Parallel Compression with 4 workers 37 39 2 0.0 289956.1 1.3X
-Parallel Compression with 8 workers 39 41 1 0.0 306975.2 1.2X
-Parallel Compression with 16 workers 44 45 1 0.0 340992.0 1.1X
+Parallel Compression with 0 workers 58 59 1 0.0 452489.9 1.0X
+Parallel Compression with 1 workers 42 45 4 0.0 330066.0 1.4X
+Parallel Compression with 2 workers 40 42 1 0.0 312560.3 1.4X
+Parallel Compression with 4 workers 40 42 2 0.0 308802.7 1.5X
+Parallel Compression with 8 workers 41 45 3 0.0 321331.3 1.4X
+Parallel Compression with 16 workers 44 45 1 0.0 343311.5 1.3X
-OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Parallel Compression at level 9: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Parallel Compression with 0 workers 156 158 1 0.0 1220760.5 1.0X
-Parallel Compression with 1 workers 191 192 2 0.0 1495168.2 0.8X
-Parallel Compression with 2 workers 111 117 5 0.0 864459.9 1.4X
-Parallel Compression with 4 workers 106 109 2 0.0 831025.5 1.5X
-Parallel Compression with 8 workers 112 115 2 0.0 875732.7 1.4X
-Parallel Compression with 16 workers 110 114 2 0.0 858160.9 1.4X
+Parallel Compression with 0 workers 158 160 2 0.0 1234257.6 1.0X
+Parallel Compression with 1 workers 193 194 1 0.0 1507686.4 0.8X
+Parallel Compression with 2 workers 113 127 11 0.0 881068.0 1.4X
+Parallel Compression with 4 workers 109 111 2 0.0 849241.3 1.5X
+Parallel Compression with 8 workers 111 115 3 0.0 869455.2 1.4X
+Parallel Compression with 16 workers 113 116 2 0.0 881832.5 1.4X
diff --git a/core/benchmarks/ZStandardBenchmark-results.txt b/core/benchmarks/ZStandardBenchmark-results.txt
index b230f825feca..136f0333590c 100644
--- a/core/benchmarks/ZStandardBenchmark-results.txt
+++ b/core/benchmarks/ZStandardBenchmark-results.txt
@@ -2,48 +2,48 @@
Benchmark ZStandardCompressionCodec
================================================================================================
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------
-Compression 10000 times at level 1 without buffer pool 638 638 0 0.0 63765.0 1.0X
-Compression 10000 times at level 2 without buffer pool 675 676 1 0.0 67529.4 0.9X
-Compression 10000 times at level 3 without buffer pool 775 783 11 0.0 77531.6 0.8X
-Compression 10000 times at level 1 with buffer pool 572 573 1 0.0 57223.2 1.1X
-Compression 10000 times at level 2 with buffer pool 603 605 1 0.0 60323.7 1.1X
-Compression 10000 times at level 3 with buffer pool 720 727 6 0.0 71980.9 0.9X
+Compression 10000 times at level 1 without buffer pool 257 259 2 0.0 25704.2 1.0X
+Compression 10000 times at level 2 without buffer pool 674 676 2 0.0 67396.3 0.4X
+Compression 10000 times at level 3 without buffer pool 775 787 11 0.0 77497.9 0.3X
+Compression 10000 times at level 1 with buffer pool 573 574 0 0.0 57347.3 0.4X
+Compression 10000 times at level 2 with buffer pool 602 603 2 0.0 60162.8 0.4X
+Compression 10000 times at level 3 with buffer pool 722 725 3 0.0 72247.3 0.4X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------
-Decompression 10000 times from level 1 without buffer pool 584 585 1 0.0 58381.0 1.0X
-Decompression 10000 times from level 2 without buffer pool 585 585 0 0.0 58465.9 1.0X
-Decompression 10000 times from level 3 without buffer pool 585 586 1 0.0 58499.5 1.0X
-Decompression 10000 times from level 1 with buffer pool 534 534 0 0.0 53375.7 1.1X
-Decompression 10000 times from level 2 with buffer pool 533 533 0 0.0 53312.3 1.1X
-Decompression 10000 times from level 3 with buffer pool 533 533 1 0.0 53255.1 1.1X
+Decompression 10000 times from level 1 without buffer pool 176 177 1 0.1 17641.2 1.0X
+Decompression 10000 times from level 2 without buffer pool 176 178 1 0.1 17628.9 1.0X
+Decompression 10000 times from level 3 without buffer pool 175 176 0 0.1 17506.1 1.0X
+Decompression 10000 times from level 1 with buffer pool 151 152 1 0.1 15051.5 1.2X
+Decompression 10000 times from level 2 with buffer pool 150 151 1 0.1 14998.0 1.2X
+Decompression 10000 times from level 3 with buffer pool 150 151 0 0.1 15019.4 1.2X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Parallel Compression at level 3: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Parallel Compression with 0 workers 46 48 1 0.0 360483.5 1.0X
-Parallel Compression with 1 workers 34 36 2 0.0 265816.1 1.4X
-Parallel Compression with 2 workers 33 36 2 0.0 254525.8 1.4X
-Parallel Compression with 4 workers 34 37 1 0.0 266270.8 1.4X
-Parallel Compression with 8 workers 37 39 1 0.0 289289.2 1.2X
-Parallel Compression with 16 workers 41 43 1 0.0 320243.3 1.1X
+Parallel Compression with 0 workers 57 57 0 0.0 444425.2 1.0X
+Parallel Compression with 1 workers 42 44 3 0.0 325107.6 1.4X
+Parallel Compression with 2 workers 38 39 2 0.0 294840.0 1.5X
+Parallel Compression with 4 workers 36 37 1 0.0 282143.1 1.6X
+Parallel Compression with 8 workers 39 40 1 0.0 303793.6 1.5X
+Parallel Compression with 16 workers 41 43 1 0.0 324165.5 1.4X
-OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.5.0-1025-azure
+OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Linux 6.8.0-1014-azure
AMD EPYC 7763 64-Core Processor
Parallel Compression at level 9: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Parallel Compression with 0 workers 154 156 2 0.0 1205934.0 1.0X
-Parallel Compression with 1 workers 191 194 4 0.0 1495729.9 0.8X
-Parallel Compression with 2 workers 110 114 5 0.0 859158.9 1.4X
-Parallel Compression with 4 workers 105 108 3 0.0 822932.2 1.5X
-Parallel Compression with 8 workers 109 113 2 0.0 851560.0 1.4X
-Parallel Compression with 16 workers 111 115 2 0.0 870695.9 1.4X
+Parallel Compression with 0 workers 156 158 1 0.0 1220298.8 1.0X
+Parallel Compression with 1 workers 188 189 1 0.0 1467911.4 0.8X
+Parallel Compression with 2 workers 111 118 7 0.0 866985.2 1.4X
+Parallel Compression with 4 workers 106 109 2 0.0 827592.1 1.5X
+Parallel Compression with 8 workers 114 116 2 0.0 888419.5 1.4X
+Parallel Compression with 16 workers 111 115 2 0.0 868463.5 1.4X
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 419625f48fa1..88526995293f 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -280,4 +280,4 @@ xz/1.10//xz-1.10.jar
zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar
zookeeper-jute/3.9.2//zookeeper-jute-3.9.2.jar
zookeeper/3.9.2//zookeeper-3.9.2.jar
-zstd-jni/1.5.6-5//zstd-jni-1.5.6-5.jar
+zstd-jni/1.5.6-6//zstd-jni-1.5.6-6.jar
diff --git a/pom.xml b/pom.xml
index b7c87beec0f9..131e754da815 100644
--- a/pom.xml
+++ b/pom.xml
@@ -835,7 +835,7 @@
<dependency>
<groupId>com.github.luben</groupId>
<artifactId>zstd-jni</artifactId>
- <version>1.5.6-5</version>
+ <version>1.5.6-6</version>
</dependency>
<dependency>
<groupId>com.clearspring.analytics</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org