You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2022/09/04 13:27:39 UTC
[spark] branch master updated: [SPARK-40321][BUILD] Upgrade rocksdbjni to 7.5.3
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new de0e4cce034 [SPARK-40321][BUILD] Upgrade rocksdbjni to 7.5.3
de0e4cce034 is described below
commit de0e4cce0343cd82870b2904d75352ca50647706
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Sun Sep 4 08:27:20 2022 -0500
[SPARK-40321][BUILD] Upgrade rocksdbjni to 7.5.3
### What changes were proposed in this pull request?
This PR aims to upgrade RocksDB JNI library from 7.4.5 to 7.5.3.
### Why are the changes needed?
This version bring performance improvements(related to write throughput and compaction) and some bug fix，the release note as follows:
- https://github.com/facebook/rocksdb/releases/tag/v7.5.3
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GA
- The benchmark result :
**Before 7.4.5**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.340 0.265 0.510 0.510
dbCreation 4 78.628 3.499 303.732 303.732
naturalIndexCreateIterator 1024 0.005 0.002 1.473 0.006
naturalIndexDescendingCreateIterator 1024 0.006 0.005 0.062 0.007
naturalIndexDescendingIteration 1024 0.006 0.004 0.268 0.008
naturalIndexIteration 1024 0.006 0.004 0.053 0.009
randomDeleteIndexed 1024 0.025 0.019 0.183 0.033
randomDeletesNoIndex 1024 0.015 0.012 0.037 0.017
randomUpdatesIndexed 1024 0.081 0.033 30.773 0.084
randomUpdatesNoIndex 1024 0.034 0.031 0.580 0.037
randomWritesIndexed 1024 0.117 0.035 52.383 0.120
randomWritesNoIndex 1024 0.039 0.034 1.556 0.044
refIndexCreateIterator 1024 0.005 0.004 0.019 0.006
refIndexDescendingCreateIterator 1024 0.003 0.002 0.028 0.004
refIndexDescendingIteration 1024 0.006 0.005 0.043 0.008
refIndexIteration 1024 0.007 0.005 0.068 0.009
sequentialDeleteIndexed 1024 0.021 0.017 0.096 0.026
sequentialDeleteNoIndex 1024 0.015 0.012 0.041 0.019
sequentialUpdatesIndexed 1024 0.044 0.037 0.747 0.051
sequentialUpdatesNoIndex 1024 0.039 0.032 0.771 0.046
sequentialWritesIndexed 1024 0.049 0.042 1.766 0.056
sequentialWritesNoIndex 1024 0.039 0.032 2.282 0.041
```
**After 7.5.3**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.342 0.212 0.683 0.683
dbCreation 4 77.603 3.403 301.357 301.357
naturalIndexCreateIterator 1024 0.005 0.002 1.507 0.007
naturalIndexDescendingCreateIterator 1024 0.005 0.005 0.076 0.007
naturalIndexDescendingIteration 1024 0.006 0.004 0.244 0.010
naturalIndexIteration 1024 0.006 0.004 0.065 0.009
randomDeleteIndexed 1024 0.028 0.019 1.385 0.038
randomDeletesNoIndex 1024 0.016 0.013 0.042 0.019
randomUpdatesIndexed 1024 0.083 0.033 32.094 0.086
randomUpdatesNoIndex 1024 0.038 0.034 0.469 0.042
randomWritesIndexed 1024 0.122 0.034 54.517 0.126
randomWritesNoIndex 1024 0.042 0.037 1.369 0.048
refIndexCreateIterator 1024 0.005 0.005 0.019 0.007
refIndexDescendingCreateIterator 1024 0.003 0.003 0.034 0.005
refIndexDescendingIteration 1024 0.006 0.005 0.044 0.008
refIndexIteration 1024 0.007 0.005 0.072 0.011
sequentialDeleteIndexed 1024 0.022 0.017 0.104 0.027
sequentialDeleteNoIndex 1024 0.015 0.012 0.038 0.019
sequentialUpdatesIndexed 1024 0.046 0.039 0.862 0.054
sequentialUpdatesNoIndex 1024 0.043 0.030 0.752 0.055
sequentialWritesIndexed 1024 0.051 0.039 1.949 0.060
sequentialWritesNoIndex 1024 0.038 0.032 2.858 0.042
```
Closes #37783 from LuciferYang/SPARK-40321.
Authored-by: yangjie01 <ya...@baidu.com>
Signed-off-by: Sean Owen <sr...@gmail.com>
---
dev/deps/spark-deps-hadoop-2-hive-2.3 | 2 +-
dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +-
pom.xml | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2-hive-2.3 b/dev/deps/spark-deps-hadoop-2-hive-2.3
index 9f91c5e075b..66882b2a4b4 100644
--- a/dev/deps/spark-deps-hadoop-2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2-hive-2.3
@@ -234,7 +234,7 @@ pickle/1.2//pickle-1.2.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
py4j/0.10.9.7//py4j-0.10.9.7.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
-rocksdbjni/7.4.5//rocksdbjni-7.4.5.jar
+rocksdbjni/7.5.3//rocksdbjni-7.5.3.jar
scala-collection-compat_2.12/2.7.0//scala-collection-compat_2.12-2.7.0.jar
scala-compiler/2.12.16//scala-compiler-2.12.16.jar
scala-library/2.12.16//scala-library-2.12.16.jar
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 514afc41800..60dd8b190de 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -221,7 +221,7 @@ pickle/1.2//pickle-1.2.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
py4j/0.10.9.7//py4j-0.10.9.7.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
-rocksdbjni/7.4.5//rocksdbjni-7.4.5.jar
+rocksdbjni/7.5.3//rocksdbjni-7.5.3.jar
scala-collection-compat_2.12/2.7.0//scala-collection-compat_2.12-2.7.0.jar
scala-compiler/2.12.16//scala-compiler-2.12.16.jar
scala-library/2.12.16//scala-library-2.12.16.jar
diff --git a/pom.xml b/pom.xml
index 1fc694ba43f..1ac8dbd1d00 100644
--- a/pom.xml
+++ b/pom.xml
@@ -674,7 +674,7 @@
<dependency>
<groupId>org.rocksdb</groupId>
<artifactId>rocksdbjni</artifactId>
- <version>7.4.5</version>
+ <version>7.5.3</version>
</dependency>
<dependency>
<groupId>${leveldbjni.group}</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org