You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2022/06/25 11:40:20 UTC
[spark] branch master updated: [SPARK-39595][BUILD] Upgrade rocksdbjni to 7.3.1
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 094fa3de0b7 [SPARK-39595][BUILD] Upgrade rocksdbjni to 7.3.1
094fa3de0b7 is described below
commit 094fa3de0b78eb2e2117a4ce7c960299d6de7cc8
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Sat Jun 25 20:40:08 2022 +0900
[SPARK-39595][BUILD] Upgrade rocksdbjni to 7.3.1
### What changes were proposed in this pull request?
This PR aims to upgrade RocksDB JNI library from 7.2.2 to 7.3.1.
### Why are the changes needed?
This will bring some bug fix of RocksDB JNI.
- https://github.com/facebook/rocksdb/releases/tag/v7.3.1
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GA
- The benchmark result :
**Before 7.2.2**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.350 0.278 0.535 0.535
dbCreation 4 75.780 3.486 294.500 294.500
naturalIndexCreateIterator 1024 0.005 0.002 1.403 0.007
naturalIndexDescendingCreateIterator 1024 0.005 0.005 0.061 0.007
naturalIndexDescendingIteration 1024 0.006 0.004 0.244 0.008
naturalIndexIteration 1024 0.006 0.004 0.060 0.008
randomDeleteIndexed 1024 0.026 0.019 0.310 0.035
randomDeletesNoIndex 1024 0.015 0.013 0.044 0.017
randomUpdatesIndexed 1024 0.081 0.034 30.300 0.082
randomUpdatesNoIndex 1024 0.035 0.031 0.530 0.040
randomWritesIndexed 1024 0.115 0.034 51.592 0.120
randomWritesNoIndex 1024 0.038 0.033 1.612 0.042
refIndexCreateIterator 1024 0.004 0.004 0.016 0.006
refIndexDescendingCreateIterator 1024 0.003 0.003 0.027 0.005
refIndexDescendingIteration 1024 0.006 0.005 0.044 0.008
refIndexIteration 1024 0.007 0.005 0.051 0.009
sequentialDeleteIndexed 1024 0.021 0.017 0.094 0.026
sequentialDeleteNoIndex 1024 0.015 0.013 0.045 0.017
sequentialUpdatesIndexed 1024 0.043 0.038 0.752 0.049
sequentialUpdatesNoIndex 1024 0.040 0.031 0.748 0.049
sequentialWritesIndexed 1024 0.048 0.042 1.910 0.054
sequentialWritesNoIndex 1024 0.038 0.032 2.311 0.041
```
**After 7.3.1**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.340 0.274 0.509 0.509
dbCreation 4 76.287 3.469 296.518 296.518
naturalIndexCreateIterator 1024 0.005 0.002 1.425 0.007
naturalIndexDescendingCreateIterator 1024 0.005 0.005 0.061 0.007
naturalIndexDescendingIteration 1024 0.006 0.004 0.260 0.008
naturalIndexIteration 1024 0.006 0.004 0.060 0.008
randomDeleteIndexed 1024 0.026 0.019 0.283 0.033
randomDeletesNoIndex 1024 0.015 0.012 0.044 0.017
randomUpdatesIndexed 1024 0.081 0.032 30.606 0.084
randomUpdatesNoIndex 1024 0.034 0.031 0.545 0.036
randomWritesIndexed 1024 0.116 0.033 51.262 0.124
randomWritesNoIndex 1024 0.040 0.034 1.515 0.046
refIndexCreateIterator 1024 0.005 0.005 0.019 0.006
refIndexDescendingCreateIterator 1024 0.003 0.002 0.027 0.004
refIndexDescendingIteration 1024 0.006 0.005 0.041 0.008
refIndexIteration 1024 0.007 0.005 0.061 0.009
sequentialDeleteIndexed 1024 0.021 0.017 0.101 0.026
sequentialDeleteNoIndex 1024 0.016 0.013 0.041 0.018
sequentialUpdatesIndexed 1024 0.042 0.038 0.752 0.049
sequentialUpdatesNoIndex 1024 0.041 0.031 0.714 0.050
sequentialWritesIndexed 1024 0.048 0.039 1.887 0.055
sequentialWritesNoIndex 1024 0.039 0.031 2.305 0.042
```
From the benchmark results, there is no significant performance difference between these two versions.
Closes #36979 from LuciferYang/rocksdb-731.
Authored-by: yangjie01 <ya...@baidu.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
dev/deps/spark-deps-hadoop-2-hive-2.3 | 2 +-
dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +-
pom.xml | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2-hive-2.3 b/dev/deps/spark-deps-hadoop-2-hive-2.3
index bcd773264fa..c93b83ed26a 100644
--- a/dev/deps/spark-deps-hadoop-2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2-hive-2.3
@@ -235,7 +235,7 @@ pickle/1.2//pickle-1.2.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
py4j/0.10.9.5//py4j-0.10.9.5.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
-rocksdbjni/7.2.2//rocksdbjni-7.2.2.jar
+rocksdbjni/7.3.1//rocksdbjni-7.3.1.jar
scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar
scala-compiler/2.12.16//scala-compiler-2.12.16.jar
scala-library/2.12.16//scala-library-2.12.16.jar
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 53420957d75..5810ecd4ccc 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -224,7 +224,7 @@ pickle/1.2//pickle-1.2.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
py4j/0.10.9.5//py4j-0.10.9.5.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
-rocksdbjni/7.2.2//rocksdbjni-7.2.2.jar
+rocksdbjni/7.3.1//rocksdbjni-7.3.1.jar
scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar
scala-compiler/2.12.16//scala-compiler-2.12.16.jar
scala-library/2.12.16//scala-library-2.12.16.jar
diff --git a/pom.xml b/pom.xml
index d14e3aab24f..017ae07c03b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -678,7 +678,7 @@
<dependency>
<groupId>org.rocksdb</groupId>
<artifactId>rocksdbjni</artifactId>
- <version>7.2.2</version>
+ <version>7.3.1</version>
</dependency>
<dependency>
<groupId>${leveldbjni.group}</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org