You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2022/07/27 04:52:35 UTC
[spark] branch master updated: [SPARK-39882][BUILD] Upgrade rocksdbjni to 7.4.3
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e79b9f05360 [SPARK-39882][BUILD] Upgrade rocksdbjni to 7.4.3
e79b9f05360 is described below
commit e79b9f053605369a60f15c606f1a4bc0b4f31329
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Tue Jul 26 21:52:23 2022 -0700
[SPARK-39882][BUILD] Upgrade rocksdbjni to 7.4.3
### What changes were proposed in this pull request?
This PR aims to upgrade RocksDB JNI library from 7.3.1 to 7.4.3.
### Why are the changes needed?
This version brings some bug fix, the release note as follows:
- https://github.com/facebook/rocksdb/releases/tag/v7.4.3
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GA
- The benchmark result :
**Before 7.3.1**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.346 0.266 0.539 0.539
dbCreation 4 78.389 3.583 301.763 301.763
naturalIndexCreateIterator 1024 0.005 0.002 1.469 0.007
naturalIndexDescendingCreateIterator 1024 0.007 0.006 0.064 0.008
naturalIndexDescendingIteration 1024 0.005 0.004 0.024 0.007
naturalIndexIteration 1024 0.006 0.004 0.053 0.008
randomDeleteIndexed 1024 0.026 0.019 0.293 0.035
randomDeletesNoIndex 1024 0.015 0.012 0.053 0.017
randomUpdatesIndexed 1024 0.081 0.033 30.752 0.083
randomUpdatesNoIndex 1024 0.037 0.034 0.502 0.041
randomWritesIndexed 1024 0.116 0.035 52.128 0.121
randomWritesNoIndex 1024 0.042 0.036 1.869 0.046
refIndexCreateIterator 1024 0.004 0.004 0.019 0.006
refIndexDescendingCreateIterator 1024 0.003 0.002 0.026 0.004
refIndexDescendingIteration 1024 0.006 0.005 0.042 0.008
refIndexIteration 1024 0.007 0.005 0.314 0.009
sequentialDeleteIndexed 1024 0.021 0.018 0.104 0.026
sequentialDeleteNoIndex 1024 0.015 0.012 0.053 0.019
sequentialUpdatesIndexed 1024 0.044 0.038 0.802 0.053
sequentialUpdatesNoIndex 1024 0.045 0.032 1.552 0.054
sequentialWritesIndexed 1024 0.048 0.040 1.862 0.055
sequentialWritesNoIndex 1024 0.044 0.032 2.929 0.048
```
**After 7.4.3**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.364 0.305 0.525 0.525
dbCreation 4 80.553 3.635 310.735 310.735
naturalIndexCreateIterator 1024 0.006 0.002 1.569 0.007
naturalIndexDescendingCreateIterator 1024 0.006 0.005 0.073 0.008
naturalIndexDescendingIteration 1024 0.006 0.004 0.269 0.008
naturalIndexIteration 1024 0.006 0.004 0.059 0.009
randomDeleteIndexed 1024 0.027 0.020 0.339 0.036
randomDeletesNoIndex 1024 0.015 0.012 0.037 0.018
randomUpdatesIndexed 1024 0.084 0.032 32.286 0.090
randomUpdatesNoIndex 1024 0.023 0.019 0.584 0.026
randomWritesIndexed 1024 0.122 0.035 53.989 0.129
randomWritesNoIndex 1024 0.027 0.022 1.547 0.031
refIndexCreateIterator 1024 0.005 0.005 0.027 0.007
refIndexDescendingCreateIterator 1024 0.003 0.003 0.035 0.005
refIndexDescendingIteration 1024 0.006 0.005 0.051 0.008
refIndexIteration 1024 0.007 0.005 0.096 0.009
sequentialDeleteIndexed 1024 0.023 0.018 1.396 0.027
sequentialDeleteNoIndex 1024 0.015 0.012 0.054 0.019
sequentialUpdatesIndexed 1024 0.045 0.040 0.773 0.053
sequentialUpdatesNoIndex 1024 0.027 0.021 0.867 0.034
sequentialWritesIndexed 1024 0.051 0.039 1.916 0.059
sequentialWritesNoIndex 1024 0.031 0.024 3.339 0.033
```
Closes #37302 from LuciferYang/SPARK-39882.
Authored-by: yangjie01 <ya...@baidu.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
dev/deps/spark-deps-hadoop-2-hive-2.3 | 2 +-
dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +-
pom.xml | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2-hive-2.3 b/dev/deps/spark-deps-hadoop-2-hive-2.3
index d8521647522..f26fb36675b 100644
--- a/dev/deps/spark-deps-hadoop-2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2-hive-2.3
@@ -234,7 +234,7 @@ pickle/1.2//pickle-1.2.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
py4j/0.10.9.5//py4j-0.10.9.5.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
-rocksdbjni/7.3.1//rocksdbjni-7.3.1.jar
+rocksdbjni/7.4.3//rocksdbjni-7.4.3.jar
scala-collection-compat_2.12/2.2.0//scala-collection-compat_2.12-2.2.0.jar
scala-compiler/2.12.16//scala-compiler-2.12.16.jar
scala-library/2.12.16//scala-library-2.12.16.jar
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3
index c50d8c66c21..2bc30a9f523 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -223,7 +223,7 @@ pickle/1.2//pickle-1.2.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
py4j/0.10.9.5//py4j-0.10.9.5.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
-rocksdbjni/7.3.1//rocksdbjni-7.3.1.jar
+rocksdbjni/7.4.3//rocksdbjni-7.4.3.jar
scala-collection-compat_2.12/2.2.0//scala-collection-compat_2.12-2.2.0.jar
scala-compiler/2.12.16//scala-compiler-2.12.16.jar
scala-library/2.12.16//scala-library-2.12.16.jar
diff --git a/pom.xml b/pom.xml
index 40d82261e43..34dbbf8b46d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -678,7 +678,7 @@
<dependency>
<groupId>org.rocksdb</groupId>
<artifactId>rocksdbjni</artifactId>
- <version>7.3.1</version>
+ <version>7.4.3</version>
</dependency>
<dependency>
<groupId>${leveldbjni.group}</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org