You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2022/08/18 13:47:55 UTC
[spark] branch master updated: [SPARK-39975][BUILD] Upgrade rocksdbjni to 7.4.5
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9381fe6891d [SPARK-39975][BUILD] Upgrade rocksdbjni to 7.4.5
9381fe6891d is described below
commit 9381fe6891de5a9898a6cfd10e7ed846c4b47089
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Thu Aug 18 08:47:40 2022 -0500
[SPARK-39975][BUILD] Upgrade rocksdbjni to 7.4.5
### What changes were proposed in this pull request?
This PR aims to upgrade RocksDB JNI library from 7.4.4 to 7.4.5.
### Why are the changes needed?
Rocksdbjni 7.4.5 only contains one patch compared with version 7.4.4 and this patch fixes a bug that causes data loss:
- [Fix serious FSDirectory use-after-Close bug (missing fsync) ](https://github.com/facebook/rocksdb/pull/10460)
```
Fix a bug starting in 7.4.0 in which some fsync operations might be skipped in a DB after any DropColumnFamily on that DB, until it is re-opened. This can lead to data loss on power loss. (For custom FileSystem implementations, this could lead to FSDirectory::Fsync or FSDirectory::Close after the first FSDirectory::Close; Also, valgrind could report call to close() with fd=-1.)
```
The release note as follows:
- https://github.com/facebook/rocksdb/releases/tag/v7.4.5
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GA
- The benchmark result :
**Before 7.4.4**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.362 0.292 0.525 0.525
dbCreation 4 77.126 3.527 299.975 299.975
naturalIndexCreateIterator 1024 0.005 0.002 1.465 0.006
naturalIndexDescendingCreateIterator 1024 0.006 0.005 0.064 0.008
naturalIndexDescendingIteration 1024 0.006 0.004 0.266 0.008
naturalIndexIteration 1024 0.006 0.004 0.049 0.009
randomDeleteIndexed 1024 0.026 0.018 0.288 0.034
randomDeletesNoIndex 1024 0.015 0.013 0.041 0.018
randomUpdatesIndexed 1024 0.082 0.033 31.463 0.083
randomUpdatesNoIndex 1024 0.035 0.031 0.618 0.039
randomWritesIndexed 1024 0.118 0.035 52.198 0.122
randomWritesNoIndex 1024 0.042 0.035 1.650 0.048
refIndexCreateIterator 1024 0.005 0.004 0.020 0.007
refIndexDescendingCreateIterator 1024 0.003 0.002 0.027 0.004
refIndexDescendingIteration 1024 0.006 0.005 0.044 0.008
refIndexIteration 1024 0.007 0.005 0.071 0.011
sequentialDeleteIndexed 1024 0.022 0.018 0.222 0.027
sequentialDeleteNoIndex 1024 0.015 0.012 0.044 0.018
sequentialUpdatesIndexed 1024 0.048 0.040 0.860 0.062
sequentialUpdatesNoIndex 1024 0.038 0.030 0.871 0.045
sequentialWritesIndexed 1024 0.051 0.039 2.405 0.058
sequentialWritesNoIndex 1024 0.038 0.032 2.937 0.042
```
**After 7.4.5**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.392 0.239 0.738 0.738
dbCreation 4 78.741 3.764 303.031 303.031
naturalIndexCreateIterator 1024 0.005 0.002 1.549 0.006
naturalIndexDescendingCreateIterator 1024 0.005 0.005 0.068 0.007
naturalIndexDescendingIteration 1024 0.006 0.004 0.295 0.010
naturalIndexIteration 1024 0.006 0.004 0.084 0.009
randomDeleteIndexed 1024 0.027 0.020 0.335 0.037
randomDeletesNoIndex 1024 0.015 0.013 0.039 0.017
randomUpdatesIndexed 1024 0.084 0.034 31.642 0.088
randomUpdatesNoIndex 1024 0.037 0.035 0.573 0.041
randomWritesIndexed 1024 0.122 0.035 53.724 0.126
randomWritesNoIndex 1024 0.043 0.036 1.694 0.047
refIndexCreateIterator 1024 0.005 0.005 0.023 0.007
refIndexDescendingCreateIterator 1024 0.003 0.003 0.029 0.005
refIndexDescendingIteration 1024 0.006 0.005 0.044 0.008
refIndexIteration 1024 0.007 0.005 0.084 0.011
sequentialDeleteIndexed 1024 0.024 0.018 2.101 0.027
sequentialDeleteNoIndex 1024 0.016 0.013 0.044 0.020
sequentialUpdatesIndexed 1024 0.046 0.040 0.870 0.053
sequentialUpdatesNoIndex 1024 0.040 0.031 0.750 0.048
sequentialWritesIndexed 1024 0.051 0.039 2.093 0.060
sequentialWritesNoIndex 1024 0.038 0.031 2.827 0.041
```
Closes #37543 from LuciferYang/SPARK-39975.
Authored-by: yangjie01 <ya...@baidu.com>
Signed-off-by: Sean Owen <sr...@gmail.com>
---
dev/deps/spark-deps-hadoop-2-hive-2.3 | 2 +-
dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +-
pom.xml | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2-hive-2.3 b/dev/deps/spark-deps-hadoop-2-hive-2.3
index b68295f4eb5..e2b193fb3f4 100644
--- a/dev/deps/spark-deps-hadoop-2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2-hive-2.3
@@ -234,7 +234,7 @@ pickle/1.2//pickle-1.2.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
py4j/0.10.9.7//py4j-0.10.9.7.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
-rocksdbjni/7.4.4//rocksdbjni-7.4.4.jar
+rocksdbjni/7.4.5//rocksdbjni-7.4.5.jar
scala-collection-compat_2.12/2.2.0//scala-collection-compat_2.12-2.2.0.jar
scala-compiler/2.12.16//scala-compiler-2.12.16.jar
scala-library/2.12.16//scala-library-2.12.16.jar
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 02b065f10df..550336aa59e 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -221,7 +221,7 @@ pickle/1.2//pickle-1.2.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
py4j/0.10.9.7//py4j-0.10.9.7.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
-rocksdbjni/7.4.4//rocksdbjni-7.4.4.jar
+rocksdbjni/7.4.5//rocksdbjni-7.4.5.jar
scala-collection-compat_2.12/2.2.0//scala-collection-compat_2.12-2.2.0.jar
scala-compiler/2.12.16//scala-compiler-2.12.16.jar
scala-library/2.12.16//scala-library-2.12.16.jar
diff --git a/pom.xml b/pom.xml
index 77d676051d2..261d5278c7a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -674,7 +674,7 @@
<dependency>
<groupId>org.rocksdb</groupId>
<artifactId>rocksdbjni</artifactId>
- <version>7.4.4</version>
+ <version>7.4.5</version>
</dependency>
<dependency>
<groupId>${leveldbjni.group}</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org