You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/09/03 13:03:06 UTC
[GitHub] [spark] LuciferYang opened a new pull request, #37783: [SPARK-40321][BUILD] Upgrade rocksdbjni to 7.5.3
LuciferYang opened a new pull request, #37783:
URL: https://github.com/apache/spark/pull/37783
### What changes were proposed in this pull request?
This PR aims to upgrade RocksDB JNI library from 7.4.5 to 7.5.3.
### Why are the changes needed?
This version bring performance improvements(related to write throughput and compaction) and some bug fix，the release note as follows:
- https://github.com/facebook/rocksdb/releases/tag/v7.5.3
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GA
- The benchmark result :
**Before 7.4.5**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.340 0.265 0.510 0.510
dbCreation 4 78.628 3.499 303.732 303.732
naturalIndexCreateIterator 1024 0.005 0.002 1.473 0.006
naturalIndexDescendingCreateIterator 1024 0.006 0.005 0.062 0.007
naturalIndexDescendingIteration 1024 0.006 0.004 0.268 0.008
naturalIndexIteration 1024 0.006 0.004 0.053 0.009
randomDeleteIndexed 1024 0.025 0.019 0.183 0.033
randomDeletesNoIndex 1024 0.015 0.012 0.037 0.017
randomUpdatesIndexed 1024 0.081 0.033 30.773 0.084
randomUpdatesNoIndex 1024 0.034 0.031 0.580 0.037
randomWritesIndexed 1024 0.117 0.035 52.383 0.120
randomWritesNoIndex 1024 0.039 0.034 1.556 0.044
refIndexCreateIterator 1024 0.005 0.004 0.019 0.006
refIndexDescendingCreateIterator 1024 0.003 0.002 0.028 0.004
refIndexDescendingIteration 1024 0.006 0.005 0.043 0.008
refIndexIteration 1024 0.007 0.005 0.068 0.009
sequentialDeleteIndexed 1024 0.021 0.017 0.096 0.026
sequentialDeleteNoIndex 1024 0.015 0.012 0.041 0.019
sequentialUpdatesIndexed 1024 0.044 0.037 0.747 0.051
sequentialUpdatesNoIndex 1024 0.039 0.032 0.771 0.046
sequentialWritesIndexed 1024 0.049 0.042 1.766 0.056
sequentialWritesNoIndex 1024 0.039 0.032 2.282 0.041
```
**After 7.5.3**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.342 0.212 0.683 0.683
dbCreation 4 77.603 3.403 301.357 301.357
naturalIndexCreateIterator 1024 0.005 0.002 1.507 0.007
naturalIndexDescendingCreateIterator 1024 0.005 0.005 0.076 0.007
naturalIndexDescendingIteration 1024 0.006 0.004 0.244 0.010
naturalIndexIteration 1024 0.006 0.004 0.065 0.009
randomDeleteIndexed 1024 0.028 0.019 1.385 0.038
randomDeletesNoIndex 1024 0.016 0.013 0.042 0.019
randomUpdatesIndexed 1024 0.083 0.033 32.094 0.086
randomUpdatesNoIndex 1024 0.038 0.034 0.469 0.042
randomWritesIndexed 1024 0.122 0.034 54.517 0.126
randomWritesNoIndex 1024 0.042 0.037 1.369 0.048
refIndexCreateIterator 1024 0.005 0.005 0.019 0.007
refIndexDescendingCreateIterator 1024 0.003 0.003 0.034 0.005
refIndexDescendingIteration 1024 0.006 0.005 0.044 0.008
refIndexIteration 1024 0.007 0.005 0.072 0.011
sequentialDeleteIndexed 1024 0.022 0.017 0.104 0.027
sequentialDeleteNoIndex 1024 0.015 0.012 0.038 0.019
sequentialUpdatesIndexed 1024 0.046 0.039 0.862 0.054
sequentialUpdatesNoIndex 1024 0.043 0.030 0.752 0.055
sequentialWritesIndexed 1024 0.051 0.039 1.949 0.060
sequentialWritesNoIndex 1024 0.038 0.032 2.858 0.042
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #37783: [SPARK-40321][BUILD] Upgrade rocksdbjni to 7.5.3
Posted by GitBox <gi...@apache.org>.
srowen commented on PR #37783:
URL: https://github.com/apache/spark/pull/37783#issuecomment-1236342063
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #37783: [SPARK-40321][BUILD] Upgrade rocksdbjni to 7.5.3
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #37783: [SPARK-40321][BUILD] Upgrade rocksdbjni to 7.5.3
URL: https://github.com/apache/spark/pull/37783
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #37783: [SPARK-40321][BUILD] Upgrade rocksdbjni to 7.5.3
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #37783:
URL: https://github.com/apache/spark/pull/37783#issuecomment-1236344680
thanks @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org