You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/27 21:54:03 UTC
[GitHub] [spark] dongjoon-hyun opened a new pull request #35034: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
dongjoon-hyun opened a new pull request #35034:
URL: https://github.com/apache/spark/pull/35034
### What changes were proposed in this pull request?
This PR aims to optimize RocksDB KVStore configuration.
### Why are the changes needed?
Currently, `RocksDB` KVStore performance is not optimized.
```
[INFO] Running org.apache.spark.util.kvstore.LevelDBBenchmark
count mean min max 95th
dbClose 4 0.802 0.777 0.824 0.824
dbCreation 4 71.029 2.528 278.049 278.049
naturalIndexCreateIterator 1024 0.006 0.002 1.559 0.007
naturalIndexDescendingCreateIterator 1024 0.003 0.003 0.043 0.004
naturalIndexDescendingIteration 1024 0.007 0.005 0.053 0.008
naturalIndexIteration 1024 0.007 0.005 0.080 0.009
randomDeleteIndexed 1024 0.020 0.017 0.148 0.024
randomDeletesNoIndex 1024 0.025 0.008 14.739 0.013
randomUpdatesIndexed 1024 0.079 0.033 23.527 0.075
randomUpdatesNoIndex 1024 0.078 0.072 0.584 0.082
randomWritesIndexed 1024 0.121 0.043 42.030 0.141
randomWritesNoIndex 1024 0.086 0.059 1.204 0.090
refIndexCreateIterator 1024 0.003 0.002 0.033 0.004
refIndexDescendingCreateIterator 1024 0.002 0.002 0.018 0.003
refIndexDescendingIteration 1024 0.007 0.006 0.045 0.008
refIndexIteration 1024 0.009 0.006 0.261 0.010
sequentialDeleteIndexed 1024 0.019 0.016 0.090 0.024
sequentialDeleteNoIndex 1024 0.013 0.010 0.083 0.014
sequentialUpdatesIndexed 1024 0.031 0.027 0.560 0.035
sequentialUpdatesNoIndex 1024 0.028 0.024 0.618 0.032
sequentialWritesIndexed 1024 0.037 0.031 1.301 0.041
sequentialWritesNoIndex 1024 0.038 0.026 1.699 0.065
```
**AS-IS**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.621 0.455 0.830 0.830
dbCreation 4 93.812 4.566 365.368 365.368
naturalIndexCreateIterator 1024 0.005 0.002 1.349 0.007
naturalIndexDescendingCreateIterator 1024 0.006 0.005 0.054 0.007
naturalIndexDescendingIteration 1024 0.007 0.005 0.034 0.009
naturalIndexIteration 1024 0.007 0.005 0.062 0.009
randomDeleteIndexed 1024 0.138 0.079 0.636 0.186
randomDeletesNoIndex 1024 0.104 0.058 0.223 0.143
randomUpdatesIndexed 1024 0.203 0.145 23.849 0.243
randomUpdatesNoIndex 1024 0.161 0.122 0.509 0.196
randomWritesIndexed 1024 0.239 0.149 42.780 0.279
randomWritesNoIndex 1024 0.143 0.080 1.593 0.221
refIndexCreateIterator 1024 0.003 0.003 0.032 0.005
refIndexDescendingCreateIterator 1024 0.003 0.003 0.028 0.005
refIndexDescendingIteration 1024 0.008 0.006 0.038 0.009
refIndexIteration 1024 0.009 0.007 0.265 0.011
sequentialDeleteIndexed 1024 0.122 0.106 0.251 0.150
sequentialDeleteNoIndex 1024 0.101 0.059 0.315 0.121
sequentialUpdatesIndexed 1024 0.147 0.109 0.812 0.197
sequentialUpdatesNoIndex 1024 0.143 0.112 0.896 0.184
sequentialWritesIndexed 1024 0.170 0.130 1.715 0.235
sequentialWritesNoIndex 1024 0.140 0.097 2.607 0.163
```
**AFTER THIS PR**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.673 0.472 0.921 0.921
dbCreation 4 97.226 4.324 376.853 376.853
naturalIndexCreateIterator 1024 0.005 0.002 1.364 0.006
naturalIndexDescendingCreateIterator 1024 0.005 0.005 0.054 0.006
naturalIndexDescendingIteration 1024 0.007 0.005 0.039 0.008
naturalIndexIteration 1024 0.008 0.006 0.061 0.009
randomDeleteIndexed 1024 0.031 0.025 0.224 0.039
randomDeletesNoIndex 1024 0.019 0.016 0.096 0.021
randomUpdatesIndexed 1024 0.091 0.045 24.200 0.092
randomUpdatesNoIndex 1024 0.051 0.031 0.509 0.102
randomWritesIndexed 1024 0.134 0.055 41.833 0.144
randomWritesNoIndex 1024 0.047 0.034 1.077 0.085
refIndexCreateIterator 1024 0.003 0.003 0.019 0.005
refIndexDescendingCreateIterator 1024 0.002 0.002 0.017 0.003
refIndexDescendingIteration 1024 0.007 0.006 0.035 0.008
refIndexIteration 1024 0.009 0.006 0.270 0.010
sequentialDeleteIndexed 1024 0.026 0.022 0.098 0.034
sequentialDeleteNoIndex 1024 0.022 0.016 0.165 0.040
sequentialUpdatesIndexed 1024 0.041 0.037 0.663 0.047
sequentialUpdatesNoIndex 1024 0.037 0.033 0.571 0.043
sequentialWritesIndexed 1024 0.046 0.040 1.249 0.052
sequentialWritesNoIndex 1024 0.044 0.034 1.694 0.098
```
### Does this PR introduce _any_ user-facing change?
No because `RocksDB` KVStore backend is a new feature.
### How was this patch tested?
Manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #35034: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #35034:
URL: https://github.com/apache/spark/pull/35034#discussion_r775665182
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java
##########
@@ -63,15 +59,38 @@
/** DB key where type aliases are stored. */
private static final byte[] TYPE_ALIASES_KEY = "__types__".getBytes(UTF_8);
+ /**
+ * Use full filter.
+ *
+ * https://github.com/facebook/rocksdb/wiki/RocksDB-Bloom-Filter#full-filters-new-format
+ */
+ private static final BloomFilter fullFilter =
+ new BloomFilter(10.0D /* BloomFilter.DEFAULT_BITS_PER_KEY */, false);
+
+ /** Disable compression in index data. */
private static final BlockBasedTableConfig tableFormatConfig = new BlockBasedTableConfig()
+ .setFilterPolicy(fullFilter)
+ .setEnableIndexCompression(false)
.setFormatVersion(5);
+ /**
+ * - Use ZSTD at the bottom most level to reduce the disk space
+ * - Use LZ4 at the other levels because it's better than Snappy in general.
+ *
+ * https://github.com/facebook/rocksdb/wiki/Compression#configuration
+ */
private static final Options dbOptions = new Options()
.setCreateIfMissing(true)
- .setTableFormatConfig(tableFormatConfig)
- .setStatistics(new Statistics());
Review comment:
Curious why this was removed ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #35034: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #35034:
URL: https://github.com/apache/spark/pull/35034
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #35034: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #35034:
URL: https://github.com/apache/spark/pull/35034#discussion_r775763139
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java
##########
@@ -63,15 +59,38 @@
/** DB key where type aliases are stored. */
private static final byte[] TYPE_ALIASES_KEY = "__types__".getBytes(UTF_8);
+ /**
+ * Use full filter.
+ *
+ * https://github.com/facebook/rocksdb/wiki/RocksDB-Bloom-Filter#full-filters-new-format
+ */
+ private static final BloomFilter fullFilter =
+ new BloomFilter(10.0D /* BloomFilter.DEFAULT_BITS_PER_KEY */, false);
+
+ /** Disable compression in index data. */
private static final BlockBasedTableConfig tableFormatConfig = new BlockBasedTableConfig()
+ .setFilterPolicy(fullFilter)
+ .setEnableIndexCompression(false)
.setFormatVersion(5);
+ /**
+ * - Use ZSTD at the bottom most level to reduce the disk space
+ * - Use LZ4 at the other levels because it's better than Snappy in general.
+ *
+ * https://github.com/facebook/rocksdb/wiki/Compression#configuration
+ */
private static final Options dbOptions = new Options()
.setCreateIfMissing(true)
- .setTableFormatConfig(tableFormatConfig)
- .setStatistics(new Statistics());
Review comment:
Sounds good, was curious, that is all :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35034: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35034:
URL: https://github.com/apache/spark/pull/35034#issuecomment-1001823016
The GitHub Action UT failures are irrelevant to this PR.
```
[info] *** 1 TEST FAILED ***
[error] Failed: Total 3179, Failed 1, Errors 0, Passed 3178, Ignored 8, Canceled 1
[error] Failed tests:
[error] org.apache.spark.SparkContextSuite
```
I'll merge this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #35034: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #35034:
URL: https://github.com/apache/spark/pull/35034#discussion_r775661665
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java
##########
@@ -63,15 +59,38 @@
/** DB key where type aliases are stored. */
private static final byte[] TYPE_ALIASES_KEY = "__types__".getBytes(UTF_8);
+ /**
+ * Use full filter.
+ *
+ * https://github.com/facebook/rocksdb/wiki/RocksDB-Bloom-Filter#full-filters-new-format
+ */
+ private static final BloomFilter fullFilter =
+ new BloomFilter(10.0D /* BloomFilter.DEFAULT_BITS_PER_KEY */, false);
+
+ /** Disable compression in index data. */
private static final BlockBasedTableConfig tableFormatConfig = new BlockBasedTableConfig()
+ .setFilterPolicy(fullFilter)
+ .setEnableIndexCompression(false)
Review comment:
Does `index_block_restart_interval` (8 or 16) help too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #35034: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #35034:
URL: https://github.com/apache/spark/pull/35034#issuecomment-1001911610
Late LGTM.
+CC @thejdeep
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35034: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35034:
URL: https://github.com/apache/spark/pull/35034#discussion_r775666511
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java
##########
@@ -63,15 +59,38 @@
/** DB key where type aliases are stored. */
private static final byte[] TYPE_ALIASES_KEY = "__types__".getBytes(UTF_8);
+ /**
+ * Use full filter.
+ *
+ * https://github.com/facebook/rocksdb/wiki/RocksDB-Bloom-Filter#full-filters-new-format
+ */
+ private static final BloomFilter fullFilter =
+ new BloomFilter(10.0D /* BloomFilter.DEFAULT_BITS_PER_KEY */, false);
+
+ /** Disable compression in index data. */
private static final BlockBasedTableConfig tableFormatConfig = new BlockBasedTableConfig()
+ .setFilterPolicy(fullFilter)
+ .setEnableIndexCompression(false)
.setFormatVersion(5);
+ /**
+ * - Use ZSTD at the bottom most level to reduce the disk space
+ * - Use LZ4 at the other levels because it's better than Snappy in general.
+ *
+ * https://github.com/facebook/rocksdb/wiki/Compression#configuration
+ */
private static final Options dbOptions = new Options()
.setCreateIfMissing(true)
- .setTableFormatConfig(tableFormatConfig)
- .setStatistics(new Statistics());
Review comment:
Thank you for review, @mridulm . Apache Spark doesn't use `Statistics` and it's removed due to the non-negligible overhead.
- https://github.com/facebook/rocksdb/issues/2489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35034: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35034:
URL: https://github.com/apache/spark/pull/35034#issuecomment-1001823257
Thank you, @viirya and @mridulm .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35034: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35034:
URL: https://github.com/apache/spark/pull/35034#issuecomment-1001919521
Got you. Thanks, @mridulm .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35034: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35034:
URL: https://github.com/apache/spark/pull/35034#discussion_r775665367
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java
##########
@@ -63,15 +59,38 @@
/** DB key where type aliases are stored. */
private static final byte[] TYPE_ALIASES_KEY = "__types__".getBytes(UTF_8);
+ /**
+ * Use full filter.
+ *
+ * https://github.com/facebook/rocksdb/wiki/RocksDB-Bloom-Filter#full-filters-new-format
+ */
+ private static final BloomFilter fullFilter =
+ new BloomFilter(10.0D /* BloomFilter.DEFAULT_BITS_PER_KEY */, false);
+
+ /** Disable compression in index data. */
private static final BlockBasedTableConfig tableFormatConfig = new BlockBasedTableConfig()
+ .setFilterPolicy(fullFilter)
+ .setEnableIndexCompression(false)
Review comment:
Thank you. According to the test result, `8` could be used. Let me add it.
- `.setIndexBlockRestartInterval(8)`
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.669 0.633 0.700 0.700
dbCreation 4 94.437 4.312 367.823 367.823
naturalIndexCreateIterator 1024 0.005 0.002 1.371 0.006
naturalIndexDescendingCreateIterator 1024 0.005 0.004 0.052 0.006
naturalIndexDescendingIteration 1024 0.007 0.005 0.060 0.008
naturalIndexIteration 1024 0.007 0.005 0.061 0.009
randomDeleteIndexed 1024 0.032 0.026 0.411 0.040
randomDeletesNoIndex 1024 0.021 0.017 0.138 0.022
randomUpdatesIndexed 1024 0.087 0.042 23.607 0.089
randomUpdatesNoIndex 1024 0.035 0.031 0.421 0.041
randomWritesIndexed 1024 0.123 0.054 41.442 0.128
randomWritesNoIndex 1024 0.040 0.033 1.473 0.045
refIndexCreateIterator 1024 0.003 0.003 0.021 0.005
refIndexDescendingCreateIterator 1024 0.003 0.002 0.036 0.004
refIndexDescendingIteration 1024 0.008 0.006 0.053 0.009
refIndexIteration 1024 0.009 0.006 0.288 0.011
sequentialDeleteIndexed 1024 0.026 0.022 0.116 0.031
sequentialDeleteNoIndex 1024 0.022 0.017 0.146 0.040
sequentialUpdatesIndexed 1024 0.041 0.036 0.720 0.048
sequentialUpdatesNoIndex 1024 0.039 0.034 0.590 0.046
sequentialWritesIndexed 1024 0.051 0.041 1.427 0.088
sequentialWritesNoIndex 1024 0.042 0.034 2.019 0.045
```
- `.setIndexBlockRestartInterval(16)`
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max 95th
dbClose 4 0.709 0.546 0.898 0.898
dbCreation 4 93.111 4.301 358.929 358.929
naturalIndexCreateIterator 1024 0.005 0.002 1.389 0.006
naturalIndexDescendingCreateIterator 1024 0.006 0.005 0.063 0.007
naturalIndexDescendingIteration 1024 0.007 0.006 0.055 0.008
naturalIndexIteration 1024 0.007 0.005 0.061 0.009
randomDeleteIndexed 1024 0.033 0.024 0.214 0.049
randomDeletesNoIndex 1024 0.018 0.015 0.082 0.019
randomUpdatesIndexed 1024 0.089 0.045 23.830 0.086
randomUpdatesNoIndex 1024 0.076 0.031 0.496 0.085
randomWritesIndexed 1024 0.147 0.053 41.833 0.165
randomWritesNoIndex 1024 0.039 0.033 1.265 0.045
refIndexCreateIterator 1024 0.003 0.002 0.024 0.004
refIndexDescendingCreateIterator 1024 0.002 0.002 0.026 0.003
refIndexDescendingIteration 1024 0.008 0.006 0.050 0.009
refIndexIteration 1024 0.009 0.006 0.278 0.012
sequentialDeleteIndexed 1024 0.027 0.022 0.107 0.033
sequentialDeleteNoIndex 1024 0.019 0.016 0.068 0.020
sequentialUpdatesIndexed 1024 0.042 0.038 0.569 0.050
sequentialUpdatesNoIndex 1024 0.070 0.066 0.557 0.077
sequentialWritesIndexed 1024 0.047 0.041 1.381 0.053
sequentialWritesNoIndex 1024 0.074 0.035 1.690 0.120
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35034: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35034:
URL: https://github.com/apache/spark/pull/35034#discussion_r775666698
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java
##########
@@ -63,15 +59,38 @@
/** DB key where type aliases are stored. */
private static final byte[] TYPE_ALIASES_KEY = "__types__".getBytes(UTF_8);
+ /**
+ * Use full filter.
+ *
+ * https://github.com/facebook/rocksdb/wiki/RocksDB-Bloom-Filter#full-filters-new-format
+ */
+ private static final BloomFilter fullFilter =
+ new BloomFilter(10.0D /* BloomFilter.DEFAULT_BITS_PER_KEY */, false);
+
+ /** Disable compression in index data. */
private static final BlockBasedTableConfig tableFormatConfig = new BlockBasedTableConfig()
+ .setFilterPolicy(fullFilter)
+ .setEnableIndexCompression(false)
.setFormatVersion(5);
+ /**
+ * - Use ZSTD at the bottom most level to reduce the disk space
+ * - Use LZ4 at the other levels because it's better than Snappy in general.
+ *
+ * https://github.com/facebook/rocksdb/wiki/Compression#configuration
+ */
private static final Options dbOptions = new Options()
.setCreateIfMissing(true)
- .setTableFormatConfig(tableFormatConfig)
- .setStatistics(new Statistics());
Review comment:
Our benchmark suite is too small to show the overhead, but there was some websites about mentioning 5~10% overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35034: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35034:
URL: https://github.com/apache/spark/pull/35034#issuecomment-1001790634
Do you have any other comment, @mridulm ? 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35034: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35034:
URL: https://github.com/apache/spark/pull/35034#issuecomment-1001780168
Could you review this PR, @viirya ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #35034: [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #35034:
URL: https://github.com/apache/spark/pull/35034#discussion_r775661665
##########
File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java
##########
@@ -63,15 +59,38 @@
/** DB key where type aliases are stored. */
private static final byte[] TYPE_ALIASES_KEY = "__types__".getBytes(UTF_8);
+ /**
+ * Use full filter.
+ *
+ * https://github.com/facebook/rocksdb/wiki/RocksDB-Bloom-Filter#full-filters-new-format
+ */
+ private static final BloomFilter fullFilter =
+ new BloomFilter(10.0D /* BloomFilter.DEFAULT_BITS_PER_KEY */, false);
+
+ /** Disable compression in index data. */
private static final BlockBasedTableConfig tableFormatConfig = new BlockBasedTableConfig()
+ .setFilterPolicy(fullFilter)
+ .setEnableIndexCompression(false)
Review comment:
Does `index_block_restart_interval` help too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org