You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by mm...@apache.org on 2019/05/19 00:49:41 UTC
[pulsar] 26/26: By default,
auto configure the size of Bookie read/write cache (#4297)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch branch-2.3
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit ea70a2c8bdba3b5f1b046334f8394b7a06e7efdb
Author: Matteo Merli <mm...@apache.org>
AuthorDate: Fri May 17 17:16:50 2019 -0700
By default, auto configure the size of Bookie read/write cache (#4297)
* By default, auto configure the size of Bookie read/write cache
* Updated ref docs
---
conf/bookkeeper.conf | 13 ++++++++-----
conf/standalone.conf | 13 ++++++++-----
site2/docs/reference-configuration.md | 6 +++---
3 files changed, 19 insertions(+), 13 deletions(-)
diff --git a/conf/bookkeeper.conf b/conf/bookkeeper.conf
index 9df49d2..da68a3d 100644
--- a/conf/bookkeeper.conf
+++ b/conf/bookkeeper.conf
@@ -628,12 +628,15 @@ httpServerClass=org.apache.bookkeeper.http.vertx.VertxHttpServer
# Size of Write Cache. Memory is allocated from JVM direct memory.
# Write cache is used to buffer entries before flushing into the entry log
-# For good performance, it should be big enough to hold a sub
-dbStorage_writeCacheMaxSizeMb=512
+# For good performance, it should be big enough to hold a substantial amount
+# of entries in the flush interval
+# By default it will be allocated to 1/4th of the available direct memory
+dbStorage_writeCacheMaxSizeMb=
# Size of Read cache. Memory is allocated from JVM direct memory.
# This read cache is pre-filled doing read-ahead whenever a cache miss happens
-dbStorage_readAheadCacheMaxSizeMb=256
+# By default it will be allocated to 1/4th of the available direct memory
+dbStorage_readAheadCacheMaxSizeMb=
# How many entries to pre-fill in cache after a read cache miss
dbStorage_readAheadCacheBatchSize=1000
@@ -645,8 +648,8 @@ dbStorage_readAheadCacheBatchSize=1000
# Size of RocksDB block-cache. For best performance, this cache
# should be big enough to hold a significant portion of the index
# database which can reach ~2GB in some cases
-# Default is 256 MBytes
-dbStorage_rocksDB_blockCacheSize=268435456
+# Default is to use 10% of the direct memory size
+dbStorage_rocksDB_blockCacheSize=
# Other RocksDB specific tunables
dbStorage_rocksDB_writeBufferSizeMB=64
diff --git a/conf/standalone.conf b/conf/standalone.conf
index 22bbc70..edadc2a 100644
--- a/conf/standalone.conf
+++ b/conf/standalone.conf
@@ -461,12 +461,15 @@ ledgerStorageClass=org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage
# Size of Write Cache. Memory is allocated from JVM direct memory.
# Write cache is used to buffer entries before flushing into the entry log
-# For good performance, it should be big enough to hold a sub
-dbStorage_writeCacheMaxSizeMb=256
+# For good performance, it should be big enough to hold a substantial amount
+# of entries in the flush interval
+# By default it will be allocated to 1/4th of the available direct memory
+dbStorage_writeCacheMaxSizeMb=
# Size of Read cache. Memory is allocated from JVM direct memory.
# This read cache is pre-filled doing read-ahead whenever a cache miss happens
-dbStorage_readAheadCacheMaxSizeMb=64
+# By default it will be allocated to 1/4th of the available direct memory
+dbStorage_readAheadCacheMaxSizeMb=
# How many entries to pre-fill in cache after a read cache miss
dbStorage_readAheadCacheBatchSize=1000
@@ -480,8 +483,8 @@ flushInterval=60000
# Size of RocksDB block-cache. For best performance, this cache
# should be big enough to hold a significant portion of the index
# database which can reach ~2GB in some cases
-# Default is 16 MBytes
-dbStorage_rocksDB_blockCacheSize=16777216
+# Default is to use 10% of the direct memory size
+dbStorage_rocksDB_blockCacheSize=
# Other RocksDB specific tunables
dbStorage_rocksDB_writeBufferSizeMB=4
diff --git a/site2/docs/reference-configuration.md b/site2/docs/reference-configuration.md
index 2865da3..ee94920 100644
--- a/site2/docs/reference-configuration.md
+++ b/site2/docs/reference-configuration.md
@@ -84,10 +84,10 @@ BookKeeper is a replicated log storage system that Pulsar uses for durable stora
|useHostNameAsBookieID|Whether the bookie should use its hostname to register with the coordination service (e.g.: zookeeper service). When false, bookie will use its ipaddress for the registration.|false|
|statsProviderClass||org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider|
|prometheusStatsHttpPort||8000|
-|dbStorage_writeCacheMaxSizeMb|Size of Write Cache. Memory is allocated from JVM direct memory. Write cache is used to buffer entries before flushing into the entry log For good performance, it should be big enough to hold a sub|512|
-|dbStorage_readAheadCacheMaxSizeMb|Size of Read cache. Memory is allocated from JVM direct memory. This read cache is pre-filled doing read-ahead whenever a cache miss happens|256|
+|dbStorage_writeCacheMaxSizeMb|Size of Write Cache. Memory is allocated from JVM direct memory. Write cache is used to buffer entries before flushing into the entry log For good performance, it should be big enough to hold a sub|25% of direct memory|
+|dbStorage_readAheadCacheMaxSizeMb|Size of Read cache. Memory is allocated from JVM direct memory. This read cache is pre-filled doing read-ahead whenever a cache miss happens|25% of direct memory|
|dbStorage_readAheadCacheBatchSize|How many entries to pre-fill in cache after a read cache miss|1000|
-|dbStorage_rocksDB_blockCacheSize|Size of RocksDB block-cache. For best performance, this cache should be big enough to hold a significant portion of the index database which can reach ~2GB in some cases|268435456|
+|dbStorage_rocksDB_blockCacheSize|Size of RocksDB block-cache. For best performance, this cache should be big enough to hold a significant portion of the index database which can reach ~2GB in some cases|10% of direct memory|
|dbStorage_rocksDB_writeBufferSizeMB||64|
|dbStorage_rocksDB_sstSizeInMB||64|
|dbStorage_rocksDB_blockSize||65536|