You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/02/03 23:23:21 UTC
[GitHub] [kafka] mjsax opened a new pull request #10046: MINOR: Extends RocksDB docs
mjsax opened a new pull request #10046:
URL: https://github.com/apache/kafka/pull/10046
We recommend users to switch to jemalloc for RocksDB
Call for review @ableegoldman @JimGalasyn @rodesai
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rodesai commented on a change in pull request #10046: MINOR: Extends RocksDB docs
Posted by GitBox <gi...@apache.org>.
rodesai commented on a change in pull request #10046:
URL: https://github.com/apache/kafka/pull/10046#discussion_r570627385
##########
File path: docs/streams/developer-guide/memory-mgmt.html
##########
@@ -168,7 +168,15 @@
<h2><a class="toc-backref" href="#id3">RocksDB</a><a class="headerlink" href="#rocksdb" title="Permalink to this headline"></a></h2>
<p> Each instance of RocksDB allocates off-heap memory for a block cache, index and filter blocks, and memtable (write buffer). Critical configs (for RocksDB version 4.1.0) include
<code class="docutils literal"><span class="pre">block_cache_size</span></code>, <code class="docutils literal"><span class="pre">write_buffer_size</span></code> and <code class="docutils literal"><span class="pre">max_write_buffer_number</span></code>. These can be specified through the
- <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</li>
+ <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</p>
+ <p>Also, we recommend changing RocksDB's default memory allocator, because the default allocator may lead to increased memory consumption.
+ To change the memory allocator to <code>jemalloc</code>, you need to set the an environment variable before you start your Kafka Streams application:</p>
+ <pre>
+# example: install jemalloc (on Debian)
+$ apt install -y libjemalloc-dev
+# set LD_PRELOAD before you start your Kafka Streams application
+$ LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so”
Review comment:
just doing `LD_PRELOAD=` will only set the variable for that one command, which isn't doing anything. This should either be:
```
$ export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so"
$ <application>
```
OR
```
$ LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so" <application>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #10046: MINOR: Extends RocksDB docs
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #10046:
URL: https://github.com/apache/kafka/pull/10046#discussion_r570683354
##########
File path: docs/streams/developer-guide/memory-mgmt.html
##########
@@ -168,7 +168,15 @@
<h2><a class="toc-backref" href="#id3">RocksDB</a><a class="headerlink" href="#rocksdb" title="Permalink to this headline"></a></h2>
<p> Each instance of RocksDB allocates off-heap memory for a block cache, index and filter blocks, and memtable (write buffer). Critical configs (for RocksDB version 4.1.0) include
<code class="docutils literal"><span class="pre">block_cache_size</span></code>, <code class="docutils literal"><span class="pre">write_buffer_size</span></code> and <code class="docutils literal"><span class="pre">max_write_buffer_number</span></code>. These can be specified through the
- <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</li>
+ <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</p>
+ <p>Also, we recommend changing RocksDB's default memory allocator, because the default allocator may lead to increased memory consumption.
+ To change the memory allocator to <code>jemalloc</code>, you need to set the an environment variable before you start your Kafka Streams application:</p>
+ <pre>
+# example: install jemalloc (on Debian)
+$ apt install -y libjemalloc-dev
+# set LD_PRELOAD before you start your Kafka Streams application
+$ LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so”
Review comment:
Good catch!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10046: MINOR: Extends RocksDB docs
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #10046:
URL: https://github.com/apache/kafka/pull/10046#discussion_r569830121
##########
File path: docs/streams/developer-guide/memory-mgmt.html
##########
@@ -168,7 +168,15 @@
<h2><a class="toc-backref" href="#id3">RocksDB</a><a class="headerlink" href="#rocksdb" title="Permalink to this headline"></a></h2>
<p> Each instance of RocksDB allocates off-heap memory for a block cache, index and filter blocks, and memtable (write buffer). Critical configs (for RocksDB version 4.1.0) include
<code class="docutils literal"><span class="pre">block_cache_size</span></code>, <code class="docutils literal"><span class="pre">write_buffer_size</span></code> and <code class="docutils literal"><span class="pre">max_write_buffer_number</span></code>. These can be specified through the
- <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</li>
+ <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</p>
+ <p>We further recommend, to change RocksDB's default memory allocator, because the default allocator may lead to increased memory consumption.
+ To change the memory allocator to <code>jemalloc</code>, you need to set the an environment variable before you start your Kafka Streams application:</p>
Review comment:
```suggestion
To change the memory allocator to <code>jemalloc</code>, you need to set the environment variable before you start your Kafka Streams application:</p>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #10046: MINOR: Extends RocksDB docs
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #10046:
URL: https://github.com/apache/kafka/pull/10046#discussion_r569832333
##########
File path: docs/streams/developer-guide/memory-mgmt.html
##########
@@ -168,7 +168,15 @@
<h2><a class="toc-backref" href="#id3">RocksDB</a><a class="headerlink" href="#rocksdb" title="Permalink to this headline"></a></h2>
<p> Each instance of RocksDB allocates off-heap memory for a block cache, index and filter blocks, and memtable (write buffer). Critical configs (for RocksDB version 4.1.0) include
<code class="docutils literal"><span class="pre">block_cache_size</span></code>, <code class="docutils literal"><span class="pre">write_buffer_size</span></code> and <code class="docutils literal"><span class="pre">max_write_buffer_number</span></code>. These can be specified through the
- <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</li>
+ <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</p>
+ <p>We further recommend, to change RocksDB's default memory allocator, because the default allocator may lead to increased memory consumption.
+ To change the memory allocator to <code>jemalloc</code>, you need to set the an environment variable before you start your Kafka Streams application:</p>
Review comment:
I guess it should be `set an environment variable` as we don't include the name?
Or should we change it to `set the environment variable LD_PRELOAD`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10046: MINOR: Extends RocksDB docs
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #10046:
URL: https://github.com/apache/kafka/pull/10046#discussion_r569893054
##########
File path: docs/streams/developer-guide/memory-mgmt.html
##########
@@ -168,7 +168,15 @@
<h2><a class="toc-backref" href="#id3">RocksDB</a><a class="headerlink" href="#rocksdb" title="Permalink to this headline"></a></h2>
<p> Each instance of RocksDB allocates off-heap memory for a block cache, index and filter blocks, and memtable (write buffer). Critical configs (for RocksDB version 4.1.0) include
<code class="docutils literal"><span class="pre">block_cache_size</span></code>, <code class="docutils literal"><span class="pre">write_buffer_size</span></code> and <code class="docutils literal"><span class="pre">max_write_buffer_number</span></code>. These can be specified through the
- <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</li>
+ <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</p>
+ <p>We further recommend, to change RocksDB's default memory allocator, because the default allocator may lead to increased memory consumption.
+ To change the memory allocator to <code>jemalloc</code>, you need to set the an environment variable before you start your Kafka Streams application:</p>
Review comment:
Mm good point. I think `set the environment variable LD_PRELOAD` is best
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] JimGalasyn commented on a change in pull request #10046: MINOR: Extends RocksDB docs
Posted by GitBox <gi...@apache.org>.
JimGalasyn commented on a change in pull request #10046:
URL: https://github.com/apache/kafka/pull/10046#discussion_r569825889
##########
File path: docs/streams/developer-guide/memory-mgmt.html
##########
@@ -168,7 +168,15 @@
<h2><a class="toc-backref" href="#id3">RocksDB</a><a class="headerlink" href="#rocksdb" title="Permalink to this headline"></a></h2>
<p> Each instance of RocksDB allocates off-heap memory for a block cache, index and filter blocks, and memtable (write buffer). Critical configs (for RocksDB version 4.1.0) include
<code class="docutils literal"><span class="pre">block_cache_size</span></code>, <code class="docutils literal"><span class="pre">write_buffer_size</span></code> and <code class="docutils literal"><span class="pre">max_write_buffer_number</span></code>. These can be specified through the
- <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</li>
+ <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</p>
+ <p>We further recommend, to change RocksDB's default memory allocator, because the default allocator may lead to increased memory consumption.
Review comment:
```suggestion
<p>Also, we recommend changing RocksDB's default memory allocator, because the default allocator may lead to increased memory consumption.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #10046: MINOR: Extends RocksDB docs
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #10046:
URL: https://github.com/apache/kafka/pull/10046#discussion_r569822623
##########
File path: docs/streams/developer-guide/memory-mgmt.html
##########
@@ -201,7 +209,7 @@ <h2><a class="toc-backref" href="#id3">RocksDB</a><a class="headerlink" href="#r
<span class="c1">// Cache and WriteBufferManager should not be closed here, as the same objects are shared by every store instance.</span>
<span class="o">}</span>
<span class="o">}</span>
- </div>
Review comment:
Just some cleanup... (here and below). My editor flagged those...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax merged pull request #10046: MINOR: Extends RocksDB docs
Posted by GitBox <gi...@apache.org>.
mjsax merged pull request #10046:
URL: https://github.com/apache/kafka/pull/10046
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #10046: MINOR: Extends RocksDB docs
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #10046:
URL: https://github.com/apache/kafka/pull/10046#discussion_r570683354
##########
File path: docs/streams/developer-guide/memory-mgmt.html
##########
@@ -168,7 +168,15 @@
<h2><a class="toc-backref" href="#id3">RocksDB</a><a class="headerlink" href="#rocksdb" title="Permalink to this headline"></a></h2>
<p> Each instance of RocksDB allocates off-heap memory for a block cache, index and filter blocks, and memtable (write buffer). Critical configs (for RocksDB version 4.1.0) include
<code class="docutils literal"><span class="pre">block_cache_size</span></code>, <code class="docutils literal"><span class="pre">write_buffer_size</span></code> and <code class="docutils literal"><span class="pre">max_write_buffer_number</span></code>. These can be specified through the
- <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</li>
+ <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</p>
+ <p>Also, we recommend changing RocksDB's default memory allocator, because the default allocator may lead to increased memory consumption.
+ To change the memory allocator to <code>jemalloc</code>, you need to set the an environment variable before you start your Kafka Streams application:</p>
+ <pre>
+# example: install jemalloc (on Debian)
+$ apt install -y libjemalloc-dev
+# set LD_PRELOAD before you start your Kafka Streams application
+$ LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so”
Review comment:
Good catch!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rodesai commented on a change in pull request #10046: MINOR: Extends RocksDB docs
Posted by GitBox <gi...@apache.org>.
rodesai commented on a change in pull request #10046:
URL: https://github.com/apache/kafka/pull/10046#discussion_r570627385
##########
File path: docs/streams/developer-guide/memory-mgmt.html
##########
@@ -168,7 +168,15 @@
<h2><a class="toc-backref" href="#id3">RocksDB</a><a class="headerlink" href="#rocksdb" title="Permalink to this headline"></a></h2>
<p> Each instance of RocksDB allocates off-heap memory for a block cache, index and filter blocks, and memtable (write buffer). Critical configs (for RocksDB version 4.1.0) include
<code class="docutils literal"><span class="pre">block_cache_size</span></code>, <code class="docutils literal"><span class="pre">write_buffer_size</span></code> and <code class="docutils literal"><span class="pre">max_write_buffer_number</span></code>. These can be specified through the
- <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</li>
+ <code class="docutils literal"><span class="pre">rocksdb.config.setter</span></code> configuration.</p>
+ <p>Also, we recommend changing RocksDB's default memory allocator, because the default allocator may lead to increased memory consumption.
+ To change the memory allocator to <code>jemalloc</code>, you need to set the an environment variable before you start your Kafka Streams application:</p>
+ <pre>
+# example: install jemalloc (on Debian)
+$ apt install -y libjemalloc-dev
+# set LD_PRELOAD before you start your Kafka Streams application
+$ LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so”
Review comment:
just doing `LD_PRELOAD=` will only set the variable for that one command, which isn't doing anything. This should either be:
```
$ export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so"
$ <application>
```
OR
```
$ LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so" <application>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax merged pull request #10046: MINOR: Extends RocksDB docs
Posted by GitBox <gi...@apache.org>.
mjsax merged pull request #10046:
URL: https://github.com/apache/kafka/pull/10046
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org