You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "szcom (via GitHub)" <gi...@apache.org> on 2024/12/20 08:39:34 UTC
[PR] Add a recommendation on choosing the size of the block cache for RocksDB [kafka]
szcom opened a new pull request, #18280:
URL: https://github.com/apache/kafka/pull/18280
Improving documentation by warning a user to be more careful when choosing size of RocksDB cache.
Correcting false claims on ability to limit _total_ memory usage of the library (rocksdb).
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a recommendation on choosing the size of the block cache for RocksDB [kafka]
Posted by "cadonna (via GitHub)" <gi...@apache.org>.
cadonna commented on code in PR #18280:
URL: https://github.com/apache/kafka/pull/18280#discussion_r1893902897
##########
docs/streams/developer-guide/memory-mgmt.html:
##########
@@ -176,7 +177,7 @@
BlockBasedTableConfig tableConfig = (BlockBasedTableConfig) options.tableFormatConfig();
- // These three options in combination will limit the memory used by RocksDB to the size passed to the block cache (TOTAL_OFF_HEAP_MEMORY)
+ // These three options in combination will limit the memory used by RocksDB for cache, indexes, and write buffers to the size passed to (TOTAL_OFF_HEAP_MEMORY)
Review Comment:
```suggestion
// These three options in combination will limit the memory used by RocksDB for cache, indexes, filters, and write buffers to TOTAL_OFF_HEAP_MEMORY
```
##########
docs/streams/developer-guide/memory-mgmt.html:
##########
@@ -165,7 +165,8 @@
$ apt install -y libjemalloc-dev
# set LD_PRELOAD before you start your Kafka Streams application
$ export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so"</code></pre>
- <p> As of 2.3.0 the memory usage across all instances can be bounded, limiting the total off-heap memory of your Kafka Streams application. To do so you must configure RocksDB to cache the index and filter blocks in the block cache, limit the memtable memory through a shared <a class="reference external" href="https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager">WriteBufferManager</a> and count its memory against the block cache, and then pass the same Cache object to each instance. See <a class="reference external" href="https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB">RocksDB Memory Usage</a> for details. An example RocksDBConfigSetter implementing this is shown below:</p>
+ <p> As of 2.3.0 the memory usage across all instances can be bounded, limiting the off-heap memory of your Kafka Streams application. To do so you must configure RocksDB to cache the index and filter blocks in the block cache, limit the memtable memory through a shared <a class="reference external" href="https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager">WriteBufferManager</a> and count its memory against the block cache, and then pass the same Cache object to each instance.
+ However, don't reserve more than 40-50% of native memory budget for the cache alone to begin with, as RocksDB needs memory for internal housekeeping, test your workload to find the optimal size. Smaller cache size does not necessary increase IO due to page cache kept by OS. See <a class="reference external" href="https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB">RocksDB Memory Usage</a> for details. An example RocksDBConfigSetter implementing this is shown below:</p>
Review Comment:
I would not be such imperative. What about something like:
```
Find a good limit by testing with your workload. For example, start limiting the off-heap memory used by RocksDB to 40-50% of your memory budget after detracting the memory reserved for the JVM. Note that a smaller cache size does not necessarily increase I/O since the OS maintains a page cache.
```
BTW, I checked our general-purpose Kafka Streams deployments and we use around 70% of the memory after detracting the memory reserved for the JVM. As far as I know, we rarely run into OOMs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a recommendation on choosing the size of the block cache for RocksDB [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #18280: Add a recommendation on choosing the size of the block cache for RocksDB
URL: https://github.com/apache/kafka/pull/18280
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a recommendation on choosing the size of the block cache for RocksDB [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18280:
URL: https://github.com/apache/kafka/pull/18280#issuecomment-2861648709
This PR has been closed since it has not had any activity in 120 days. If you feel like this
was a mistake, or you would like to continue working on it, please feel free to re-open the
PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a recommendation on choosing the size of the block cache for RocksDB [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18280:
URL: https://github.com/apache/kafka/pull/18280#issuecomment-2785142809
This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.
<p>
If you are having difficulty finding a reviewer, please reach out on the
[mailing list](https://kafka.apache.org/contact).
<p>
If this PR is no longer valid or desired, please feel free to close it. If no activity
occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a recommendation on choosing the size of the block cache for RocksDB [kafka]
Posted by "szcom (via GitHub)" <gi...@apache.org>.
szcom commented on PR #18280:
URL: https://github.com/apache/kafka/pull/18280#issuecomment-2575072248
@cadonna thanks for the feedback, comments are fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a recommendation on choosing the size of the block cache for RocksDB [kafka]
Posted by "szcom (via GitHub)" <gi...@apache.org>.
szcom commented on code in PR #18280:
URL: https://github.com/apache/kafka/pull/18280#discussion_r1893913279
##########
docs/streams/developer-guide/memory-mgmt.html:
##########
@@ -165,7 +165,8 @@
$ apt install -y libjemalloc-dev
# set LD_PRELOAD before you start your Kafka Streams application
$ export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so"</code></pre>
- <p> As of 2.3.0 the memory usage across all instances can be bounded, limiting the total off-heap memory of your Kafka Streams application. To do so you must configure RocksDB to cache the index and filter blocks in the block cache, limit the memtable memory through a shared <a class="reference external" href="https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager">WriteBufferManager</a> and count its memory against the block cache, and then pass the same Cache object to each instance. See <a class="reference external" href="https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB">RocksDB Memory Usage</a> for details. An example RocksDBConfigSetter implementing this is shown below:</p>
+ <p> As of 2.3.0 the memory usage across all instances can be bounded, limiting the off-heap memory of your Kafka Streams application. To do so you must configure RocksDB to cache the index and filter blocks in the block cache, limit the memtable memory through a shared <a class="reference external" href="https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager">WriteBufferManager</a> and count its memory against the block cache, and then pass the same Cache object to each instance.
+ However, don't reserve more than 40-50% of native memory budget for the cache alone to begin with, as RocksDB needs memory for internal housekeeping, test your workload to find the optimal size. Smaller cache size does not necessary increase IO due to page cache kept by OS. See <a class="reference external" href="https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB">RocksDB Memory Usage</a> for details. An example RocksDBConfigSetter implementing this is shown below:</p>
Review Comment:
updated. 70% and no OOMs - probably cache hit rate is good, > 50% ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a recommendation on choosing the size of the block cache for RocksDB [kafka]
Posted by "szcom (via GitHub)" <gi...@apache.org>.
szcom commented on code in PR #18280:
URL: https://github.com/apache/kafka/pull/18280#discussion_r1893913467
##########
docs/streams/developer-guide/memory-mgmt.html:
##########
@@ -176,7 +177,7 @@
BlockBasedTableConfig tableConfig = (BlockBasedTableConfig) options.tableFormatConfig();
- // These three options in combination will limit the memory used by RocksDB to the size passed to the block cache (TOTAL_OFF_HEAP_MEMORY)
+ // These three options in combination will limit the memory used by RocksDB for cache, indexes, and write buffers to the size passed to (TOTAL_OFF_HEAP_MEMORY)
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org