You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Damian Guy (JIRA)" <ji...@apache.org> on 2017/02/09 14:42:41 UTC
[jira] [Updated] (KAFKA-4484) Set more conservative default values
on RocksDB for memory usage
[ https://issues.apache.org/jira/browse/KAFKA-4484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Damian Guy updated KAFKA-4484:
------------------------------
Status: Patch Available (was: In Progress)
> Set more conservative default values on RocksDB for memory usage
> ----------------------------------------------------------------
>
> Key: KAFKA-4484
> URL: https://issues.apache.org/jira/browse/KAFKA-4484
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Guozhang Wang
> Assignee: Damian Guy
> Labels: performance
>
> Quoting from email thread:
> {code}
> The block cache size defaults to a whopping 100Mb per store, and that gets
> expensive
> fast. I reduced it to a few megabytes. My data size is so big that I doubt
> it is very effective
> anyway. Now it seems more stable.
> I'd say that a smaller default makes sense, especially because the failure
> case is
> so opaque (running all tests just fine but with a serious dataset it dies
> slowly)
> {code}
> {code}
> Before we have the a single-knob memory management feature, I'd like to propose reducing the Streams' default config values for RocksDB caching and memory block size. For example, I remember Henry has done some fine tuning on the RocksDB config for his use case:
> https://github.com/HenryCaiHaiying/kafka/commit/b297f7c585f5a883ee068277e5f0f1224c347bd4
> https://github.com/HenryCaiHaiying/kafka/commit/eed1726d16e528d813755a6e66b49d0bf14e8803
> https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
> We could check if some of those changes are appropriate in general and if yes change the default settings accordingly.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)