[jira] [Commented] (KAFKA-12559) Add a top-level Streams config for bounding off-heap memory

["Aditya Upadhyaya (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-12559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452128#comment-17452128 ] 

Aditya Upadhyaya commented on KAFKA-12559:
------------------------------------------

[~ableegoldman]: I have a fair bit of idea on what needs to be done and how to get started (thanks to the discussion section above + the memory management documentation). If I understand correctly, the first step is to create a KIP. Do I first run the KIP by you or should I put it on the mailing list directly? Additionally, I do have a few questions regarding the KIP section '{*}Migration Plan and{*} {*}Compatibility'{*}. What would be a good time for you to discuss this?

> Add a top-level Streams config for bounding off-heap memory
> -----------------------------------------------------------
>
>                 Key: KAFKA-12559
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12559
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: A. Sophie Blee-Goldman
>            Assignee: Aditya Upadhyaya
>            Priority: Major
>              Labels: needs-kip, newbie, newbie++
>
> At the moment we provide an example of how to bound the memory usage of rocskdb in the [Memory Management|https://kafka.apache.org/27/documentation/streams/developer-guide/memory-mgmt.html#rocksdb] section of the docs. This requires implementing a custom RocksDBConfigSetter class and setting a number of rocksdb options for relatively advanced concepts and configurations. It seems a fair number of users either fail to find this or consider it to be for more advanced use cases/users. But RocksDB can eat up a lot of off-heap memory and it's not uncommon for users to come across a {{RocksDBException: Cannot allocate memory}}
> It would probably be a much better user experience if we implemented this memory bound out-of-the-box and just gave users a top-level StreamsConfig to tune the off-heap memory given to rocksdb, like we have for on-heap cache memory with cache.max.bytes.buffering. More advanced users can continue to fine-tune their memory bounding and apply other configs with a custom config setter, while new or more casual users can cap on the off-heap memory without getting their hands dirty with rocksdb.
> I would propose to add the following top-level config:
> rocksdb.max.bytes.off.heap: medium priority, default to -1 (unbounded), valid values are [0, inf]
> I'd also want to consider adding a second, lower priority top-level config to give users a knob for adjusting how much of that total off-heap memory goes to the block cache + index/filter blocks, and how much of it is afforded to the write buffers. I'm struggling to come up with a good name for this config, but it would be something like
> rocksdb.memtable.to.block.cache.off.heap.memory.ratio: low priority, default to 0.5, valid values are [0, 1]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)