[jira] [Commented] (KAFKA-19992) Allow to configure max memory for restore path

["Muralidhar Basani (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18080840#comment-18080840 ] 

Muralidhar Basani commented on KAFKA-19992:
-------------------------------------------

https://cwiki.apache.org/confluence/display/KAFKA/KIP-1325%3A+Add+restore.buffered.records.per.partition+to+cap+Kafka+Streams+state-restore+memory

> Allow to configure max memory for restore path
> ----------------------------------------------
>
>                 Key: KAFKA-19992
>                 URL: https://issues.apache.org/jira/browse/KAFKA-19992
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Matthias J. Sax
>            Assignee: Muralidhar Basani
>            Priority: Major
>              Labels: needs-kip
>
> Kafka Streams allows users to configure many different memory limits, to pro-actively manage memory usage to avoid OutOfMemory errors.
> However, on the restore code path, there is no such memory management. The `StoreChangelogReader` fetches a batch of message, and buffers them inside `ChangelogMetadata`, and there is no limit enforced on this buffer.
> We could either re-use the existing config for input topic buffer (`
> buffered.records.per.partition`) or introduce a new config (this would require a KIP though). – Even if we re-use the existing config, not 100% sure if we would need a KIP or not?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)