[jira] [Comment Edited] (KAFKA-19992) Allow to configure max memory for restore path

["Muralidhar Basani (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18068751#comment-18068751 ] 

Muralidhar Basani edited comment on KAFKA-19992 at 3/26/26 3:42 PM:
--------------------------------------------------------------------

[~mjsax] I tried to look at this. 

So only in the normal processing path, streams enforce a buffer limit via buffered.records.per.partition (default 1000), but when restore happens, there is no such logic in RestoreChangelogReader.

Either we can apply the same config buffered.records.per.partition in RestoreChangelogReader, but 1000 looks low, as it can slow down restoration and we should also consider polling. If polling is faster and restore is slow, we notice a lag.

We can introduce a new config *restore.buffered.records.per.partition* default to a higher default 10000 may be. and the update the [restore()|https://github.com/apache/kafka/blob/ccd8330e9eb0e2439f1d025282bef8b724d6ec1d/streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java#L438] method here

If you are ok, I am open a new kip

 


was (Author: muralibasani):
[~mjsax] I tried to look at this. 

So only in the normal processing path, streams enforce a buffer limit via buffered.records.per.partition (default 1000), but when restore happens, there is no such logic in RestoreChangelogReader.

Either we can apply the same config buffered.records.per.partition in RestoreChangelogReader, but 1000 looks low, as it can slow down restoration and we should also consider polling. If polling is faster and restore is slow, we notice a lag.

We can introduce a new config *restore.buffered.records.per.partition* default to a higher default 10000 may be. and the update the [restore()|https://github.com/apache/kafka/blob/ccd8330e9eb0e2439f1d025282bef8b724d6ec1d/streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java#L438] method here

 

> Allow to configure max memory for restore path
> ----------------------------------------------
>
>                 Key: KAFKA-19992
>                 URL: https://issues.apache.org/jira/browse/KAFKA-19992
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Matthias J. Sax
>            Priority: Major
>              Labels: needs-kip
>
> Kafka Streams allows users to configure many different memory limits, to pro-actively manage memory usage to avoid OutOfMemory errors.
> However, on the restore code path, there is no such memory management. The `StoreChangelogReader` fetches a batch of message, and buffers them inside `ChangelogMetadata`, and there is no limit enforced on this buffer.
> We could either re-use the existing config for input topic buffer (`
> buffered.records.per.partition`) or introduce a new config (this would require a KIP though). – Even if we re-use the existing config, not 100% sure if we would need a KIP or not?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)