[jira] [Commented] (KAFKA-13549) Add "delete interval" config

["Nicholas Telford (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-13549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17461528#comment-17461528 ] 

Nicholas Telford commented on KAFKA-13549:
------------------------------------------

I drafted a patch for this and only noticed the "needs-kip" label after I submitted the PR. Hopefully it at least serves as a starting point :D

> Add "delete interval" config
> ----------------------------
>
>                 Key: KAFKA-13549
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13549
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Matthias J. Sax
>            Priority: Major
>              Labels: needs-kip
>
> Kafka Streams uses "delete record" requests to aggressively purge data from repartition topics. Those request are sent each time we commit.
> For at-least-once with a default commit interval of 30 seconds, this works fine. However, for exactly-once with a default commit interval of 100ms, it's very aggressive. The main issue is broker side, because the broker logs every "delete record" request, and thus broker logs are spammed if EOS is enabled.
> We should consider to add a new config (eg `delete.record.interval.ms` or similar) to have a dedicated config for "delete record" requests, to decouple it from the commit interval config and allow to purge data less aggressively, even if the commit interval is small to avoid the broker side log spamming.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)