[jira] [Updated] (KAFKA-14069) Allow custom configuration of foreign key join internal topics

["Emmanuel Brard (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Emmanuel Brard updated KAFKA-14069:
-----------------------------------
    Affects Version/s: 2.8.0

> Allow custom configuration of foreign key join internal topics
> --------------------------------------------------------------
>
>                 Key: KAFKA-14069
>                 URL: https://issues.apache.org/jira/browse/KAFKA-14069
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>    Affects Versions: 2.8.0
>            Reporter: Emmanuel Brard
>            Priority: Minor
>
> Internal topic supporting foreign key joins (-subscription-registration-topic and -subscription-response-topic) are automatically created with_ infinite retention_ (retention.ms=-1, retention.bytes=-1).
> As far as I understand those topics are used for communication between tasks that are involved in the FK, the intermediate result though is persisted in a compacted topic (-subscription-store-changelog).
> This means, if I understood right, that during normal operation of the stream application, once a message is read from the registration/subscription topic, it will not be read again, even in case of recovery (the position in those topics is committed).
> Because we have very large tables being joined this way with very high changes frequency, we end up with FK internal topics in the order of 1 or 2 TB. This is complicated to maintain especially in term of disk space.
> I was wondering if:
> - this infinite retention is really a required configuration and if not
> - this infinite retention could be replaced with a configurable one (for example of 1 week, meaning that I accept that in case of failure I must this my app within one week)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)