You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "John Roesler (JIRA)" <ji...@apache.org> on 2018/12/04 00:42:00 UTC
[jira] [Commented] (KAFKA-7591) Changelog retention period doesn't
synchronise with window-store size
[ https://issues.apache.org/jira/browse/KAFKA-7591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708021#comment-16708021 ]
John Roesler commented on KAFKA-7591:
-------------------------------------
I looked into the code, and for reference, the reason the config doesn't get propagated is that in `org.apache.kafka.streams.processor.internals.InternalTopicManager#makeReady`, we first fetch the number of partitions for each internal topic in the topology. If the topic exists and has the right number of partitions, then we do nothing.
There's no idempotent create operation, so we'd either have to idempotently "alter configs" on every topic or fetch the configs and only alter the ones that don't match the configs that come from the topology.
I don't think it would be practical, without changing the code structure, to do this only for window stores, and it might be nice to have changes from other internal topic configs propagated.
> Changelog retention period doesn't synchronise with window-store size
> ---------------------------------------------------------------------
>
> Key: KAFKA-7591
> URL: https://issues.apache.org/jira/browse/KAFKA-7591
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Jon Bates
> Priority: Major
>
> When a new windowed state store is created, the associated changelog topic's `retention.ms` value is set to `window-size + CHANGELOG_ADDITIONAL_RETENTION_MS`
> h3. Expected Behaviour
> If the window-size is updated, the changelog topic's `retention.ms` config should be updated to reflect the new size
> h3. Actual Behaviour
> The changelog-topic's `retention.ms` setting is not amended, resulting in possible loss of data upon application restart
>
> n.b. Although it is easy to update changelog topic config, I logged this as `major` due to the potential for data-loss for any user of Kafka-Streams who may not be intimately aware of the relationship between a windowed store and the changelog config
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)