You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Matthias J. Sax (JIRA)" <ji...@apache.org> on 2019/02/19 02:36:00 UTC
[jira] [Commented] (KAFKA-5510) Streams should commit all offsets
regularly
[ https://issues.apache.org/jira/browse/KAFKA-5510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771512#comment-16771512 ]
Matthias J. Sax commented on KAFKA-5510:
----------------------------------------
With https://issues.apache.org/jira/browse/KAFKA-4682 being fixed, this ticket is still an issue if Kafka Streams connect to older brokers – however, I am wondering if we need to address it because https://issues.apache.org/jira/browse/KAFKA-4682 seems to provide a proper fix and thus it seems not necessary to resolve this ticket any longer. (Also because there is an existing workaround by preserving offsets longer). Thus, I am wondering if we should close this ticket? Thoughts?
> Streams should commit all offsets regularly
> -------------------------------------------
>
> Key: KAFKA-5510
> URL: https://issues.apache.org/jira/browse/KAFKA-5510
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Matthias J. Sax
> Priority: Major
>
> Currently, Streams commits only offsets of partitions it did process records for. Thus, if a partition does not have any data for longer then {{offsets.retention.minutes}} (default 1 day) the latest committed offset get's lost. On failure or restart {{auto.offset.rese}} kicks in potentially resulting in reprocessing old data.
> Thus, Streams should commit _all_ offset on a regular basis. Not sure what the overhead of a commit is -- if it's too expensive to commit all offsets on regular commit, we could also have a second config that specifies an "commit.all.interval".
> This relates to https://issues.apache.org/jira/browse/KAFKA-3806, so we should sync to get a solid overall solution.
> At the same time, it might be better to change the semantics of {{offsets.retention.minutes}} in the first place. It might be better to apply this setting only if the consumer group is completely dead (and not on "last commit" and "per partition" basis). Thus, this JIRA would be a workaround fix if core cannot be changed quickly enough.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)