You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Matthias J. Sax (JIRA)" <ji...@apache.org> on 2017/06/24 19:29:01 UTC
[jira] [Created] (KAFKA-5510) Streams should commit all offsets
regularly
Matthias J. Sax created KAFKA-5510:
--------------------------------------
Summary: Streams should commit all offsets regularly
Key: KAFKA-5510
URL: https://issues.apache.org/jira/browse/KAFKA-5510
Project: Kafka
Issue Type: Bug
Components: streams
Reporter: Matthias J. Sax
Currently, Streams commits only offsets of partitions it did process records for. Thus, if a partition does not have any data for longer then {{offsets.retention.minutes}} (default 1 day) the latest committed offset get's lost. On failure or restart {{auto.offset.rese}} kicks in potentially resulting in reprocessing old data.
Thus, Streams should commit _all_ offset on a regular basis. Not sure what the overhead of a commit is -- if it's too expensive to commit all offsets on regular commit, we could also have a second config that specifies an "commit.all.interval".
This relates to https://issues.apache.org/jira/browse/KAFKA-3806, so we should sync to get a solid overall solution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)