You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Patrik Kleindl (Jira)" <ji...@apache.org> on 2019/08/27 10:49:00 UTC
[jira] [Updated] (KAFKA-8766) Allow a custom offset policy for
Kafka Streams applications
[ https://issues.apache.org/jira/browse/KAFKA-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrik Kleindl updated KAFKA-8766:
----------------------------------
Description:
Currently when starting a new streams application (= new consumer group) you can only choose between starting from the beginning of all topics or only processing newly arriving records.
-To start processing at any give point in the past (e.g. only processing data of the last month) the application has to be started (so the consumer group exists), stopped, the offsets reset and then restarted.-
To start processing at any give point in the past (e.g. only processing data of the last month) the offsets have to be set with the streams application reset tool before the application is started. This is a new finding as the previous assumption was that the consumer group has to exist before doing the offset reset. Still, it would be easier if the offsets could be set without invoking a separate (command-line) tool.
It would be helpful if this could be passed in with the help of some kind of "offset reset strategy" which could be provided by the user.
was:
Currently when starting a new streams application (= new consumer group) you can only choose between starting from the beginning of all topics or only processing newly arriving records.
To start processing at any give point in the past (e.g. only processing data of the last month) the application has to be started (so the consumer group exists), stopped, the offsets reset and then restarted.
It would be helpful if this could be passed in with the help of some kind of "offset reset strategy" which could be provided by the user.
> Allow a custom offset policy for Kafka Streams applications
> ------------------------------------------------------------
>
> Key: KAFKA-8766
> URL: https://issues.apache.org/jira/browse/KAFKA-8766
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Patrik Kleindl
> Priority: Minor
>
> Currently when starting a new streams application (= new consumer group) you can only choose between starting from the beginning of all topics or only processing newly arriving records.
> -To start processing at any give point in the past (e.g. only processing data of the last month) the application has to be started (so the consumer group exists), stopped, the offsets reset and then restarted.-
> To start processing at any give point in the past (e.g. only processing data of the last month) the offsets have to be set with the streams application reset tool before the application is started. This is a new finding as the previous assumption was that the consumer group has to exist before doing the offset reset. Still, it would be easier if the offsets could be set without invoking a separate (command-line) tool.
> It would be helpful if this could be passed in with the help of some kind of "offset reset strategy" which could be provided by the user.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)