kafka streams offsets.retention.minutes

[Tom Dearman <to...@gmail.com>]

I recently realised that our consumers, which don’t use auto commit, will suffer from the issue outlined here (https://issues.apache.org/jira/browse/KAFKA-3806 <https://issues.apache.org/jira/browse/KAFKA-3806>).  Looking through the kafka streams code it will also suffer from this issue.  I know there is a work around to set the retention time to some large value, but given that stream aggregation is likely to be something that can happy over a long period of time, and that there might per times when a given partition is inactive, would the streams guys consider putting in some protection similar to what Jun Rao suggested in the issue ticket?  Something which will check the last actual commit offset time and override commitOffsetNeeded if it hasn’t ben saved within that time.

Thanks.

Re: kafka streams offsets.retention.minutes

[Guozhang Wang <wa...@gmail.com>]

Hello Tom,

I think there is a general solution proposed to tackle KAFKA-3806, which is
to re-define the semantics of "offset.retention.minutes" such that we will
only start timing it with this config value after the consumer group has
been removed at the coordinator side (i.e. all members are gone for this
group). But I'm not sure if it is agreed upon for this approach. CC'ing
Jason if he can provide some more insights.


Guozhang


On Fri, Mar 3, 2017 at 5:43 AM, Tom Dearman <to...@gmail.com> wrote:

> I recently realised that our consumers, which don’t use auto commit, will
> suffer from the issue outlined here (https://issues.apache.org/
> jira/browse/KAFKA-3806 <https://issues.apache.org/jira/browse/KAFKA-3806>).
> Looking through the kafka streams code it will also suffer from this
> issue.  I know there is a work around to set the retention time to some
> large value, but given that stream aggregation is likely to be something
> that can happy over a long period of time, and that there might per times
> when a given partition is inactive, would the streams guys consider putting
> in some protection similar to what Jun Rao suggested in the issue ticket?
> Something which will check the last actual commit offset time and override
> commitOffsetNeeded if it hasn’t ben saved within that time.
>
> Thanks.
>
>


-- 
-- Guozhang