Is there expiration for committed Offset in the partition

[Dinesh Subramanian <ds...@apptivo.co.in>]

Hi,

Facing duplication in below scenario

Last commit is happened in 3 days back in the consumer, after that no
messages produced in the topic. so no commits..
so after 3 days am stopping and restarting the consumer..  this time i
faced duplication issue in the consumer as i have this consumer
property "*auto.offset.reset
= earliest*", It is consumed again from the beginning.. any helps will be
appreciated.

*Thanks & Regards,*

*Dinesh S*

Re: Is there expiration for committed Offset in the partition

["Matthias J. Sax" <ma...@confluent.io>]

It is an know issue. You can increase the retention time for stored
offsets via configs thought.

There is already an open PR to fix this issue:
https://issues.apache.org/jira/browse/KAFKA-4682


-Matthias

On 6/1/18 2:00 AM, Dinesh Subramanian wrote:
> Hi M. Manna,
> 
> Am planning to store outside kafka.. will this be a solution ?
> 
> 
> *Thanks & Regards,*
> 
> *Dinesh S*
> 
> On Fri, Jun 1, 2018 at 2:05 PM, M. Manna <ma...@gmail.com> wrote:
> 
>> This can happen for two reasons:
>>
>> 1) Your offsets are expired and removed. So your consumers don't know where
>> to start from - earliest means "Start from the beginning"
>> 2) You are actually starting as part of a totally new consumer group - in
>> which case it's as designed too - start from the beginning.
>>
>> I would check your offset retention policy by size/time - and tune that if
>> necessary.
>>
>> On 1 June 2018 at 09:03, Dinesh Subramanian <ds...@apptivo.co.in>
>> wrote:
>>
>>> Hi,
>>>
>>> Facing duplication in below scenario
>>>
>>> Last commit is happened in 3 days back in the consumer, after that no
>>> messages produced in the topic. so no commits..
>>> so after 3 days am stopping and restarting the consumer..  this time i
>>> faced duplication issue in the consumer as i have this consumer
>>> property "*auto.offset.reset
>>> = earliest*", It is consumed again from the beginning.. any helps will be
>>> appreciated.
>>>
>>> *Thanks & Regards,*
>>>
>>> *Dinesh S*
>>>
>>
> 

Re: Is there expiration for committed Offset in the partition

[Dinesh Subramanian <ds...@apptivo.co.in>]

Hi M. Manna,

Am planning to store outside kafka.. will this be a solution ?


*Thanks & Regards,*

*Dinesh S*

On Fri, Jun 1, 2018 at 2:05 PM, M. Manna <ma...@gmail.com> wrote:

> This can happen for two reasons:
>
> 1) Your offsets are expired and removed. So your consumers don't know where
> to start from - earliest means "Start from the beginning"
> 2) You are actually starting as part of a totally new consumer group - in
> which case it's as designed too - start from the beginning.
>
> I would check your offset retention policy by size/time - and tune that if
> necessary.
>
> On 1 June 2018 at 09:03, Dinesh Subramanian <ds...@apptivo.co.in>
> wrote:
>
> > Hi,
> >
> > Facing duplication in below scenario
> >
> > Last commit is happened in 3 days back in the consumer, after that no
> > messages produced in the topic. so no commits..
> > so after 3 days am stopping and restarting the consumer..  this time i
> > faced duplication issue in the consumer as i have this consumer
> > property "*auto.offset.reset
> > = earliest*", It is consumed again from the beginning.. any helps will be
> > appreciated.
> >
> > *Thanks & Regards,*
> >
> > *Dinesh S*
> >
>

Re: Is there expiration for committed Offset in the partition

["M. Manna" <ma...@gmail.com>]

This can happen for two reasons:

1) Your offsets are expired and removed. So your consumers don't know where
to start from - earliest means "Start from the beginning"
2) You are actually starting as part of a totally new consumer group - in
which case it's as designed too - start from the beginning.

I would check your offset retention policy by size/time - and tune that if
necessary.

On 1 June 2018 at 09:03, Dinesh Subramanian <ds...@apptivo.co.in>
wrote:

> Hi,
>
> Facing duplication in below scenario
>
> Last commit is happened in 3 days back in the consumer, after that no
> messages produced in the topic. so no commits..
> so after 3 days am stopping and restarting the consumer..  this time i
> faced duplication issue in the consumer as i have this consumer
> property "*auto.offset.reset
> = earliest*", It is consumed again from the beginning.. any helps will be
> appreciated.
>
> *Thanks & Regards,*
>
> *Dinesh S*
>

Re: Is there expiration for committed Offset in the partition

[Hans Jespersen <ha...@confluent.io>]

You should just recommit the same offsets sooner than every 24 hours (or whatever your commit topic retention period is set to). The expiry of offsets is based on the timestamp of the commits.

-hans

> On Jun 1, 2018, at 1:03 AM, Dinesh Subramanian <ds...@apptivo.co.in> wrote:
> 
> Hi,
> 
> Facing duplication in below scenario
> 
> Last commit is happened in 3 days back in the consumer, after that no
> messages produced in the topic. so no commits..
> so after 3 days am stopping and restarting the consumer..  this time i
> faced duplication issue in the consumer as i have this consumer
> property "*auto.offset.reset
> = earliest*", It is consumed again from the beginning.. any helps will be
> appreciated.
> 
> *Thanks & Regards,*
> 
> *Dinesh S*