You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mahendra Kariya <ma...@go-jek.com> on 2017/01/12 11:13:12 UTC
Kafka consumer offset info lost
Hey All,
We have a Kafka cluster hosted on Google Cloud. There was some network
issue on the cloud and suddenly, the offset for a particular consumer group
got reset to earliest and all of a sudden the lag was in millions. We
aren't able to figure out what went wrong. Has anybody faced the
same/similar issue? Does anybody have any debugging tips?
Some relevant info:
- The auto.offset.reset config for the consumer is set to earliest
- The offsets are stored on Kafka
- Total nodes on cluster: 4
- Replication factor: 3
- Partitions: 50
Re: Kafka consumer offset info lost
Posted by Mahendra Kariya <ma...@go-jek.com>.
Producers were publishing data for the topic. And consumers were also
connected, sending heartbeat pings every 100 ms.
On Thu, 12 Jan 2017 at 17:15 Michael Freeman <mi...@gmail.com> wrote:
> If the topic has not seen traffic for a while then Kafka will remove the
> stored offset. When your consumer reconnects Kafka no longer has the offset
> so it will reprocess from earliest.
>
> Michael
>
> > On 12 Jan 2017, at 11:13, Mahendra Kariya <ma...@go-jek.com>
> wrote:
> >
> > Hey All,
> >
> > We have a Kafka cluster hosted on Google Cloud. There was some network
> > issue on the cloud and suddenly, the offset for a particular consumer
> group
> > got reset to earliest and all of a sudden the lag was in millions. We
> > aren't able to figure out what went wrong. Has anybody faced the
> > same/similar issue? Does anybody have any debugging tips?
> >
> > Some relevant info:
> >
> > - The auto.offset.reset config for the consumer is set to earliest
> > - The offsets are stored on Kafka
> > - Total nodes on cluster: 4
> > - Replication factor: 3
> > - Partitions: 50
>
Re: Kafka consumer offset info lost
Posted by Michael Freeman <mi...@gmail.com>.
If the topic has not seen traffic for a while then Kafka will remove the stored offset. When your consumer reconnects Kafka no longer has the offset so it will reprocess from earliest.
Michael
> On 12 Jan 2017, at 11:13, Mahendra Kariya <ma...@go-jek.com> wrote:
>
> Hey All,
>
> We have a Kafka cluster hosted on Google Cloud. There was some network
> issue on the cloud and suddenly, the offset for a particular consumer group
> got reset to earliest and all of a sudden the lag was in millions. We
> aren't able to figure out what went wrong. Has anybody faced the
> same/similar issue? Does anybody have any debugging tips?
>
> Some relevant info:
>
> - The auto.offset.reset config for the consumer is set to earliest
> - The offsets are stored on Kafka
> - Total nodes on cluster: 4
> - Replication factor: 3
> - Partitions: 50