[jira] [Created] (KAFKA-13979) Kafka resets committed offset after rebalance

["Fathima Khazana Abdul Haiyum (Jira)" <ji...@apache.org>]

Fathima Khazana Abdul Haiyum created KAFKA-13979:
----------------------------------------------------

             Summary: Kafka resets committed offset after rebalance
                 Key: KAFKA-13979
                 URL: https://issues.apache.org/jira/browse/KAFKA-13979
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 2.6.2
            Reporter: Fathima Khazana Abdul Haiyum


We have 3 nodes in our MSK cluster which run Apache Kafka 2.6.2. We have 15 partitions for a topic and 5 consumers in our consumer group, where each consumer runs on it's own java application server. Whenever we deploy(rolling) to our servers, we notice a huge consumer lag on *some* of the 15 partitions. It appears that the consumer after rebalancing resets its committed offset and reprocesses messages. For example: this is what I'm seeing:

{{}}
{code:java}

{code}
{{logger_name:org.apache.kafka.clients.consumer.internals.ConsumerCoordinator message:[Consumer clientId=myService-mytopic-0, groupId=myService-mytopic] Committed offset 3044 for partition mytopic-0}}

So we know for a fact that the offset 3044 has been committed for partition 0.

 

Running {{`./kafka-consumer-groups.sh --describe` }} gives the following:

{{}}
{code:java}

{code}
{{GROUP PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID CLIENT-ID myService-mytopic 0 3044 3044 0 myService-mytopic-0}}

{{  }}

 

After a deploy, which removes the consumer from the group and triggers a rebalance + adds the consumer back, I see this:

{{}}
{code:java}

{code}
{{GROUP PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID CLIENT-ID myService-mytopic 0 1890 3047 1157 myService-mytopic-0}}

{{  }}

 

In the application logs, I see this:

{{}}
{code:java}

{code}
{{logger_name:org.apache.kafka.clients.consumer.internals.Fetcher message:[Consumer clientId=myService-mytopic-0, groupId=myService-mytopic] Fetch position FetchPosition\{offset=1890, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[b-3.kafka-mytestserver.1gkwlu.c16.kafka.us-east-1.amazonaws.com:9098 (id: 3 rack: use1-az1)], epoch=0}} is out of range for partition mytopic-0, resetting offset}}

{{ }} Why is kafka fetching the current-offset 1890 which is beyond the committed offset for the partition after rebalance? This is on a test environment where less than 1 message is produced per second. This issue occurs for both auto commit (default interval) and manual commit mechanisms and on kafka versions 2.6.2 and 2.8.1. On production, we have much more traffic and causes reprocessing of around 2 million messages per partition. 

{{`auto.offset.reset=latest`}} if that matters. We're using the java client `kafka-clients` version 3.0.0.

 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)