[jira] [Created] (KAFKA-13131) Consumer offsets lost during partition reassignment

["Sergejs Andrejevs (Jira)" <ji...@apache.org>]

Sergejs Andrejevs created KAFKA-13131:
-----------------------------------------

             Summary: Consumer offsets lost during partition reassignment
                 Key: KAFKA-13131
                 URL: https://issues.apache.org/jira/browse/KAFKA-13131
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 2.6.0
            Reporter: Sergejs Andrejevs


While doing replicas reassignment of a *___consumer_offsets_* partition from one set of brokers to another, the consumer group offset got lost (seems to be reset to earliest).

Initial setup:
 __consumer_offsets-18 
 Replicas: 9,7,6

Desired setup:
 __consumer_offsets-18
 Replicas: 11,10,5

File_with_desired_state:
{code:java}
{
  "version": 1,
  "partitions": [
    {
      "topic": "__consumer_offsets",
      "partition": 18,
      "replicas": [
        11,
        10,
        5
      ],
      "log_dirs": [
        "/path_replica_1",
        "/path_replica_2",
        "/path_replica_3"
      ]
    }
  ]
}
{code}
Reassignment command:
{code:java}
/opt/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --execute --reassignment-json-file File_with_desired_state --throttle 104857600 --replica-alter-log-dirs-throttle 104857600
{code}
The error in logs at the broker:
{code:java}
[2021-07-22 05:28:11,777] ERROR [GroupMetadataManager brokerId=11] Error loading offsets from __consumer_offsets-18 (kafka.coordinator.group.GroupMetadataManager)
java.lang.NullPointerException
        at kafka.log.OffsetIndex.$anonfun$lookup$1(OffsetIndex.scala:90)
        at kafka.log.AbstractIndex.maybeLock(AbstractIndex.scala:338)
        at kafka.log.OffsetIndex.lookup(OffsetIndex.scala:89)
        at kafka.log.LogSegment.translateOffset(LogSegment.scala:274)
        at kafka.log.LogSegment.read(LogSegment.scala:298)
        at kafka.log.Log.$anonfun$read$2(Log.scala:1522)
        at kafka.log.Log.read(Log.scala:2340)
        at kafka.coordinator.group.GroupMetadataManager.loadGroupsAndOffsets(GroupMetadataManager.scala:589)
        at kafka.coordinator.group.GroupMetadataManager.$anonfun$scheduleLoadGroupAndOffsets$2(GroupMetadataManager.scala:537)
        at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
{code}
It was tried to reproduce at test environments, but so far unsuccessfully.


 Let me know if any other configuration/parameters/details shall be added.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)