[jira] [Created] (KAFKA-12274) Transactional operation fails when broker is replaced using the same broker ID.

["Michał Łukowicz (Jira)" <ji...@apache.org>]

Michał Łukowicz created KAFKA-12274:
---------------------------------------

             Summary: Transactional operation fails when broker is replaced using the same broker ID.
                 Key: KAFKA-12274
                 URL: https://issues.apache.org/jira/browse/KAFKA-12274
             Project: Kafka
          Issue Type: Bug
          Components: controller, producer 
    Affects Versions: 2.6.0
            Reporter: Michał Łukowicz


Hello Team!

One of our clusters is being used to:
 * process transactional writes
 * had ack set to all

We are using java client and followed all recommendation regarding avoiding dead fencing issues, etc.

We spotted the problem during upgrading kafka hosts to stronger machines:
 * stop old broker
 * 
start a new clean broker node (a different hostname) reusing the same broker.id 

During the operation we found that while kafka is normally replicating partitions to recover after very short period of time (1 - 3 mins) we start to see error on kafka broker:
broker=13] Error processing append operation on partition <partition>
org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order sequence number for producerId 51119 at offset 16878080903 in partition <partition>: 2930356 (incoming seq. number), 2930213 (current end sequence number)
And we are starting to see records buffered on the Producer side, and eventually, the producer send requests failed with::
Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for <topic>:120892 ms has passed since batch creation
The only additional thing we observed is that for some reason couple of paritions ISR had been reduced to 1.

The same situation can be observed when adding new brokers to cluster and performing rebalacing (using kafka cruise control) and setting concurrent partition and leader movements to higher value.

Can you please let me know if this is a bug ... or we are doing something wrong?

Kafka 2.6.0

min.insync.replica for topics is set to 1

replication.factor is 3

all transaction settings are currently default.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)