[jira] [Created] (KAFKA-7563) Single broker sends incorrect metadata for topic partitions

["Martin Kamp Jensen (JIRA)" <ji...@apache.org>]

Martin Kamp Jensen created KAFKA-7563:
-----------------------------------------

             Summary: Single broker sends incorrect metadata for topic partitions
                 Key: KAFKA-7563
                 URL: https://issues.apache.org/jira/browse/KAFKA-7563
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 2.0.0
            Reporter: Martin Kamp Jensen
         Attachments: kafka.log, zookeeper.log

When starting our Kafka Streams application in a test setup with just one Kafka broker we are seeing the following error roughly 1 out of 15 runs:

{{StreamsException: Existing internal topic alarm-message-streams-alarm-from-unknown-asset-changelog has invalid partitions: expected: 32; actual: 25. Use 'kafka.tools.StreamsResetter' tool to clean up invalid topics before processing.}}

(Note: It is not always the same topic that causes the error.)

When we see the error above the actual number of partitions varies (expected is 32, actual is above 0 and below 32).

Before each test run the Kafka broker is started without data (using [https://hub.docker.com/r/wurstmeister/kafka/]).

We have never seen this happen in non-test where we are running with 6 Kafka brokers. However, we are running a significantly higher number of test runs than deploys to non-test.

After some investigation (including using AdminClient to describe the topics when the Kafka Streams application got the StreamsException and confirming that AdminClient also reports that a topic has the wrong number of partitions!) we implemented the following workaround: When the Kafka Streams application fails with the exception, we stop the application, stop the Kafka broker, start the Kafka broker, and finally start the application. Then the exception is not thrown. Of course this does not explain or fix the real issue at hand but it is still important because we all hate flaky tests.

Kafka and ZooKeeper log files from a run where the exception above occurred and where applying the workaround described above enabled us to continue without the exception are attached.

This issue was created by request of Matthias J. Sax at https://stackoverflow.com/questions/52943653/existing-internal-topic-has-invalid-partitions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)