You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by di...@apache.org on 2023/07/10 14:11:00 UTC

[kafka] branch 3.5 updated (9b9de0fc28e -> a523d6dc33f)

This is an automated email from the ASF dual-hosted git repository.

divijv pushed a change to branch 3.5
in repository https://gitbox.apache.org/repos/asf/kafka.git


    from 9b9de0fc28e KAFKA-15149: Fix handling of new partitions in dual-write mode (#13968)
     new ff0c1df067d KAFKA-14929: Fixing flaky test putTopicStateRetriableFailure (#13634)
     new 5fc11164481 MINOR: Fix ListOffsetsRequestTest.testResponseIncludesLeaderEpoch (#13747)
     new cafefe70853 MINOR: Fix flaky DistributedHerderTest cases related to zombie fencing (#13806)
     new c73b603399a MINOR: Fix ZkAclMigrationClientTest.testAclsMigrateAndDualWrite test  (#13809)
     new b4c951104cb MINOR: Make sure replicas will not be removed in initial ISR (#13844)
     new b7a754477f0 MINOR: Add more information in assertion failure for non daemon threads (#13858)
     new 045c356701a MINOR: fix flaky ZkMigrationIntegrationTest.testNewAndChangedTopicsInDualWrite (#13902)
     new a523d6dc33f KAFKA-15140: Use TestUtils methods and add logs for assertion failure at TopicCommandIntegrationTest (#13950)

The 8 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runtime/distributed/DistributedHerder.java     |   7 +
 .../runtime/distributed/DistributedHerderTest.java |  35 +++--
 .../storage/KafkaStatusBackingStoreFormatTest.java |   4 +-
 .../kafka/admin/TopicCommandIntegrationTest.scala  | 169 ++++++++-------------
 .../kafka/zk/ZkMigrationIntegrationTest.scala      |   3 +-
 .../scala/unit/kafka/cluster/PartitionTest.scala   |   3 +-
 .../unit/kafka/server/ListOffsetsRequestTest.scala |   2 +-
 .../test/scala/unit/kafka/utils/TestUtils.scala    |   6 +-
 .../zk/migration/ZkAclMigrationClientTest.scala    |  15 +-
 9 files changed, 109 insertions(+), 135 deletions(-)


[kafka] 06/08: MINOR: Add more information in assertion failure for non daemon threads (#13858)

Posted by di...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

divijv pushed a commit to branch 3.5
in repository https://gitbox.apache.org/repos/asf/kafka.git

commit b7a754477f0c89d4c36e97cdbf233d7f854574c5
Author: Divij Vaidya <di...@amazon.com>
AuthorDate: Fri Jun 16 15:17:57 2023 +0200

    MINOR: Add more information in assertion failure for non daemon threads (#13858)
    
    Reviewers: Luke Chen <sh...@gmail.com>
---
 core/src/test/scala/unit/kafka/utils/TestUtils.scala | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/core/src/test/scala/unit/kafka/utils/TestUtils.scala b/core/src/test/scala/unit/kafka/utils/TestUtils.scala
index a499ac38344..3b1830da9e8 100755
--- a/core/src/test/scala/unit/kafka/utils/TestUtils.scala
+++ b/core/src/test/scala/unit/kafka/utils/TestUtils.scala
@@ -1337,10 +1337,12 @@ object TestUtils extends Logging {
   // Note: Call this method in the test itself, rather than the @AfterEach method.
   // Because of the assert, if assertNoNonDaemonThreads fails, nothing after would be executed.
   def assertNoNonDaemonThreads(threadNamePrefix: String): Unit = {
-    val threadCount = Thread.getAllStackTraces.keySet.asScala.count { t =>
+    val nonDaemonThreads = Thread.getAllStackTraces.keySet.asScala.filter { t =>
       !t.isDaemon && t.isAlive && t.getName.startsWith(threadNamePrefix)
     }
-    assertEquals(0, threadCount)
+
+    val threadCount = nonDaemonThreads.size
+    assertEquals(0, threadCount, s"Found unexpected $threadCount NonDaemon threads=${nonDaemonThreads.map(t => t.getName).mkString(", ")}")
   }
 
   def allThreadStackTraces(): String = {


[kafka] 07/08: MINOR: fix flaky ZkMigrationIntegrationTest.testNewAndChangedTopicsInDualWrite (#13902)

Posted by di...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

divijv pushed a commit to branch 3.5
in repository https://gitbox.apache.org/repos/asf/kafka.git

commit 045c356701aeac3d36ff4b57ecac25ea66ffebb1
Author: Chia-Ping Tsai <ch...@gmail.com>
AuthorDate: Wed Jun 28 22:45:26 2023 +0800

    MINOR: fix flaky ZkMigrationIntegrationTest.testNewAndChangedTopicsInDualWrite (#13902)
    
    Reviewers: David Arthur <mu...@gmail.com>
---
 .../test/scala/integration/kafka/zk/ZkMigrationIntegrationTest.scala   | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/core/src/test/scala/integration/kafka/zk/ZkMigrationIntegrationTest.scala b/core/src/test/scala/integration/kafka/zk/ZkMigrationIntegrationTest.scala
index 662f706b095..ed9b8370a1d 100644
--- a/core/src/test/scala/integration/kafka/zk/ZkMigrationIntegrationTest.scala
+++ b/core/src/test/scala/integration/kafka/zk/ZkMigrationIntegrationTest.scala
@@ -508,8 +508,7 @@ class ZkMigrationIntegrationTest {
       // Verify the changes seen in Zk.
       verifyTopicPartitionMetadata(topicName, existingPartitions ++ Seq(new TopicPartition(topicName, 2)), zkClient)
     } finally {
-      zkCluster.stop()
-      kraftCluster.close()
+      shutdownInSequence(zkCluster, kraftCluster)
     }
   }
 


[kafka] 08/08: KAFKA-15140: Use TestUtils methods and add logs for assertion failure at TopicCommandIntegrationTest (#13950)

Posted by di...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

divijv pushed a commit to branch 3.5
in repository https://gitbox.apache.org/repos/asf/kafka.git

commit a523d6dc33f5d32358ff072b7eed439e77135bbf
Author: DL1231 <53...@users.noreply.github.com>
AuthorDate: Tue Jul 4 22:02:39 2023 +0800

    KAFKA-15140: Use TestUtils methods and add logs for assertion failure at TopicCommandIntegrationTest (#13950)
    
    This commit utilizes TestUtils methods to create a topic and adds logs when assertions fail.
    
    Reviewers: Divij Vaidya <di...@amazon.com>
    
    ---------
    
    Co-authored-by: d00791190 <di...@huawei.com>
---
 .../kafka/admin/TopicCommandIntegrationTest.scala  | 169 ++++++++-------------
 1 file changed, 65 insertions(+), 104 deletions(-)

diff --git a/core/src/test/scala/integration/kafka/admin/TopicCommandIntegrationTest.scala b/core/src/test/scala/integration/kafka/admin/TopicCommandIntegrationTest.scala
index ea4c748da92..9e1beaf6adf 100644
--- a/core/src/test/scala/integration/kafka/admin/TopicCommandIntegrationTest.scala
+++ b/core/src/test/scala/integration/kafka/admin/TopicCommandIntegrationTest.scala
@@ -54,7 +54,7 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
     * `testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress`.
     */
   override def generateConfigs: Seq[KafkaConfig] = TestUtils.createBrokerConfigs(
-    numConfigs = 6,
+    numConfigs = numBrokers,
     zkConnect = zkConnectOrNull,
     rackInfo = Map(0 -> "rack1", 1 -> "rack2", 2 -> "rack2", 3 -> "rack1", 4 -> "rack3", 5 -> "rack3"),
     numPartitions = numPartitions,
@@ -66,6 +66,8 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
 
   private val numPartitions = 1
   private val defaultReplicationFactor = 1.toShort
+  private val numBrokers = 6
+  private val lineSeparator = System.lineSeparator()
 
   private var topicService: TopicService = _
   private var adminClient: Admin = _
@@ -253,7 +255,7 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
     val output = TestUtils.grabConsoleOutput(
       topicService.listTopics(new TopicCommandOptions(Array())))
 
-    assertTrue(output.contains(testTopicName))
+    assertTrue(output.contains(testTopicName), s"Unexpected output: $output")
   }
 
   @ParameterizedTest(name = TestInfoUtils.TestWithParameterizedQuorumName)
@@ -262,46 +264,36 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
     val topic1 = "kafka.testTopic1"
     val topic2 = "kafka.testTopic2"
     val topic3 = "oooof.testTopic1"
-    adminClient.createTopics(
-      List(new NewTopic(topic1, 2, 2.toShort),
-        new NewTopic(topic2, 2, 2.toShort),
-        new NewTopic(topic3, 2, 2.toShort)).asJavaCollection)
-      .all().get()
-    waitForTopicCreated(topic1)
-    waitForTopicCreated(topic2)
-    waitForTopicCreated(topic3)
+    TestUtils.createTopicWithAdmin(adminClient, topic1, brokers, 2, 2)
+    TestUtils.createTopicWithAdmin(adminClient, topic2, brokers, 2, 2)
+    TestUtils.createTopicWithAdmin(adminClient, topic3, brokers, 2, 2)
 
     val output = TestUtils.grabConsoleOutput(
       topicService.listTopics(new TopicCommandOptions(Array("--topic", "kafka.*"))))
 
-    assertTrue(output.contains(topic1))
-    assertTrue(output.contains(topic2))
-    assertFalse(output.contains(topic3))
+    assertTrue(output.contains(topic1), s"Unexpected output: $output")
+    assertTrue(output.contains(topic2), s"Unexpected output: $output")
+    assertFalse(output.contains(topic3), s"Unexpected output: $output")
   }
 
   @ParameterizedTest(name = TestInfoUtils.TestWithParameterizedQuorumName)
   @ValueSource(strings = Array("zk", "kraft"))
   def testListTopicsWithExcludeInternal(quorum: String): Unit = {
     val topic1 = "kafka.testTopic1"
-    adminClient.createTopics(
-      List(new NewTopic(topic1, 2, 2.toShort),
-        new NewTopic(Topic.GROUP_METADATA_TOPIC_NAME, 2, 2.toShort)).asJavaCollection)
-      .all().get()
-    waitForTopicCreated(topic1)
+    TestUtils.createTopicWithAdmin(adminClient, topic1, brokers, 2, 2)
+    TestUtils.createTopicWithAdmin(adminClient, Topic.GROUP_METADATA_TOPIC_NAME, brokers, 2, 2)
 
     val output = TestUtils.grabConsoleOutput(
       topicService.listTopics(new TopicCommandOptions(Array("--exclude-internal"))))
 
-    assertTrue(output.contains(topic1))
-    assertFalse(output.contains(Topic.GROUP_METADATA_TOPIC_NAME))
+    assertTrue(output.contains(topic1), s"Unexpected output: $output")
+    assertFalse(output.contains(Topic.GROUP_METADATA_TOPIC_NAME), s"Unexpected output: $output")
   }
 
   @ParameterizedTest(name = TestInfoUtils.TestWithParameterizedQuorumName)
   @ValueSource(strings = Array("zk", "kraft"))
   def testAlterPartitionCount(quorum: String): Unit = {
-    adminClient.createTopics(
-      List(new NewTopic(testTopicName, 2, 2.toShort)).asJavaCollection).all().get()
-    waitForTopicCreated(testTopicName)
+    TestUtils.createTopicWithAdmin(adminClient, testTopicName, brokers, 2, 2)
 
     topicService.alterTopic(new TopicCommandOptions(
       Array("--topic", testTopicName, "--partitions", "3")))
@@ -316,9 +308,7 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
   @ParameterizedTest(name = TestInfoUtils.TestWithParameterizedQuorumName)
   @ValueSource(strings = Array("zk", "kraft"))
   def testAlterAssignment(quorum: String): Unit = {
-    adminClient.createTopics(
-      Collections.singletonList(new NewTopic(testTopicName, 2, 2.toShort))).all().get()
-    waitForTopicCreated(testTopicName)
+    TestUtils.createTopicWithAdmin(adminClient, testTopicName, brokers, 2, 2)
 
     topicService.alterTopic(new TopicCommandOptions(
       Array("--topic", testTopicName, "--replica-assignment", "5:3,3:1,4:2", "--partitions", "3")))
@@ -334,9 +324,7 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
   @ParameterizedTest(name = TestInfoUtils.TestWithParameterizedQuorumName)
   @ValueSource(strings = Array("zk", "kraft"))
   def testAlterAssignmentWithMoreAssignmentThanPartitions(quorum: String): Unit = {
-    adminClient.createTopics(
-      List(new NewTopic(testTopicName, 2, 2.toShort)).asJavaCollection).all().get()
-    waitForTopicCreated(testTopicName)
+    TestUtils.createTopicWithAdmin(adminClient, testTopicName, brokers, 2, 2)
 
     assertThrows(classOf[ExecutionException],
       () => topicService.alterTopic(new TopicCommandOptions(
@@ -346,9 +334,7 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
   @ParameterizedTest(name = TestInfoUtils.TestWithParameterizedQuorumName)
   @ValueSource(strings = Array("zk", "kraft"))
   def testAlterAssignmentWithMorePartitionsThanAssignment(quorum: String): Unit = {
-    adminClient.createTopics(
-      List(new NewTopic(testTopicName, 2, 2.toShort)).asJavaCollection).all().get()
-    waitForTopicCreated(testTopicName)
+    TestUtils.createTopicWithAdmin(adminClient, testTopicName, brokers, 2, 2)
 
     assertThrows(classOf[ExecutionException],
       () => topicService.alterTopic(new TopicCommandOptions(
@@ -534,15 +520,13 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
   @ParameterizedTest(name = TestInfoUtils.TestWithParameterizedQuorumName)
   @ValueSource(strings = Array("zk", "kraft"))
   def testDescribe(quorum: String): Unit = {
-    adminClient.createTopics(
-      Collections.singletonList(new NewTopic(testTopicName, 2, 2.toShort))).all().get()
-    waitForTopicCreated(testTopicName)
+    TestUtils.createTopicWithAdmin(adminClient, testTopicName, brokers, 2, 2)
 
     val output = TestUtils.grabConsoleOutput(
       topicService.describeTopic(new TopicCommandOptions(Array("--topic", testTopicName))))
-    val rows = output.split("\n")
-    assertEquals(3, rows.size)
-    assertTrue(rows(0).startsWith(s"Topic: $testTopicName"))
+    val rows = output.split(lineSeparator)
+    assertEquals(3, rows.size, s"Unexpected output: $output")
+    assertTrue(rows(0).startsWith(s"Topic: $testTopicName"), s"Unexpected output: ${rows(0)}")
   }
 
   @ParameterizedTest(name = TestInfoUtils.TestWithParameterizedQuorumName)
@@ -561,9 +545,7 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
   @ParameterizedTest(name = TestInfoUtils.TestWithParameterizedQuorumName)
   @ValueSource(strings = Array("zk", "kraft"))
   def testDescribeUnavailablePartitions(quorum: String): Unit = {
-    adminClient.createTopics(
-      Collections.singletonList(new NewTopic(testTopicName, 6, 1.toShort))).all().get()
-    waitForTopicCreated(testTopicName)
+    TestUtils.createTopicWithAdmin(adminClient, testTopicName, brokers, numBrokers, 1)
 
     try {
       // check which partition is on broker 0 which we'll kill
@@ -594,9 +576,9 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
       val output = TestUtils.grabConsoleOutput(
           topicService.describeTopic(new TopicCommandOptions(
             Array("--topic", testTopicName, "--unavailable-partitions"))))
-      val rows = output.split("\n")
-      assertTrue(rows(0).startsWith(s"\tTopic: $testTopicName"))
-      assertTrue(rows(0).contains("Leader: none\tReplicas: 0\tIsr:"))
+      val rows = output.split(lineSeparator)
+      assertTrue(rows(0).startsWith(s"\tTopic: $testTopicName"), s"Unexpected output: ${rows(0)}")
+      assertTrue(rows(0).contains("Leader: none\tReplicas: 0\tIsr:"), s"Unexpected output: ${rows(0)}")
     } finally {
       restartDeadBrokers()
     }
@@ -605,9 +587,7 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
   @ParameterizedTest(name = TestInfoUtils.TestWithParameterizedQuorumName)
   @ValueSource(strings = Array("zk", "kraft"))
   def testDescribeUnderReplicatedPartitions(quorum: String): Unit = {
-    adminClient.createTopics(
-      Collections.singletonList(new NewTopic(testTopicName, 1, 6.toShort))).all().get()
-    waitForTopicCreated(testTopicName)
+    TestUtils.createTopicWithAdmin(adminClient, testTopicName, brokers, 1, numBrokers)
 
     try {
       killBroker(0)
@@ -618,7 +598,7 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
       }
       val output = TestUtils.grabConsoleOutput(
         topicService.describeTopic(new TopicCommandOptions(Array("--under-replicated-partitions"))))
-      val rows = output.split("\n")
+      val rows = output.split(lineSeparator)
       assertTrue(rows(0).startsWith(s"\tTopic: $testTopicName"), s"Unexpected output: ${rows(0)}")
     } finally {
       restartDeadBrokers()
@@ -628,12 +608,11 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
   @ParameterizedTest(name = TestInfoUtils.TestWithParameterizedQuorumName)
   @ValueSource(strings = Array("zk", "kraft"))
   def testDescribeUnderMinIsrPartitions(quorum: String): Unit = {
-    val configMap = new java.util.HashMap[String, String]()
-    configMap.put(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "6")
+    val topicProps = new Properties()
+    topicProps.setProperty(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, numBrokers.toString)
 
-    adminClient.createTopics(
-      Collections.singletonList(new NewTopic(testTopicName, 1, 6.toShort).configs(configMap))).all().get()
-    waitForTopicCreated(testTopicName)
+    // create topic
+    TestUtils.createTopicWithAdmin(adminClient, testTopicName, brokers, 1, numBrokers, topicConfig = topicProps)
 
     try {
       killBroker(0)
@@ -647,8 +626,8 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
       }
       val output = TestUtils.grabConsoleOutput(
         topicService.describeTopic(new TopicCommandOptions(Array("--under-min-isr-partitions"))))
-      val rows = output.split("\n")
-      assertTrue(rows(0).startsWith(s"\tTopic: $testTopicName"))
+      val rows = output.split(lineSeparator)
+      assertTrue(rows(0).startsWith(s"\tTopic: $testTopicName"), s"Unexpected output: ${rows(0)}")
     } finally {
       restartDeadBrokers()
     }
@@ -657,14 +636,9 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
   @ParameterizedTest(name = TestInfoUtils.TestWithParameterizedQuorumName)
   @ValueSource(strings = Array("zk", "kraft"))
   def testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress(quorum: String): Unit = {
-    val configMap = new java.util.HashMap[String, String]()
-    val replicationFactor: Short = 1
-    val partitions = 1
     val tp = new TopicPartition(testTopicName, 0)
 
-    adminClient.createTopics(
-      Collections.singletonList(new NewTopic(testTopicName, partitions, replicationFactor).configs(configMap))).all().get()
-    waitForTopicCreated(testTopicName)
+    TestUtils.createTopicWithAdmin(adminClient, testTopicName, brokers, 1, 1)
 
     // Produce multiple batches.
     TestUtils.generateAndProduceMessages(brokers, testTopicName, numMessages = 10, acks = -1)
@@ -693,9 +667,9 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
     // describe the topic and test if it's under-replicated
     val simpleDescribeOutput = TestUtils.grabConsoleOutput(
       topicService.describeTopic(new TopicCommandOptions(Array("--topic", testTopicName))))
-    val simpleDescribeOutputRows = simpleDescribeOutput.split("\n")
-    assertTrue(simpleDescribeOutputRows(0).startsWith(s"Topic: $testTopicName"))
-    assertEquals(2, simpleDescribeOutputRows.size)
+    val simpleDescribeOutputRows = simpleDescribeOutput.split(lineSeparator)
+    assertTrue(simpleDescribeOutputRows(0).startsWith(s"Topic: $testTopicName"), s"Unexpected output: ${simpleDescribeOutputRows(0)}")
+    assertEquals(2, simpleDescribeOutputRows.size, s"Unexpected output: $simpleDescribeOutput")
 
     val underReplicatedOutput = TestUtils.grabConsoleOutput(
       topicService.describeTopic(new TopicCommandOptions(Array("--under-replicated-partitions"))))
@@ -712,12 +686,11 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
   @ParameterizedTest(name = TestInfoUtils.TestWithParameterizedQuorumName)
   @ValueSource(strings = Array("zk", "kraft"))
   def testDescribeAtMinIsrPartitions(quorum: String): Unit = {
-    val configMap = new java.util.HashMap[String, String]()
-    configMap.put(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "4")
+    val topicProps = new Properties()
+    topicProps.setProperty(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "4")
 
-    adminClient.createTopics(
-      Collections.singletonList(new NewTopic(testTopicName, 1, 6.toShort).configs(configMap))).all().get()
-    waitForTopicCreated(testTopicName)
+    // create topic
+    TestUtils.createTopicWithAdmin(adminClient, testTopicName, brokers, 1, numBrokers, topicConfig = topicProps)
 
     try {
       killBroker(0)
@@ -734,9 +707,9 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
 
       val output = TestUtils.grabConsoleOutput(
         topicService.describeTopic(new TopicCommandOptions(Array("--at-min-isr-partitions"))))
-      val rows = output.split("\n")
-      assertTrue(rows(0).startsWith(s"\tTopic: $testTopicName"))
-      assertEquals(1, rows.length)
+      val rows = output.split(lineSeparator)
+      assertTrue(rows(0).startsWith(s"\tTopic: $testTopicName"), s"Unexpected output: ${rows(0)}")
+      assertEquals(1, rows.length, s"Unexpected output: $output")
     } finally {
       restartDeadBrokers()
     }
@@ -758,21 +731,14 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
     val notUnderMinIsrTopic = "not-under-min-isr-topic"
     val offlineTopic = "offline-topic"
     val fullyReplicatedTopic = "fully-replicated-topic"
+    val topicProps = new Properties()
+    topicProps.setProperty(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, numBrokers.toString)
 
-    val configMap = new java.util.HashMap[String, String]()
-    configMap.put(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "6")
-
-    adminClient.createTopics(
-      java.util.Arrays.asList(
-        new NewTopic(underMinIsrTopic, 1, 6.toShort).configs(configMap),
-        new NewTopic(notUnderMinIsrTopic, 1, 6.toShort),
-        new NewTopic(offlineTopic, Collections.singletonMap(0, Collections.singletonList(0))),
-        new NewTopic(fullyReplicatedTopic, Collections.singletonMap(0, java.util.Arrays.asList(1, 2, 3))))).all().get()
-
-    waitForTopicCreated(underMinIsrTopic)
-    waitForTopicCreated(notUnderMinIsrTopic)
-    waitForTopicCreated(offlineTopic)
-    waitForTopicCreated(fullyReplicatedTopic)
+    // create topic
+    TestUtils.createTopicWithAdmin(adminClient, underMinIsrTopic, brokers, 1, numBrokers, topicConfig = topicProps)
+    TestUtils.createTopicWithAdmin(adminClient, notUnderMinIsrTopic, brokers, 1, numBrokers)
+    TestUtils.createTopicWithAdmin(adminClient, offlineTopic, brokers, 1, replicaAssignment = Map(0 -> Seq(0)))
+    TestUtils.createTopicWithAdmin(adminClient, fullyReplicatedTopic, brokers, 1, replicaAssignment = Map(0 -> Seq(1, 2, 3)))
 
     try {
       killBroker(0)
@@ -782,17 +748,17 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
         TestUtils.waitUntilTrue(
           () => aliveBrokers.forall(
             broker =>
-              broker.metadataCache.getPartitionInfo(underMinIsrTopic, 0).get.isr().size() < 6 &&
+              broker.metadataCache.getPartitionInfo(underMinIsrTopic, 0).get.isr().size() < numBrokers &&
                 broker.metadataCache.getPartitionInfo(offlineTopic, 0).get.leader() == MetadataResponse.NO_LEADER_ID),
           "Timeout waiting for partition metadata propagating to brokers for underMinIsrTopic topic"
         )
       }
       val output = TestUtils.grabConsoleOutput(
         topicService.describeTopic(new TopicCommandOptions(Array("--under-min-isr-partitions"))))
-      val rows = output.split("\n")
-      assertTrue(rows(0).startsWith(s"\tTopic: $underMinIsrTopic"))
-      assertTrue(rows(1).startsWith(s"\tTopic: $offlineTopic"))
-      assertEquals(2, rows.length)
+      val rows = output.split(lineSeparator)
+      assertTrue(rows(0).startsWith(s"\tTopic: $underMinIsrTopic"), s"Unexpected output: ${rows(0)}")
+      assertTrue(rows(1).startsWith(s"\tTopic: $offlineTopic"), s"Unexpected output: ${rows(1)}")
+      assertEquals(2, rows.length, s"Unexpected output: $output")
     } finally {
       restartDeadBrokers()
     }
@@ -822,12 +788,12 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
     var output = TestUtils.grabConsoleOutput(topicService.describeTopic(new TopicCommandOptions(
       Array("--describe", "--exclude-internal"))))
     assertTrue(output.contains(testTopicName), s"Output should have contained $testTopicName")
-    assertFalse(output.contains(Topic.GROUP_METADATA_TOPIC_NAME))
+    assertFalse(output.contains(Topic.GROUP_METADATA_TOPIC_NAME), s"Unexpected output: $output")
 
     // test list
     output = TestUtils.grabConsoleOutput(topicService.listTopics(new TopicCommandOptions(Array("--list", "--exclude-internal"))))
-    assertTrue(output.contains(testTopicName))
-    assertFalse(output.contains(Topic.GROUP_METADATA_TOPIC_NAME))
+    assertTrue(output.contains(testTopicName), s"Unexpected output: $output")
+    assertFalse(output.contains(Topic.GROUP_METADATA_TOPIC_NAME), s"Unexpected output: $output")
   }
 
   @ParameterizedTest(name = TestInfoUtils.TestWithParameterizedQuorumName)
@@ -845,24 +811,19 @@ class TopicCommandIntegrationTest extends KafkaServerTestHarness with Logging wi
       Set(new TopicPartition(testTopicName, 0)).asJava
     )
 
-    adminClient.createTopics(
-      Collections.singletonList(new NewTopic(testTopicName, 1, 1.toShort))
-    ).all().get()
-    waitForTopicCreated(testTopicName)
+    TestUtils.createTopicWithAdmin(adminClient, testTopicName, brokers, 1, 1)
 
     val output = TestUtils.grabConsoleOutput(
       topicService.describeTopic(new TopicCommandOptions(Array("--topic", testTopicName))))
-    val rows = output.split("\n")
-    assertEquals(2, rows.size)
-    assertTrue(rows(0).startsWith(s"Topic: $testTopicName"))
+    val rows = output.split(lineSeparator)
+    assertEquals(2, rows.size, s"Unexpected output: $output")
+    assertTrue(rows(0).startsWith(s"Topic: $testTopicName"), s"Unexpected output: ${rows(0)}")
   }
 
   @ParameterizedTest(name = TestInfoUtils.TestWithParameterizedQuorumName)
   @ValueSource(strings = Array("zk", "kraft"))
   def testCreateWithTopicNameCollision(quorum: String): Unit = {
-    adminClient.createTopics(
-      Collections.singletonList(new NewTopic("foo_bar", 1, 6.toShort))).all().get()
-    waitForTopicCreated("foo_bar")
+    TestUtils.createTopicWithAdmin(adminClient, "foo_bar", brokers, 1, numBrokers)
 
     assertThrows(classOf[InvalidTopicException],
       () => topicService.createTopic(new TopicCommandOptions(Array("--topic", "foo.bar"))))


[kafka] 03/08: MINOR: Fix flaky DistributedHerderTest cases related to zombie fencing (#13806)

Posted by di...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

divijv pushed a commit to branch 3.5
in repository https://gitbox.apache.org/repos/asf/kafka.git

commit cafefe708530992c58924cae50270befee3672dd
Author: Chris Egerton <ch...@aiven.io>
AuthorDate: Mon Jun 5 12:50:54 2023 -0700

    MINOR: Fix flaky DistributedHerderTest cases related to zombie fencing (#13806)
    
    Reviewers: Yash Mayya <ya...@gmail.com>, Chris Egerton <ch...@aiven.io>
---
 .../runtime/distributed/DistributedHerder.java     |  7 +++++
 .../runtime/distributed/DistributedHerderTest.java | 35 ++++++++++++----------
 2 files changed, 27 insertions(+), 15 deletions(-)

diff --git a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java
index 88f92f53f17..98543e6fc21 100644
--- a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java
+++ b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java
@@ -2146,6 +2146,13 @@ public class DistributedHerder extends AbstractHerder implements Runnable {
     DistributedHerderRequest addRequest(long delayMs, Callable<Void> action, Callback<Void> callback) {
         DistributedHerderRequest req = new DistributedHerderRequest(time.milliseconds() + delayMs, requestSeqNum.incrementAndGet(), action, callback);
         requests.add(req);
+        // We don't need to synchronize here
+        // If the condition evaluates to true, we can and should trigger a wakeup
+        // If it evaluates to false because our request has suddenly been popped off of the queue, then
+        // the herder is already running the request and no wakeup is necessary
+        // If it evaluates to false because our request was not at the head of the queue, then a wakeup
+        // should already have been triggered when the request that is currently at the head of the
+        // queue was added
         if (peekWithoutException() == req)
             member.wakeup();
         return req;
diff --git a/connect/runtime/src/test/java/org/apache/kafka/connect/runtime/distributed/DistributedHerderTest.java b/connect/runtime/src/test/java/org/apache/kafka/connect/runtime/distributed/DistributedHerderTest.java
index 33b8f966e2e..cc4147737df 100644
--- a/connect/runtime/src/test/java/org/apache/kafka/connect/runtime/distributed/DistributedHerderTest.java
+++ b/connect/runtime/src/test/java/org/apache/kafka/connect/runtime/distributed/DistributedHerderTest.java
@@ -3337,8 +3337,17 @@ public class DistributedHerderTest {
 
         expectAnyTicks();
 
+        // There's a race condition in DistributedHerder::addRequest that isn't triggered
+        // often under normal circumstances, but becomes more likely during this test
+        // because we mock out member::poll and the herder no longer blocks when invoking
+        // that method
+        // This can cause the herder to skip a call to member::wakeup during that method, and
+        // since the number of calls to that method isn't vital to our testing coverage, we permit
+        // any number of calls
+        // The race condition itself is benign and should have no negative impact on herder logic;
+        // see DistributedHerder::addRequest for more detail
         member.wakeup();
-        EasyMock.expectLastCall();
+        EasyMock.expectLastCall().anyTimes();
 
         org.easymock.IExpectationSetters<RestClient.HttpResponse<Object>> expectRequest = EasyMock.expect(
                 restClient.httpRequest(
@@ -3357,7 +3366,7 @@ public class DistributedHerderTest {
             return null;
         });
 
-        expectHerderShutdown(true);
+        expectHerderShutdown();
         forwardRequestExecutor.shutdown();
         EasyMock.expectLastCall();
         EasyMock.expect(forwardRequestExecutor.awaitTermination(anyLong(), anyObject())).andReturn(true);
@@ -3442,7 +3451,7 @@ public class DistributedHerderTest {
             EasyMock.expectLastCall();
         }
 
-        expectHerderShutdown(false);
+        expectHerderShutdown();
 
         PowerMock.replayAll();
 
@@ -3477,7 +3486,7 @@ public class DistributedHerderTest {
         expectAnyTicks();
 
         member.wakeup();
-        EasyMock.expectLastCall();
+        EasyMock.expectLastCall().anyTimes();
 
         ClusterConfigState configState = exactlyOnceSnapshot(
                 sessionKey,
@@ -3515,7 +3524,7 @@ public class DistributedHerderTest {
         configBackingStore.putTaskCountRecord(CONN1, 1);
         EasyMock.expectLastCall();
 
-        expectHerderShutdown(true);
+        expectHerderShutdown();
 
         PowerMock.replayAll(workerFencingFuture, herderFencingFuture);
 
@@ -3548,7 +3557,7 @@ public class DistributedHerderTest {
         expectAnyTicks();
 
         member.wakeup();
-        EasyMock.expectLastCall();
+        EasyMock.expectLastCall().anyTimes();
 
         ClusterConfigState configState = exactlyOnceSnapshot(
                 sessionKey,
@@ -3563,7 +3572,7 @@ public class DistributedHerderTest {
         EasyMock.expect(worker.fenceZombies(EasyMock.eq(CONN1), EasyMock.eq(2), EasyMock.eq(CONN1_CONFIG)))
                 .andThrow(fencingException);
 
-        expectHerderShutdown(true);
+        expectHerderShutdown();
 
         PowerMock.replayAll();
 
@@ -3598,7 +3607,7 @@ public class DistributedHerderTest {
         expectAnyTicks();
 
         member.wakeup();
-        EasyMock.expectLastCall();
+        EasyMock.expectLastCall().anyTimes();
 
         ClusterConfigState configState = exactlyOnceSnapshot(
                 sessionKey,
@@ -3630,7 +3639,7 @@ public class DistributedHerderTest {
             });
         }
 
-        expectHerderShutdown(true);
+        expectHerderShutdown();
 
         PowerMock.replayAll(workerFencingFuture, herderFencingFuture);
 
@@ -3752,7 +3761,7 @@ public class DistributedHerderTest {
             EasyMock.expectLastCall();
         });
 
-        expectHerderShutdown(false);
+        expectHerderShutdown();
 
         PowerMock.replayAll();
 
@@ -4333,11 +4342,7 @@ public class DistributedHerderTest {
         EasyMock.expectLastCall();
     }
 
-    private void expectHerderShutdown(boolean wakeup) {
-        if (wakeup) {
-            member.wakeup();
-            EasyMock.expectLastCall();
-        }
+    private void expectHerderShutdown() {
         worker.stopAndAwaitConnectors();
         EasyMock.expectLastCall();
         worker.stopAndAwaitTasks();


[kafka] 01/08: KAFKA-14929: Fixing flaky test putTopicStateRetriableFailure (#13634)

Posted by di...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

divijv pushed a commit to branch 3.5
in repository https://gitbox.apache.org/repos/asf/kafka.git

commit ff0c1df067dcfc27292fda488fdb29c48ba0a10f
Author: vamossagar12 <sa...@gmail.com>
AuthorDate: Wed Apr 26 17:53:56 2023 +0530

    KAFKA-14929: Fixing flaky test putTopicStateRetriableFailure (#13634)
    
    Co-authored-by: Sagar Rao <sa...@Sagars-MacBook-Pro.local>
    
    Reviewers: Daniel Urban <du...@cloudera.com>, Justine Olshan <jo...@confluent.io>, Manyanda Chitimbo <ma...@gmail.com>, Viktor Somogyi-Vass <vi...@gmail.com>
---
 .../kafka/connect/storage/KafkaStatusBackingStoreFormatTest.java      | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/connect/runtime/src/test/java/org/apache/kafka/connect/storage/KafkaStatusBackingStoreFormatTest.java b/connect/runtime/src/test/java/org/apache/kafka/connect/storage/KafkaStatusBackingStoreFormatTest.java
index 6b509a08baa..6d5a4a3c726 100644
--- a/connect/runtime/src/test/java/org/apache/kafka/connect/storage/KafkaStatusBackingStoreFormatTest.java
+++ b/connect/runtime/src/test/java/org/apache/kafka/connect/storage/KafkaStatusBackingStoreFormatTest.java
@@ -48,7 +48,7 @@ import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.assertNull;
 import static org.junit.Assert.assertTrue;
-import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.timeout;
 import static org.mockito.Mockito.verify;
 
 @SuppressWarnings("unchecked")
@@ -216,7 +216,7 @@ public class KafkaStatusBackingStoreFormatTest {
         }).when(kafkaBasedLog).send(eq(key), valueCaptor.capture(), any(Callback.class));
 
         store.put(topicStatus);
-        verify(kafkaBasedLog, times(2)).send(any(), any(), any());
+        verify(kafkaBasedLog, timeout(1000).times(2)).send(any(), any(), any());
 
         // check capture state
         assertEquals(topicStatus, store.parseTopicStatus(valueCaptor.getValue()));


[kafka] 04/08: MINOR: Fix ZkAclMigrationClientTest.testAclsMigrateAndDualWrite test (#13809)

Posted by di...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

divijv pushed a commit to branch 3.5
in repository https://gitbox.apache.org/repos/asf/kafka.git

commit c73b603399acb9b06e4844e4d14a40288e1bd5b7
Author: Aman Singh <10...@users.noreply.github.com>
AuthorDate: Wed Jun 7 15:41:08 2023 +0530

    MINOR: Fix ZkAclMigrationClientTest.testAclsMigrateAndDualWrite test  (#13809)
    
    Reviewers: Manikumar Reddy <ma...@gmail.com>
---
 .../kafka/zk/migration/ZkAclMigrationClientTest.scala     | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/core/src/test/scala/unit/kafka/zk/migration/ZkAclMigrationClientTest.scala b/core/src/test/scala/unit/kafka/zk/migration/ZkAclMigrationClientTest.scala
index 7913d09c665..0107b9fa954 100644
--- a/core/src/test/scala/unit/kafka/zk/migration/ZkAclMigrationClientTest.scala
+++ b/core/src/test/scala/unit/kafka/zk/migration/ZkAclMigrationClientTest.scala
@@ -47,14 +47,14 @@ class ZkAclMigrationClientTest extends ZkMigrationTestHarness {
     authorizer: AclAuthorizer,
     resourcePattern: ResourcePattern,
     aces: Seq[AccessControlEntry],
-    pred: Seq[AclBinding] => Boolean
-  ): Seq[AclBinding] = {
+    pred: Set[AclBinding] => Boolean
+  ): Set[AclBinding] = {
     val resourceFilter = new AclBindingFilter(
       new ResourcePatternFilter(resourcePattern.resourceType(), resourcePattern.name(), resourcePattern.patternType()),
       AclBindingFilter.ANY.entryFilter()
     )
     migrationState = migrationClient.aclClient().writeResourceAcls(resourcePattern, aces.asJava, migrationState)
-    val (acls, ok) = TestUtils.computeUntilTrue(authorizer.acls(resourceFilter).asScala.toSeq)(pred)
+    val (acls, ok) = TestUtils.computeUntilTrue(authorizer.acls(resourceFilter).asScala.toSet)(pred)
     assertTrue(ok)
     acls
   }
@@ -68,7 +68,7 @@ class ZkAclMigrationClientTest extends ZkMigrationTestHarness {
       AclBindingFilter.ANY.entryFilter()
     )
     migrationState = migrationClient.aclClient().deleteResource(resourcePattern, migrationState)
-    val (_, ok) = TestUtils.computeUntilTrue(authorizer.acls(resourceFilter).asScala.toSeq)(_.isEmpty)
+    val (_, ok) = TestUtils.computeUntilTrue(authorizer.acls(resourceFilter).asScala.toSet)(_.isEmpty)
     assertTrue(ok)
   }
 
@@ -98,7 +98,7 @@ class ZkAclMigrationClientTest extends ZkMigrationTestHarness {
 
       // Remove one of resource1's ACLs
       var resource1Acls = replaceAclsAndReadWithAuthorizer(authorizer, resource1, Seq(ace1), acls => acls.size == 1)
-      assertEquals(acl1, resource1Acls.head)
+      assertTrue(resource1Acls.contains(acl1))
 
       // Delete the other ACL from resource1
       deleteResourceAndReadWithAuthorizer(authorizer, resource1)
@@ -106,13 +106,12 @@ class ZkAclMigrationClientTest extends ZkMigrationTestHarness {
       // Add a new ACL for resource1
       val newAce1 = new AccessControlEntry(principal.toString, "10.0.0.1", AclOperation.WRITE, AclPermissionType.ALLOW)
       resource1Acls = replaceAclsAndReadWithAuthorizer(authorizer, resource1, Seq(newAce1), acls => acls.size == 1)
-      assertEquals(newAce1, resource1Acls.head.entry())
+      assertTrue(resource1Acls.map(_.entry()).contains(newAce1))
 
       // Add a new ACL for resource2
       val newAce2 = new AccessControlEntry(principal.toString, "10.0.0.1", AclOperation.WRITE, AclPermissionType.ALLOW)
       val resource2Acls = replaceAclsAndReadWithAuthorizer(authorizer, resource2, Seq(acl3.entry(), newAce2), acls => acls.size == 2)
-      assertEquals(acl3, resource2Acls.head)
-      assertEquals(newAce2, resource2Acls.last.entry())
+      assertTrue(resource2Acls.map(_.entry()).subsetOf(Set(acl3.entry(), newAce2)))
     } finally {
       authorizer.close()
     }


[kafka] 02/08: MINOR: Fix ListOffsetsRequestTest.testResponseIncludesLeaderEpoch (#13747)

Posted by di...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

divijv pushed a commit to branch 3.5
in repository https://gitbox.apache.org/repos/asf/kafka.git

commit 5fc1116448171f014167eef6050351c6fdc75573
Author: David Jacot <dj...@confluent.io>
AuthorDate: Wed May 24 10:25:28 2023 +0200

    MINOR: Fix ListOffsetsRequestTest.testResponseIncludesLeaderEpoch (#13747)
    
    Fix flaky ListOffsetsRequestTest.testResponseIncludesLeaderEpoch. It specifically fixes failures seen [here](https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-13535/33/tests/).
    
    Reviewers: Luke Chen <sh...@gmail.com>
---
 core/src/test/scala/unit/kafka/server/ListOffsetsRequestTest.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/core/src/test/scala/unit/kafka/server/ListOffsetsRequestTest.scala b/core/src/test/scala/unit/kafka/server/ListOffsetsRequestTest.scala
index 654ffb5ae8d..7e1f3300e56 100644
--- a/core/src/test/scala/unit/kafka/server/ListOffsetsRequestTest.scala
+++ b/core/src/test/scala/unit/kafka/server/ListOffsetsRequestTest.scala
@@ -187,7 +187,7 @@ class ListOffsetsRequestTest extends BaseRequestTest {
     killBroker(firstLeaderId)
     val secondLeaderId = TestUtils.awaitLeaderChange(servers, partition, firstLeaderId)
     // make sure high watermark of new leader has caught up
-    TestUtils.waitUntilTrue(() => sendRequest(secondLeaderId, 0L, -1).errorCode() != Errors.OFFSET_NOT_AVAILABLE.code(),
+    TestUtils.waitUntilTrue(() => sendRequest(secondLeaderId, ListOffsetsRequest.LATEST_TIMESTAMP, -1).errorCode != Errors.OFFSET_NOT_AVAILABLE.code,
       "the second leader does not sync to follower")
     val secondLeaderEpoch = TestUtils.findLeaderEpoch(secondLeaderId, partition, servers)
 


[kafka] 05/08: MINOR: Make sure replicas will not be removed in initial ISR (#13844)

Posted by di...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

divijv pushed a commit to branch 3.5
in repository https://gitbox.apache.org/repos/asf/kafka.git

commit b4c951104cba1e968ee96c92b1ed185d661d655d
Author: Calvin Liu <83...@users.noreply.github.com>
AuthorDate: Tue Jun 13 05:02:10 2023 -0700

    MINOR: Make sure replicas will not be removed in initial ISR (#13844)
    
    Reviewers: David Jacot <dj...@confluent.io>
---
 core/src/test/scala/unit/kafka/cluster/PartitionTest.scala | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/core/src/test/scala/unit/kafka/cluster/PartitionTest.scala b/core/src/test/scala/unit/kafka/cluster/PartitionTest.scala
index 47b4a99274b..c28c6e5d9eb 100644
--- a/core/src/test/scala/unit/kafka/cluster/PartitionTest.scala
+++ b/core/src/test/scala/unit/kafka/cluster/PartitionTest.scala
@@ -1864,9 +1864,10 @@ class PartitionTest extends AbstractPartitionTest {
 
     // On initialization, the replica is considered caught up and should not be removed
     partition.maybeShrinkIsr()
+    assertEquals(alterPartitionManager.isrUpdates.size, 0)
     assertEquals(Set(brokerId, remoteBrokerId1, remoteBrokerId2), partition.partitionState.isr)
 
-    // If enough time passes without a fetch update, the ISR should shrink
+    // If enough time passes without a fetch update, the ISR should shrink after the following maybeShrinkIsr
     time.sleep(partition.replicaLagTimeMaxMs + 1)
 
     // Shrink the ISR