You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/03/17 05:33:15 UTC

[GitHub] [kafka] showuon opened a new pull request #10338: KAFKA-10251: wait for consumer rebalance completed before consuming records

showuon opened a new pull request #10338:
URL: https://github.com/apache/kafka/pull/10338


   
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] showuon commented on pull request #10338: KAFKA-10251: wait for consumer rebalance completed before consuming records

Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #10338:
URL: https://github.com/apache/kafka/pull/10338#issuecomment-821186344


   All failed tests are unrelated. Thanks.
   ```
   
       Build / JDK 15 and Scala 2.13 / kafka.server.RaftClusterTest.testCreateClusterAndCreateAndManyTopics()
       Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationSSLTest.testReplication()
       Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationSSLTest.testOneWayReplicationWithAutoOffsetSync()
       Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationTest.testReplication()
       Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationTest.testReplicationWithEmptyPartition()
       Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationTest.testOneWayReplicationWithAutoOffsetSync()
       Build / JDK 11 and Scala 2.13 / org.apache.kafka.streams.integration.KTableKTableForeignKeyInnerJoinMultiIntegrationTest.shouldInnerJoinMultiPartitionQueryable
       Build / JDK 11 and Scala 2.13 / org.apache.kafka.streams.integration.KTableKTableForeignKeyInnerJoinMultiIntegrationTest.shouldInnerJoinMultiPartitionQueryable
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] showuon commented on pull request #10338: KAFKA-10251: wait for consumer rebalance completed before consuming records

Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #10338:
URL: https://github.com/apache/kafka/pull/10338#issuecomment-818752568


   @apurvam @hachikuji  , please help review. Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] showuon commented on pull request #10338: KAFKA-10251: wait for consumer rebalance completed before consuming records

Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #10338:
URL: https://github.com/apache/kafka/pull/10338#issuecomment-844635104


   @mumrah , thanks for the comments. This flaky test only can be reproduced in a very slow machine, like jenkins. I investigate it by adding some debug log and run in jenkins, and check the logs when failed. After my fix, I run it 100 times on jenkins and no failed happened. I believe it's a reasonable fix. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] showuon commented on pull request #10338: KAFKA-10251: wait for consumer rebalance completed before consuming records

Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #10338:
URL: https://github.com/apache/kafka/pull/10338#issuecomment-856390836


   @mumrah , could you help have a 2nd review? Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] showuon commented on pull request #10338: KAFKA-10251: wait for consumer rebalance completed before consuming records

Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #10338:
URL: https://github.com/apache/kafka/pull/10338#issuecomment-848770650


   @mumrah , could you please take a look again? Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] showuon commented on pull request #10338: KAFKA-10251: wait for consumer rebalance completed before consuming records

Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #10338:
URL: https://github.com/apache/kafka/pull/10338#issuecomment-863801749


   @mumrah  ,call for 2nd review? Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] ijuma commented on pull request #10338: KAFKA-10251: wait for consumer rebalance completed before consuming records

Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #10338:
URL: https://github.com/apache/kafka/pull/10338#issuecomment-844071874


   @mumrah maybe you can help review this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] mumrah commented on a change in pull request #10338: KAFKA-10251: wait for consumer rebalance completed before consuming records

Posted by GitBox <gi...@apache.org>.
mumrah commented on a change in pull request #10338:
URL: https://github.com/apache/kafka/pull/10338#discussion_r635464179



##########
File path: core/src/test/scala/integration/kafka/api/TransactionsBounceTest.scala
##########
@@ -83,16 +84,33 @@ class TransactionsBounceTest extends IntegrationTestHarness {
       producer.sendOffsetsToTransaction(TestUtils.consumerPositions(consumer).asJava, consumer.groupMetadata()))
   }
 
+  private def waitForRebalancingCompleted(consumer: Consumer[Array[Byte], Array[Byte]],
+                                          expectedAssignment: Set[TopicPartition]): Unit = {
+    TestUtils.pollUntilTrue(consumer, () => consumer.assignment() == expectedAssignment.asJava,
+      s"Timed out while waiting expected assignment $expectedAssignment. " +
+        s"The current assignment is ${consumer.assignment()}", waitTimeMs = rebalanceTimeout)

Review comment:
       nit: alignment seems off




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] showuon commented on a change in pull request #10338: KAFKA-10251: wait for consumer rebalance completed before consuming records

Posted by GitBox <gi...@apache.org>.
showuon commented on a change in pull request #10338:
URL: https://github.com/apache/kafka/pull/10338#discussion_r635701748



##########
File path: core/src/test/scala/integration/kafka/api/TransactionsBounceTest.scala
##########
@@ -83,16 +84,33 @@ class TransactionsBounceTest extends IntegrationTestHarness {
       producer.sendOffsetsToTransaction(TestUtils.consumerPositions(consumer).asJava, consumer.groupMetadata()))
   }
 
+  private def waitForRebalancingCompleted(consumer: Consumer[Array[Byte], Array[Byte]],
+                                          expectedAssignment: Set[TopicPartition]): Unit = {
+    TestUtils.pollUntilTrue(consumer, () => consumer.assignment() == expectedAssignment.asJava,
+      s"Timed out while waiting expected assignment $expectedAssignment. " +
+        s"The current assignment is ${consumer.assignment()}", waitTimeMs = rebalanceTimeout)

Review comment:
       Updated. Thanks.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] showuon commented on pull request #10338: KAFKA-10251: wait for consumer rebalance completed before consuming records

Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #10338:
URL: https://github.com/apache/kafka/pull/10338#issuecomment-813859513


   @apurvam @hachikuji , call for review. Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] showuon commented on pull request #10338: KAFKA-10251: wait for consumer rebalance completed before consuming records

Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #10338:
URL: https://github.com/apache/kafka/pull/10338#issuecomment-800863820


   @apurvam @hachikuji , could you review this PR? Thanks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] showuon commented on pull request #10338: KAFKA-10251: wait for consumer rebalance completed before consuming records

Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #10338:
URL: https://github.com/apache/kafka/pull/10338#issuecomment-876394407


   @mumrah , it failed frequently these days. I think we should merge this fix soon. Please help have a 2nd review when available. Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] showuon commented on pull request #10338: KAFKA-10251: wait for consumer rebalance completed before consuming records

Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #10338:
URL: https://github.com/apache/kafka/pull/10338#issuecomment-804554602


   @apurvam  @hachikuji  , could you review this PR? Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] showuon commented on pull request #10338: KAFKA-10251: wait for consumer rebalance completed before consuming records

Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #10338:
URL: https://github.com/apache/kafka/pull/10338#issuecomment-844819847


   failed tests are unrelated and flaky (and no `TransactionsBounceTest` failed):
   ```
       Build / JDK 15 and Scala 2.13 / kafka.network.SocketServerTest.testClientDisconnectionUpdatesRequestMetrics()
       Build / JDK 8 and Scala 2.12 / kafka.api.TransactionsTest.testAbortTransactionTimeout()
       Build / JDK 8 and Scala 2.12 / kafka.network.SocketServerTest.testUnmuteChannelWithBufferedReceives()
       Build / JDK 11 and Scala 2.13 / kafka.server.MultipleListenersWithAdditionalJaasContextTest.testProduceConsume()
   ```
   Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org