You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "bbejeck (via GitHub)" <gi...@apache.org> on 2025/11/21 21:52:50 UTC
[PR] MINOR: Address flaky wait for producer acks logging for checking when not al… [kafka]
bbejeck opened a new pull request, #20957:
URL: https://github.com/apache/kafka/pull/20957
Adding the following for some system tests
- Increased timeout for `await_produced_messages`
- Added more logging for `test_fencing_consumer` which has exhibited flakiness and is hard to reproduce
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Address flaky wait for producer acks logging for checking when not al… [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on code in PR #20957:
URL: https://github.com/apache/kafka/pull/20957#discussion_r2553325785
##########
tests/kafkatest/tests/client/consumer_test.py:
##########
@@ -306,7 +306,7 @@ def test_fencing_static_consumer(self, num_conflict_consumers, fencing_stage, me
producer = self.setup_producer(self.TOPIC)
producer.start()
- self.await_produced_messages(producer)
+ self.await_produced_messages(producer, timeout_sec=120)
Review Comment:
We've seen starting a producer can take longer than expected but it does start so increasing the timeout to reduce flakiness
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Address flaky wait for producer acks logging for checking when not al… [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on code in PR #20957:
URL: https://github.com/apache/kafka/pull/20957#discussion_r2557562833
##########
tests/kafkatest/tests/client/consumer_test.py:
##########
@@ -330,10 +330,19 @@ def test_fencing_static_consumer(self, num_conflict_consumers, fencing_stage, me
timeout_sec=10,
err_msg="Timed out waiting for the fenced consumers to stop")
else:
- # Consumer protocol: Existing members should remain active and new conflicting ones should not be able to join.
- self.await_consumed_messages(consumer)
- assert num_rebalances == consumer.num_rebalances(), "Static consumers attempt to join with instance id in use should not cause a rebalance"
Review Comment:
The trouble with going between different repos
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Address flaky wait for producer acks logging for checking when not al… [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on PR #20957:
URL: https://github.com/apache/kafka/pull/20957#issuecomment-3577446601
merged #20957 into trunk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Address flaky wait for producer acks logging for checking when not al… [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on PR #20957:
URL: https://github.com/apache/kafka/pull/20957#issuecomment-3572496795
@lianetm updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Address flaky wait for producer acks logging for checking when not al… [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on PR #20957:
URL: https://github.com/apache/kafka/pull/20957#issuecomment-3572157687
@lianetm thanks for the review, I updated the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Address flaky wait for producer acks logging for checking when not al… [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck merged PR #20957:
URL: https://github.com/apache/kafka/pull/20957
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Address flaky wait for producer acks logging for checking when not al… [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on PR #20957:
URL: https://github.com/apache/kafka/pull/20957#issuecomment-3577548051
cherry-picked to 4.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Address flaky wait for producer acks logging for checking when not al… [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on code in PR #20957:
URL: https://github.com/apache/kafka/pull/20957#discussion_r2557295776
##########
tests/kafkatest/tests/client/consumer_test.py:
##########
@@ -346,11 +356,20 @@ def test_fencing_static_consumer(self, num_conflict_consumers, fencing_stage, me
timeout_sec=60,
err_msg="Timed out waiting for the consumer to shutdown")
# Wait until the group becomes empty to ensure the instance ID is released.
- # We use the 50-second timeout because the consumer session timeout is 45 seconds.
+ self.logger.debug("Stopped consumers waiting for group %s to be removed from the group", self.group_id)
Review Comment:
Good catch - I didn't mean to include that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Address flaky wait for producer acks logging for checking when not al… [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on code in PR #20957:
URL: https://github.com/apache/kafka/pull/20957#discussion_r2557561578
##########
tests/kafkatest/tests/client/consumer_test.py:
##########
@@ -330,10 +330,19 @@ def test_fencing_static_consumer(self, num_conflict_consumers, fencing_stage, me
timeout_sec=10,
err_msg="Timed out waiting for the fenced consumers to stop")
else:
- # Consumer protocol: Existing members should remain active and new conflicting ones should not be able to join.
- self.await_consumed_messages(consumer)
- assert num_rebalances == consumer.num_rebalances(), "Static consumers attempt to join with instance id in use should not cause a rebalance"
- assert len(consumer.joined_nodes()) == len(consumer.nodes)
+ self.logger.debug("Members status - Joined [%d/%d]: %s, Not joined: %s",
+ len(consumer.joined_nodes()),
+ len(consumer.nodes),
+ " ".join(str(node.account) for node in consumer.joined_nodes()),
+ " ".join(
+ set(str(node.account) for node in consumer.nodes) - set(consumer.joined_nodes())))
Review Comment:
I don't follow exactly - but doesn't matter I removed in favor of only logging in error cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Address flaky wait for producer acks logging for checking when not al… [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #20957:
URL: https://github.com/apache/kafka/pull/20957#discussion_r2557385410
##########
tests/kafkatest/tests/client/consumer_test.py:
##########
@@ -344,16 +353,23 @@ def test_fencing_static_consumer(self, num_conflict_consumers, fencing_stage, me
consumer.stop_all()
wait_until(lambda: len(consumer.dead_nodes()) == len(consumer.nodes),
timeout_sec=60,
- err_msg="Timed out waiting for the consumer to shutdown")
+ err_msg="Timed out waiting for the consumer to shutdown. Describe output is %s" % " ".join(self.kafka.describe_consumer_group_members(self.group_id)))
# Wait until the group becomes empty to ensure the instance ID is released.
- # We use the 50-second timeout because the consumer session timeout is 45 seconds.
+
+
+ # We use the 60-second timeout because the consumer session timeout is 45 seconds adding some time for latency.
wait_until(lambda: self.group_id in self.kafka.list_consumer_groups(state="empty"),
- timeout_sec=50,
- err_msg="Timed out waiting for the consumers to be removed from the group.")
+ timeout_sec=60,
+ err_msg="Timed out waiting for the consumers to be removed from the group Describe output is %s." % " ".join(self.kafka.describe_consumer_group_members(self.group_id)))
Review Comment:
```suggestion
err_msg="Timed out waiting for the consumers to be removed from the group. Describe output is %s." % " ".join(self.kafka.describe_consumer_group_members(self.group_id)))
```
##########
tests/kafkatest/tests/client/consumer_test.py:
##########
@@ -330,10 +330,19 @@ def test_fencing_static_consumer(self, num_conflict_consumers, fencing_stage, me
timeout_sec=10,
err_msg="Timed out waiting for the fenced consumers to stop")
else:
- # Consumer protocol: Existing members should remain active and new conflicting ones should not be able to join.
- self.await_consumed_messages(consumer)
- assert num_rebalances == consumer.num_rebalances(), "Static consumers attempt to join with instance id in use should not cause a rebalance"
- assert len(consumer.joined_nodes()) == len(consumer.nodes)
+ self.logger.debug("Members status - Joined [%d/%d]: %s, Not joined: %s",
+ len(consumer.joined_nodes()),
+ len(consumer.nodes),
+ " ".join(str(node.account) for node in consumer.joined_nodes()),
+ " ".join(
+ set(str(node.account) for node in consumer.nodes) - set(consumer.joined_nodes())))
Review Comment:
seems we're missing something here? it has set(account) - set(node)
##########
tests/kafkatest/tests/client/consumer_test.py:
##########
@@ -330,10 +330,19 @@ def test_fencing_static_consumer(self, num_conflict_consumers, fencing_stage, me
timeout_sec=10,
err_msg="Timed out waiting for the fenced consumers to stop")
else:
- # Consumer protocol: Existing members should remain active and new conflicting ones should not be able to join.
- self.await_consumed_messages(consumer)
- assert num_rebalances == consumer.num_rebalances(), "Static consumers attempt to join with instance id in use should not cause a rebalance"
Review Comment:
we still need these 2 calls right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Address flaky wait for producer acks logging for checking when not al… [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #20957:
URL: https://github.com/apache/kafka/pull/20957#discussion_r2556545264
##########
tests/kafkatest/tests/client/consumer_test.py:
##########
@@ -346,11 +356,20 @@ def test_fencing_static_consumer(self, num_conflict_consumers, fencing_stage, me
timeout_sec=60,
err_msg="Timed out waiting for the consumer to shutdown")
# Wait until the group becomes empty to ensure the instance ID is released.
- # We use the 50-second timeout because the consumer session timeout is 45 seconds.
+ self.logger.debug("Stopped consumers waiting for group %s to be removed from the group", self.group_id)
Review Comment:
I don't quite get what we want to say here?
##########
tests/kafkatest/tests/client/consumer_test.py:
##########
@@ -346,11 +356,20 @@ def test_fencing_static_consumer(self, num_conflict_consumers, fencing_stage, me
timeout_sec=60,
err_msg="Timed out waiting for the consumer to shutdown")
# Wait until the group becomes empty to ensure the instance ID is released.
- # We use the 50-second timeout because the consumer session timeout is 45 seconds.
+ self.logger.debug("Stopped consumers waiting for group %s to be removed from the group", self.group_id)
+
+ if len(consumer.joined_nodes()) != 0:
+ self.logger.debug("Not all stopped consumers removed %s. Describe output is %s", self.group_id, " ".join(self.kafka.describe_consumer_group_members(self.group_id)))
+
+ # We use the 60-second timeout because the consumer session timeout is 45 seconds adding some time for latency.
wait_until(lambda: self.group_id in self.kafka.list_consumer_groups(state="empty"),
- timeout_sec=50,
+ timeout_sec=60,
err_msg="Timed out waiting for the consumers to be removed from the group.")
conflict_consumer.start()
+
+ if len(consumer.joined_nodes()) != num_conflict_consumers:
+ self.logger.debug("All conflict members not in group %s. Describe output is %s", self.group_id, " ".join(self.kafka.describe_consumer_group_members(self.group_id)))
Review Comment:
similar with this, should this log be in the except after trying to `await_members(conflict_consumer, ..)` and not succeeding? logging before waiting would lead to false negatives I expect?
##########
tests/kafkatest/tests/client/consumer_test.py:
##########
@@ -346,11 +356,20 @@ def test_fencing_static_consumer(self, num_conflict_consumers, fencing_stage, me
timeout_sec=60,
err_msg="Timed out waiting for the consumer to shutdown")
# Wait until the group becomes empty to ensure the instance ID is released.
- # We use the 50-second timeout because the consumer session timeout is 45 seconds.
+ self.logger.debug("Stopped consumers waiting for group %s to be removed from the group", self.group_id)
+
+ if len(consumer.joined_nodes()) != 0:
Review Comment:
can this condition be true at this point? (I would imagine that if ever there were still `joined_nodes`, the condition above on `wait_until(lambda: len(consumer.dead_nodes()) == len(consumer.nodes)` would fail before hitting this line, but I could be missing something).
If what I imagine is the case, then I guess we should wait in a try/except to be able to log what we want in the catch maybe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org