You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "lianetm (via GitHub)" <gi...@apache.org> on 2024/08/22 22:54:58 UTC
[PR] KAFKA-17403: Generate HB to leave on pollOnClose if needed [kafka]
lianetm opened a new pull request, #16974:
URL: https://github.com/apache/kafka/pull/16974
Fix to ensure that the HB request to leave the group is generated when closing the HBRequestManager if the state is LEAVING. This is needed because we could end up closing the network thread without giving a chance to the HBManager to generate the request. This flow on consumer.close with short timeout:
1. app thread triggers releaseAssignmentAndLeaveGroup
2. background thread transitions to LEAVING
2.1 the next run of the background thread should poll the HB manager and generate a request
3. app thread releaseAssignmentAndLeaveGroup times out and moves on to close the network thread (stops polling managers. Calls pollOnClose to gather the final requests and send them along with the unsent)
If 3 happens in the app thread before 2.1 happens in the background, the HB manager won't have a chance to generate the request to leave. This PR implements the pollOnClose to generate the final request if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17403: Generate HB to leave on pollOnClose if needed [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #16974:
URL: https://github.com/apache/kafka/pull/16974#discussion_r1728682589
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/RequestManager.java:
##########
@@ -59,7 +59,7 @@ public interface RequestManager {
* <em>all</em> request managers is critical to ensure that we can complete as many of the consumer's shutdown
* tasks as possible within the user-provided timeout.
*/
- default PollResult pollOnClose() {
+ default PollResult pollOnClose(long currentTimeMs) {
Review Comment:
nit: Since we’ve added a new parameter `currentTimeMs: long`, I think we should add a description of this new parameter to the JavaDoc of the method. This would also help us align with the `pull` method.
WYDT ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17403: Generate HB to leave on pollOnClose if needed [kafka]
Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #16974:
URL: https://github.com/apache/kafka/pull/16974#discussion_r1733181704
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractHeartbeatRequestManager.java:
##########
@@ -195,6 +197,32 @@ public NetworkClientDelegate.PollResult poll(long currentTimeMs) {
*/
public abstract AbstractMembershipManager<R> membershipManager();
+ /**
+ * Generate a heartbeat request to leave the group if the state is still LEAVING when this is
+ * called to close the consumer.
Review Comment:
Is it possible that there's already a 'leave group' request that was sent? Won't `AsyncKafkaConsumer.releaseAssignmentAndLeaveGroup()` be called before before the request managers are closed.
Or does it not matter if we send more than one leave group request?
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetchRequestManagerTest.java:
##########
@@ -370,11 +370,10 @@ public void testFetcherCloseClosesFetchSessionsInBroker() {
final ArgumentCaptor<NetworkClientDelegate.UnsentRequest> argument = ArgumentCaptor.forClass(NetworkClientDelegate.UnsentRequest.class);
- Timer timer = time.timer(Duration.ofSeconds(10));
Review Comment:
So this timer was totally unnecessary all this time? Yeesh 😄
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ConsumerHeartbeatRequestManagerTest.java:
##########
@@ -156,7 +156,7 @@ private void createHeartbeatRequestStateWithZeroHeartbeatInterval() {
backgroundEventHandler);
}
- private void createHeartbeatStatAndRequestManager() {
+ private void createHeartbeatStateAndRequestManager() {
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17403: Generate HB to leave on pollOnClose if needed [kafka]
Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #16974:
URL: https://github.com/apache/kafka/pull/16974#discussion_r1733559475
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractHeartbeatRequestManager.java:
##########
@@ -195,6 +197,32 @@ public NetworkClientDelegate.PollResult poll(long currentTimeMs) {
*/
public abstract AbstractMembershipManager<R> membershipManager();
+ /**
+ * Generate a heartbeat request to leave the group if the state is still LEAVING when this is
+ * called to close the consumer.
Review Comment:
👍 Thanks for the explanation 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17403: Generate HB to leave on pollOnClose if needed [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #16974:
URL: https://github.com/apache/kafka/pull/16974#issuecomment-2308544944
> 2. background thread transitions to LEAVING
2.1 the next run of the background thread should poll the HB manager and generate a request
Sorry that I want to understand the root cause before reviewing. We had a discussion before (https://issues.apache.org/jira/browse/KAFKA-15305), and I assumed that step_2 and step_2.1 are executed in the same run.
```
void runOnce() {
// step_2: background thread transitions to LEAVING by handling UnsubscribeEvent
processApplicationEvents();
final long currentTimeMs = time.milliseconds();
final long pollWaitTimeMs = requestManagers.entries().stream()
.filter(Optional::isPresent)
.map(Optional::get)
// step_2.1: generate a LEAVE request as the state is `LEAVING` now
.map(rm -> rm.poll(currentTimeMs))
.map(networkClientDelegate::addAll)
.reduce(MAX_POLL_TIMEOUT_MS, Math::min);
networkClientDelegate.poll(pollWaitTimeMs, currentTimeMs);
...
}
```
I guess the scenario (step 2.1: the next run) you described is caused by `ConsumerRebalanceListener`, right? The `ConsumerRebalanceListener` must be executed by application thread, so background thread can not change the state to `LEAVING` before polling `RequestManager` if application thread does not execute `ConsumerRebalanceListener` in time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17403: Generate HB to leave on pollOnClose if needed [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #16974:
URL: https://github.com/apache/kafka/pull/16974#issuecomment-2310463005
Hey @chia7712 , good point, you're right that the transition to LEAVING can only happen within the `runOnce`, meaning that we know it will poll the managers once right after. I was indeed thinking about the callbacks, where the transition to LEAVING happens async, but I was missing the fact that it can only be when processing the CallbackCompleted event in the background (so poll after). Still, I guess the situation could be that the poll skips generating the HB (ex. no coordinator), so we end up in a pollOnClose with state LEAVING?
For context, I ended up in this corner case of pollOnClose + LEAVING while playing locally with some integration tests for close(0), with callbacks. Honestly I thought of the scenario I described above, missing the bit of the CallbackCompleted event. Still, re-thinking the root cause after your comment, I can only imagine this case: callbacks complete -> process events (transition to LEAVING) -> poll managers (skip due to no coord) -> runAtClose (pollOnClose in HB Mgr will be in LEAVING state). Makes sense? Corner case of course, but if the coordinator is known again at the time of closing, the pollOnClose would save the situation by handling LEAVING (and not hurt since most of the times, the expectation is that pollOnClose would happen with state UNSUBSCRIBED and do nothing). Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17403: Generate HB to leave on pollOnClose if needed [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 merged PR #16974:
URL: https://github.com/apache/kafka/pull/16974
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17403: Generate HB to leave on pollOnClose if needed [kafka]
Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on PR #16974:
URL: https://github.com/apache/kafka/pull/16974#issuecomment-2312989983
The `StreamThreadTest` test failures appear unrelated, as they're happening on one of my branches as well 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17403: Generate HB to leave on pollOnClose if needed [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #16974:
URL: https://github.com/apache/kafka/pull/16974#issuecomment-2307021079
Thanks for taking a look @frankvicky ! Let me know if you have any other thoughts.
Any chance you could take a look too @chia7712 ?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17403: Generate HB to leave on pollOnClose if needed [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #16974:
URL: https://github.com/apache/kafka/pull/16974#discussion_r1732216024
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManager.java:
##########
@@ -231,6 +233,15 @@ public ConsumerMembershipManager membershipManager() {
return membershipManager;
}
+ @Override
+ public PollResult pollOnClose(long currentTimeMs) {
Review Comment:
Could you please add the great comment (https://github.com/apache/kafka/pull/16974#issuecomment-2310463005)
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManager.java:
##########
@@ -231,6 +233,15 @@ public ConsumerMembershipManager membershipManager() {
return membershipManager;
}
+ @Override
+ public PollResult pollOnClose(long currentTimeMs) {
+ if (membershipManager.state() == MemberState.LEAVING) {
+ NetworkClientDelegate.UnsentRequest request = makeHeartbeatRequest(currentTimeMs, false);
Review Comment:
It is in closing, so maybe we can ignore the response?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17403: Generate HB to leave on pollOnClose if needed [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16974:
URL: https://github.com/apache/kafka/pull/16974#discussion_r1733231893
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractHeartbeatRequestManager.java:
##########
@@ -195,6 +197,32 @@ public NetworkClientDelegate.PollResult poll(long currentTimeMs) {
*/
public abstract AbstractMembershipManager<R> membershipManager();
+ /**
+ * Generate a heartbeat request to leave the group if the state is still LEAVING when this is
+ * called to close the consumer.
Review Comment:
There couldn't be a previous leave request, given that the state is LEAVING/PREPARE_LEAVING (basically the state machine controls that, when leaving, it transitions to UNSUBSCRIBED as soon as the request is sent, [here](https://github.com/apache/kafka/blob/62dc982ce9ce76b299a628d50648e3529fd88e63/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractMembershipManager.java#L646))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17403: Generate HB to leave on pollOnClose if needed [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #16974:
URL: https://github.com/apache/kafka/pull/16974#issuecomment-2312333360
> The case (close(0)) you described inspires me that processBackgroundEvents can return before processing ConsumerRebalanceListenerCallbackNeededEvent due to timeout. That means ConsumerRebalanceListener is NOT invoked and it is an inconsistent behavior.
Yeap, that would be another case, good catch. The member would stay PREPARE_LEAVING (and maybe add the CallbackNeededEvent in the background), but if the app thread does not have the time to see it while `processBackgroundEvents` we would move onto the `pollOnClose`. I included this consideration too, so basically if leaving (preapring or leaving) when closing the manager, makes sense to send the leave request, it's truly to last chance to do it before the network layer shuts down.
All comments addressed (and merged with the nice HBMgr refactoring). Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17403: Generate HB to leave on pollOnClose if needed [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16974:
URL: https://github.com/apache/kafka/pull/16974#discussion_r1728832826
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/RequestManager.java:
##########
@@ -59,7 +59,7 @@ public interface RequestManager {
* <em>all</em> request managers is critical to ensure that we can complete as many of the consumer's shutdown
* tasks as possible within the user-provided timeout.
*/
- default PollResult pollOnClose() {
+ default PollResult pollOnClose(long currentTimeMs) {
Review Comment:
makes sense, done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17403: Generate HB to leave on pollOnClose if needed [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #16974:
URL: https://github.com/apache/kafka/pull/16974#discussion_r1728682589
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/RequestManager.java:
##########
@@ -59,7 +59,7 @@ public interface RequestManager {
* <em>all</em> request managers is critical to ensure that we can complete as many of the consumer's shutdown
* tasks as possible within the user-provided timeout.
*/
- default PollResult pollOnClose() {
+ default PollResult pollOnClose(long currentTimeMs) {
Review Comment:
nit: Since we’ve added a new parameter currentTimeMs: long, I think we should add a description of this new parameter to the JavaDoc of the method. This would also help us align with the pull method.
WYDT ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org