You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "lianetm (via GitHub)" <gi...@apache.org> on 2024/07/10 21:35:07 UTC
[PR] KAFKA-16984: Complete consumer leave on response to leave request [kafka]
lianetm opened a new pull request, #16569:
URL: https://github.com/apache/kafka/pull/16569
Improvement to ensure that, even though the leave group request is sent out once, the leave group operation is considered complete only when it receives a response to the HB to leave (successful or failed).
The motivation is to avoid undesired interactions with operations triggered after the unsubscribe (ex. consumer close triggers a leave group operation, and shuts down the network thread when it completes, which before this PR could lead to responses to disconnected clients).
This is the same behaviour on close that the legacy consumer has (waits for leave responses).
Note that this PR does not change the transitions of the state machine on leave group, only the completion of the leave group future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16984: Complete consumer leave on response to leave request [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16569:
URL: https://github.com/apache/kafka/pull/16569#discussion_r1676141611
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImplTest.java:
##########
@@ -166,33 +167,13 @@ public void testMembershipManagerInitSupportsEmptyGroupInstanceId() {
createMembershipManagerJoiningGroup();
}
- @Test
- public void testMembershipManagerRegistersForClusterMetadataUpdatesOnFirstJoin() {
Review Comment:
this is just small clean-up unrelated to this PR (but I found it while on it). This test does not apply anymore, and it passes only because it's not really testing what its name suggest.
The `membershipMgr` did register for metadata updates at some point, but it stopped doing it the moment it was turned into a manager that could be polled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16984: Complete consumer leave on response to leave request [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16569:
URL: https://github.com/apache/kafka/pull/16569#discussion_r1681354714
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java:
##########
@@ -459,8 +464,30 @@ public void onHeartbeatSuccess(ConsumerGroupHeartbeatResponseData response) {
}
@Override
- public void onHeartbeatFailure() {
- metricsManager.maybeRecordRebalanceFailed();
+ public void onHeartbeatFailure(boolean retriable) {
+ if (!retriable) {
+ metricsManager.maybeRecordRebalanceFailed();
+ }
+ // The leave group request is sent out once (not retried), so we should complete the leave
+ // operation once the request completes, regardless of the response.
+ if (state == MemberState.UNSUBSCRIBED && maybeCompleteLeaveInProgress()) {
+ log.warn("Member {} with epoch {} received a failed response to the heartbeat to " +
+ "leave the group and completed the leave operation. ", memberId, memberEpoch);
+ }
+ }
+
+ /**
+ * Complete the leave in progress (if any), if the member is UNSUBSCRIBED. This is expected to
Review Comment:
the "maybe" in the name is not because of the `MemberState`, it's because of the `leaveInProgress` that may be present or not. Even though most of the time this is needed when the state is `UNSUBSCRIBED`, we also need to `maybeCompleteLeaveInProgress` when the state is `FATAL`, so that's why I chose not to include the state check at this level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16984: Complete consumer leave on response to leave request [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16569:
URL: https://github.com/apache/kafka/pull/16569#discussion_r1681358088
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java:
##########
@@ -587,6 +617,14 @@ public void transitionToFatal() {
return;
}
+ if (previousState == MemberState.LEAVING || previousState == MemberState.PREPARE_LEAVING) {
+ log.info("Member {} with epoch {} was leaving the group with state {} when it got a " +
+ "fatal error from the broker. It will discard the ongoing leave and remain in {} " +
+ "state.", memberId, memberEpoch, previousState, state);
+ maybeCompleteLeaveInProgress();
Review Comment:
All in the above response. The state is indeed not `UNSUBSCRIBE` here, the state is FATAL and we do have to `maybeCompleteLeaveInProgress` (to make sure we complete a leave operation that was running before we got the fatal error in a HB response while let's say, running onPartitionsRevoked callback to leave the group). Makes sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16984: Complete consumer leave on response to leave request [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16569:
URL: https://github.com/apache/kafka/pull/16569#discussion_r1681346936
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManager.java:
##########
@@ -78,14 +78,14 @@ public interface MembershipManager extends RequestManager {
/**
* Notify the member that an error heartbeat response was received.
*/
- void onHeartbeatFailure();
+ void onHeartbeatFailure(boolean retriable);
Review Comment:
the change is that `onHeartbeatFailure` is now called for all failures (retriable and non-retriable), so seemed sensible to have a param indicating the nature of the failure because different actions may be required depending on it. It's not specific to metrics, even though that's the only action done now on non-retriable.
I will add a param description here along the lines of "retriable -> True if the heartbeat request failed with a retriable error". Nothing specific to the metrics, it feels its very specific to the impl, and clear enough from it (and would have to be maintained the moment we come add more action to the `onHeartbeatFailure`), makes sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16984: Complete consumer leave on response to leave request [kafka]
Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #16569:
URL: https://github.com/apache/kafka/pull/16569#discussion_r1676322238
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java:
##########
@@ -587,6 +617,14 @@ public void transitionToFatal() {
return;
}
+ if (previousState == MemberState.LEAVING || previousState == MemberState.PREPARE_LEAVING) {
+ log.info("Member {} with epoch {} was leaving the group with state {} when it got a " +
+ "fatal error from the broker. It will discard the ongoing leave and remain in {} " +
+ "state.", memberId, memberEpoch, previousState, state);
+ maybeCompleteLeaveInProgress();
Review Comment:
Per the above comment, there's no immediate checks around this call to `maybeCompleteLeaveInProgress()` that the state is `UNSUBSCRIBED`. From my reading of it, line 606 sets the state to `FATAL`.
Again, not trying to nitpick here, I promise 😄 I'm just trying to understand the code and the expected state(s) when calling `maybeCompleteLeaveInProgress()`.
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManager.java:
##########
@@ -78,14 +78,14 @@ public interface MembershipManager extends RequestManager {
/**
* Notify the member that an error heartbeat response was received.
*/
- void onHeartbeatFailure();
+ void onHeartbeatFailure(boolean retriable);
Review Comment:
From my understanding of the code, the `retriable` flag here only affects whether or not we update metrics. Is that correct? I'm not suggesting we change the flag name, it's just a little confusing (to me) from an API standpoint. Perhaps some JavaDoc to explain what behavior change that flag affects?
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java:
##########
@@ -459,8 +464,30 @@ public void onHeartbeatSuccess(ConsumerGroupHeartbeatResponseData response) {
}
@Override
- public void onHeartbeatFailure() {
- metricsManager.maybeRecordRebalanceFailed();
+ public void onHeartbeatFailure(boolean retriable) {
+ if (!retriable) {
+ metricsManager.maybeRecordRebalanceFailed();
+ }
+ // The leave group request is sent out once (not retried), so we should complete the leave
+ // operation once the request completes, regardless of the response.
+ if (state == MemberState.UNSUBSCRIBED && maybeCompleteLeaveInProgress()) {
+ log.warn("Member {} with epoch {} received a failed response to the heartbeat to " +
+ "leave the group and completed the leave operation. ", memberId, memberEpoch);
+ }
+ }
+
+ /**
+ * Complete the leave in progress (if any), if the member is UNSUBSCRIBED. This is expected to
Review Comment:
Not to nitpick, but technically the `maybeCompleteLeaveInProgress()` doesn't check/enforce anything to do with the current `MemberState`. Should there be a quick sanity check that the state is `UNSUBSCRIBED` inside `maybeCompleteLeaveInProgress()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16984: Complete consumer leave on response to leave request [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru merged PR #16569:
URL: https://github.com/apache/kafka/pull/16569
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16984: Complete consumer leave on response to leave request [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #16569:
URL: https://github.com/apache/kafka/pull/16569#issuecomment-2237430835
Thanks for the review @lucasbru and @kirktrue . All comments addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16984: Complete consumer leave on response to leave request [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16569:
URL: https://github.com/apache/kafka/pull/16569#discussion_r1683127995
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImplTest.java:
##########
@@ -166,33 +167,13 @@ public void testMembershipManagerInitSupportsEmptyGroupInstanceId() {
createMembershipManagerJoiningGroup();
}
- @Test
- public void testMembershipManagerRegistersForClusterMetadataUpdatesOnFirstJoin() {
Review Comment:
we're partners in crime here :) I totally missed this in reviews
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16984: Complete consumer leave on response to leave request [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16569:
URL: https://github.com/apache/kafka/pull/16569#discussion_r1676141611
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImplTest.java:
##########
@@ -166,33 +167,13 @@ public void testMembershipManagerInitSupportsEmptyGroupInstanceId() {
createMembershipManagerJoiningGroup();
}
- @Test
- public void testMembershipManagerRegistersForClusterMetadataUpdatesOnFirstJoin() {
Review Comment:
this is just small clean-up unrelated to this PR (but I found it while on it). This test does not apply anymore, and it passes only because it's not really testing what its name suggest.
The `membershipMgr` did register for metadata updates at some point, but it stopped doing it the moment it was turned into a manager that is polled on a regular basis to trigger reconciliations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16984: Complete consumer leave on response to leave request [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #16569:
URL: https://github.com/apache/kafka/pull/16569#discussion_r1682463147
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java:
##########
@@ -459,8 +464,30 @@ public void onHeartbeatSuccess(ConsumerGroupHeartbeatResponseData response) {
}
@Override
- public void onHeartbeatFailure() {
- metricsManager.maybeRecordRebalanceFailed();
+ public void onHeartbeatFailure(boolean retriable) {
+ if (!retriable) {
+ metricsManager.maybeRecordRebalanceFailed();
+ }
+ // The leave group request is sent out once (not retried), so we should complete the leave
+ // operation once the request completes, regardless of the response.
+ if (state == MemberState.UNSUBSCRIBED && maybeCompleteLeaveInProgress()) {
+ log.warn("Member {} with epoch {} received a failed response to the heartbeat to " +
+ "leave the group and completed the leave operation. ", memberId, memberEpoch);
+ }
+ }
+
+ /**
+ * Complete the leave in progress (if any), if the member is UNSUBSCRIBED. This is expected to
Review Comment:
I think you may want to fix the Javadoc then, as it specifically mentions the member needs to be `UNSUBSCRIBED`.
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImplTest.java:
##########
@@ -166,33 +167,13 @@ public void testMembershipManagerInitSupportsEmptyGroupInstanceId() {
createMembershipManagerJoiningGroup();
}
- @Test
- public void testMembershipManagerRegistersForClusterMetadataUpdatesOnFirstJoin() {
Review Comment:
Oops, that's on me I suppose
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java:
##########
@@ -587,6 +617,14 @@ public void transitionToFatal() {
return;
}
+ if (previousState == MemberState.LEAVING || previousState == MemberState.PREPARE_LEAVING) {
+ log.info("Member {} with epoch {} was leaving the group with state {} when it got a " +
+ "fatal error from the broker. It will discard the ongoing leave and remain in {} " +
+ "state.", memberId, memberEpoch, previousState, state);
+ maybeCompleteLeaveInProgress();
Review Comment:
nit: I think `state` will always be `MemberState.FATAL` here, so I would use it directly (as in the log messages above)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org