You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "frankvicky (via GitHub)" <gi...@apache.org> on 2024/08/09 07:09:35 UTC
[PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
frankvicky opened a new pull request, #16844:
URL: https://github.com/apache/kafka/pull/16844
JIRA: [KAFKA-17293](https://issues.apache.org/jira/browse/KAFKA-17293)
Ensure that the new consumer will rediscover the coordinator on disconnect and send heartbeats to the correct coordinator.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #16844:
URL: https://github.com/apache/kafka/pull/16844#issuecomment-2313175166
> I wonder if we should also do the same on timeouts. Did we in the old implementation?
Good point @dajac . It's not explicitly handled in the classic or new consumer, but we do get the behaviour in both, because timeouts as treated as disconnections at the network layer level ([handledTimedoutRequests](https://github.com/apache/kafka/blob/62dc982ce9ce76b299a628d50648e3529fd88e63/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L866)). So if the request times out, the network client disconnects from the node, ending up in the `DisconnectExeption` that both consumer handle to mark the coordinator unknown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1731589122
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManagerTest.java:
##########
@@ -380,6 +381,30 @@ public void testNetworkTimeout() {
assertEquals(1, result.unsentRequests.size());
}
+ @Test
+ public void testDisconnect() {
+ // The initial heartbeatInterval is set to 0
+ createHeartbeatRequestStateWithZeroHeartbeatInterval();
+ NetworkClientDelegate.PollResult result = heartbeatRequestManager.poll(time.milliseconds());
+ assertEquals(1, result.unsentRequests.size());
+ // Mimic disconnect
+ result.unsentRequests.get(0).handler().onFailure(time.milliseconds(), DisconnectException.INSTANCE);
+ verify(membershipManager).onHeartbeatFailure(true);
+ // Ensure that the coordinatorManager rediscovers the coordinator
+ verify(coordinatorRequestManager).markCoordinatorUnknown(any(), anyLong());
+ verify(backgroundEventHandler, never()).add(any());
+
+ // Assure the manager will backoff on disconnect
+ time.sleep(DEFAULT_RETRY_BACKOFF_MS - 1);
+ result = heartbeatRequestManager.poll(time.milliseconds());
+ assertEquals(0, result.unsentRequests.size());
+
+ // Assure that the backoff time has been exceeded
Review Comment:
same...
`assertEquals(1, result.unsentRequests.size(), "A new request should be generated after the backoff expires");`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1732103865
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManagerTest.java:
##########
@@ -380,6 +381,30 @@ public void testNetworkTimeout() {
assertEquals(1, result.unsentRequests.size());
}
+ @Test
+ public void testDisconnect() {
+ // The initial heartbeatInterval is set to 0
+ createHeartbeatRequestStateWithZeroHeartbeatInterval();
+ NetworkClientDelegate.PollResult result = heartbeatRequestManager.poll(time.milliseconds());
+ assertEquals(1, result.unsentRequests.size());
+ // Mimic disconnect
+ result.unsentRequests.get(0).handler().onFailure(time.milliseconds(), DisconnectException.INSTANCE);
+ verify(membershipManager).onHeartbeatFailure(true);
+ // Ensure that the coordinatorManager rediscovers the coordinator
+ verify(coordinatorRequestManager).markCoordinatorUnknown(any(), anyLong());
+ verify(backgroundEventHandler, never()).add(any());
+
+ // Assure the manager will backoff on disconnect
Review Comment:
Cool, this way we can provide more clean indicate for our test code. 👍🏼
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1735483028
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ConsumerHeartbeatRequestManagerTest.java:
##########
@@ -380,6 +381,32 @@ public void testNetworkTimeout() {
assertEquals(1, result.unsentRequests.size());
}
+ @Test
+ public void testDisconnect() {
+ // The initial heartbeatInterval is set to 0
Review Comment:
Make sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1731587740
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManagerTest.java:
##########
@@ -380,6 +381,30 @@ public void testNetworkTimeout() {
assertEquals(1, result.unsentRequests.size());
}
+ @Test
+ public void testDisconnect() {
+ // The initial heartbeatInterval is set to 0
+ createHeartbeatRequestStateWithZeroHeartbeatInterval();
+ NetworkClientDelegate.PollResult result = heartbeatRequestManager.poll(time.milliseconds());
+ assertEquals(1, result.unsentRequests.size());
+ // Mimic disconnect
+ result.unsentRequests.get(0).handler().onFailure(time.milliseconds(), DisconnectException.INSTANCE);
+ verify(membershipManager).onHeartbeatFailure(true);
+ // Ensure that the coordinatorManager rediscovers the coordinator
+ verify(coordinatorRequestManager).markCoordinatorUnknown(any(), anyLong());
+ verify(backgroundEventHandler, never()).add(any());
+
+ // Assure the manager will backoff on disconnect
Review Comment:
nit: what about we add an assertion msg instead of an inline comment? I find the code would be cleaner, and this would be more helpful if ever the test fails: `assertEquals(0, result.unsentRequests.size(), "No request should be generated before the backoff expires");`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1735482852
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ConsumerHeartbeatRequestManagerTest.java:
##########
@@ -380,6 +381,32 @@ public void testNetworkTimeout() {
assertEquals(1, result.unsentRequests.size());
}
+ @Test
+ public void testDisconnect() {
+ // The initial heartbeatInterval is set to 0
+ createHeartbeatRequestStateWithZeroHeartbeatInterval();
+ NetworkClientDelegate.PollResult result = heartbeatRequestManager.poll(time.milliseconds());
+ assertEquals(1, result.unsentRequests.size());
+ // Mimic disconnect
+ result.unsentRequests.get(0).handler().onFailure(time.milliseconds(), DisconnectException.INSTANCE);
+ verify(membershipManager).onHeartbeatFailure(true);
+ // Ensure that the coordinatorManager rediscovers the coordinator
+ verify(coordinatorRequestManager).handleCoordinatorDisconnect(any(), anyLong());
+ verify(backgroundEventHandler, never()).add(any());
+
+ time.sleep(DEFAULT_RETRY_BACKOFF_MS - 1);
+ result = heartbeatRequestManager.poll(time.milliseconds());
+ assertEquals(0,
+ result.unsentRequests.size(),
Review Comment:
Yep.
I originally thought it had three arguments, so I split the line, but now it looks ugly. 😬
I will combine it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm merged PR #16844:
URL: https://github.com/apache/kafka/pull/16844
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1732103865
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManagerTest.java:
##########
@@ -380,6 +381,30 @@ public void testNetworkTimeout() {
assertEquals(1, result.unsentRequests.size());
}
+ @Test
+ public void testDisconnect() {
+ // The initial heartbeatInterval is set to 0
+ createHeartbeatRequestStateWithZeroHeartbeatInterval();
+ NetworkClientDelegate.PollResult result = heartbeatRequestManager.poll(time.milliseconds());
+ assertEquals(1, result.unsentRequests.size());
+ // Mimic disconnect
+ result.unsentRequests.get(0).handler().onFailure(time.milliseconds(), DisconnectException.INSTANCE);
+ verify(membershipManager).onHeartbeatFailure(true);
+ // Ensure that the coordinatorManager rediscovers the coordinator
+ verify(coordinatorRequestManager).markCoordinatorUnknown(any(), anyLong());
+ verify(backgroundEventHandler, never()).add(any());
+
+ // Assure the manager will backoff on disconnect
Review Comment:
Cool, this way we can provide stronger protection for our code. 👍🏼
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1738393452
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -122,6 +123,23 @@ NetworkClientDelegate.UnsentRequest makeFindCoordinatorRequest(final long curren
});
}
+ /**
+ * Handles the disconnection of the current coordinator.
+ * This method checks if the given exception is an instance of {@link DisconnectException}.
+ * If so, it marks the coordinator as unknown, indicating that the client should
+ * attempt to discover a new coordinator. For any other exception type, no action is performed.
+ *
+ * @param exception The exception to handle, which was received as part of a request response.
+ * If this is an instance of {@link DisconnectException}, the coordinator is marked as unknown.
+ * For other types of exceptions, no action is performed.
Review Comment:
Yes, it's a little verbose. I will remove it 👍🏼
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1729989299
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManagerTest.java:
##########
@@ -380,6 +381,30 @@ public void testNetworkTimeout() {
assertEquals(1, result.unsentRequests.size());
}
+ @Test
+ public void testDisconnect() {
Review Comment:
I think retry is what we truly expect, so we might not need to specify it redundantly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #16844:
URL: https://github.com/apache/kafka/pull/16844#issuecomment-2318082974
Hey @frankvicky , thanks for the updates. I left a couple of other minor comments. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1732101168
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManager.java:
##########
@@ -450,6 +452,12 @@ private void handleFatalFailure(Throwable error) {
membershipManager.transitionToFatal();
}
+ private void handleCoordinatorDisconnect(Throwable exception, long currentTimeMs) {
+ if (exception instanceof DisconnectException) {
+ coordinatorRequestManager.markCoordinatorUnknown(exception.getMessage(), currentTimeMs);
+ }
+ }
Review Comment:
Make sense 😺
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1736044010
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CommitRequestManagerTest.java:
##########
@@ -922,7 +922,11 @@ public void testEnsureBackoffRetryOnOffsetCommitRequestTimeout() {
assertRetryBackOff(commitRequestManager, retryBackoffMs);
}
- private void assertCoordinatorDisconnect() {
+ private void assertCoordinatorDisconnectOnDisconnect() {
Review Comment:
names got a bit confusing with the other func I would say. Would it be clearer if we rename this one to show that it's about assertCoordinatorDisconnectHandling maybe? (it may or may not be a coordinator disconnect, depending on the specific error, but we ensure it's being handled)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1738389019
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -122,6 +123,23 @@ NetworkClientDelegate.UnsentRequest makeFindCoordinatorRequest(final long curren
});
}
+ /**
+ * Handles the disconnection of the current coordinator.
+ * This method checks if the given exception is an instance of {@link DisconnectException}.
+ * If so, it marks the coordinator as unknown, indicating that the client should
+ * attempt to discover a new coordinator. For any other exception type, no action is performed.
+ *
+ * @param exception The exception to handle, which was received as part of a request response.
+ * If this is an instance of {@link DisconnectException}, the coordinator is marked as unknown.
+ * For other types of exceptions, no action is performed.
Review Comment:
nit: this is already said right above, so let's just remove it from here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1731586009
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManager.java:
##########
@@ -450,6 +452,12 @@ private void handleFatalFailure(Throwable error) {
membershipManager.transitionToFatal();
}
+ private void handleCoordinatorDisconnect(Throwable exception, long currentTimeMs) {
+ if (exception instanceof DisconnectException) {
+ coordinatorRequestManager.markCoordinatorUnknown(exception.getMessage(), currentTimeMs);
+ }
+ }
Review Comment:
With this we end up having the exact same logic to `handleCoordinatorDisconnect` repeated on the managers that need it. What about we encapsulate it in the `CoordinatorRequestManager`? Seems sensible for it to be the component who know how to handle the error (and we only update there if ever needed). So we would end up with the managers just calling: `coordinatorRequestManager.handleCoordinatorDisconnect(exception, currentTimeMs)`. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1736462974
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -122,6 +123,22 @@ NetworkClientDelegate.UnsentRequest makeFindCoordinatorRequest(final long curren
});
}
+ /**
+ * Handles the disconnection of the current coordinator.
+ * This method checks if the given exception is an instance of {@link DisconnectException}.
+ * If so, it marks the coordinator as unknown, indicating that the client should
+ * attempt to discover a new coordinator.
+ *
+ * @param exception The exception that caused the coordinator to be marked as unknown.
+ * This is expected to be an instance of {@link DisconnectException}.
Review Comment:
this param description is a bit confusing. This is the error to handle, received in a request response (not really expecting it to be of any type. If it's DisconnectedException, it will mark the coordinator unknown, or no-op for any other error type)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1737655465
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CommitRequestManagerTest.java:
##########
@@ -922,7 +922,11 @@ public void testEnsureBackoffRetryOnOffsetCommitRequestTimeout() {
assertRetryBackOff(commitRequestManager, retryBackoffMs);
}
- private void assertCoordinatorDisconnect() {
+ private void assertCoordinatorDisconnectOnDisconnect() {
Review Comment:
I think `assertCoordinatorDisconnectHandling` is a good one, thanks 😺
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #16844:
URL: https://github.com/apache/kafka/pull/16844#issuecomment-2310730554
Hey @frankvicky , thanks for the nice patch! Looks good overall, just left some comments for consideration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "TaiJuWu (via GitHub)" <gi...@apache.org>.
TaiJuWu commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1729940195
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManagerTest.java:
##########
@@ -380,6 +381,30 @@ public void testNetworkTimeout() {
assertEquals(1, result.unsentRequests.size());
}
+ @Test
+ public void testDisconnect() {
Review Comment:
How about `testDisconnectAndRetry` ?
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManagerTest.java:
##########
@@ -380,6 +381,30 @@ public void testNetworkTimeout() {
assertEquals(1, result.unsentRequests.size());
}
+ @Test
+ public void testDisconnect() {
Review Comment:
Nit: How about `testDisconnectAndRetry` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1734743535
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ConsumerHeartbeatRequestManagerTest.java:
##########
@@ -380,6 +381,32 @@ public void testNetworkTimeout() {
assertEquals(1, result.unsentRequests.size());
}
+ @Test
+ public void testDisconnect() {
+ // The initial heartbeatInterval is set to 0
Review Comment:
I don't see much value in this comment (it's in the func name next line), should we remove it?
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ConsumerHeartbeatRequestManagerTest.java:
##########
@@ -380,6 +381,32 @@ public void testNetworkTimeout() {
assertEquals(1, result.unsentRequests.size());
}
+ @Test
+ public void testDisconnect() {
+ // The initial heartbeatInterval is set to 0
+ createHeartbeatRequestStateWithZeroHeartbeatInterval();
+ NetworkClientDelegate.PollResult result = heartbeatRequestManager.poll(time.milliseconds());
+ assertEquals(1, result.unsentRequests.size());
+ // Mimic disconnect
+ result.unsentRequests.get(0).handler().onFailure(time.milliseconds(), DisconnectException.INSTANCE);
+ verify(membershipManager).onHeartbeatFailure(true);
+ // Ensure that the coordinatorManager rediscovers the coordinator
+ verify(coordinatorRequestManager).handleCoordinatorDisconnect(any(), anyLong());
+ verify(backgroundEventHandler, never()).add(any());
+
+ time.sleep(DEFAULT_RETRY_BACKOFF_MS - 1);
+ result = heartbeatRequestManager.poll(time.milliseconds());
+ assertEquals(0,
+ result.unsentRequests.size(),
Review Comment:
nit: is this intentionally on a separate line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #16844:
URL: https://github.com/apache/kafka/pull/16844#discussion_r1734744326
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ConsumerHeartbeatRequestManagerTest.java:
##########
@@ -380,6 +381,32 @@ public void testNetworkTimeout() {
assertEquals(1, result.unsentRequests.size());
}
+ @Test
+ public void testDisconnect() {
+ // The initial heartbeatInterval is set to 0
+ createHeartbeatRequestStateWithZeroHeartbeatInterval();
+ NetworkClientDelegate.PollResult result = heartbeatRequestManager.poll(time.milliseconds());
+ assertEquals(1, result.unsentRequests.size());
+ // Mimic disconnect
+ result.unsentRequests.get(0).handler().onFailure(time.milliseconds(), DisconnectException.INSTANCE);
+ verify(membershipManager).onHeartbeatFailure(true);
+ // Ensure that the coordinatorManager rediscovers the coordinator
+ verify(coordinatorRequestManager).handleCoordinatorDisconnect(any(), anyLong());
+ verify(backgroundEventHandler, never()).add(any());
+
+ time.sleep(DEFAULT_RETRY_BACKOFF_MS - 1);
+ result = heartbeatRequestManager.poll(time.milliseconds());
+ assertEquals(0,
+ result.unsentRequests.size(),
+ "No request should be generated before the backoff expires");
+
+ time.sleep(1);
+ result = heartbeatRequestManager.poll(time.milliseconds());
+ assertEquals(1,
+ result.unsentRequests.size(),
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #16844:
URL: https://github.com/apache/kafka/pull/16844#issuecomment-2316008226
Hey @frankvicky , thanks for the updates. Left couple of minor comments, and could you please check the test failures on the CommitRequestManagerTest? Makes sense that it probably needs tuning after the refactoring for `handleCoordinatorDisconnect `. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #16844:
URL: https://github.com/apache/kafka/pull/16844#issuecomment-2311721743
@lianetm @frankvicky I wonder if we should also do the same on timeouts. Did we in the old implementation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on PR #16844:
URL: https://github.com/apache/kafka/pull/16844#issuecomment-2311712109
Hi @lianetm
I have addressed the comments, PTAL 😺
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org