You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Mirai1129 (via GitHub)" <gi...@apache.org> on 2025/06/06 14:54:12 UTC
[PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Mirai1129 opened a new pull request, #19914:
URL: https://github.com/apache/kafka/pull/19914
Problem:
When AsyncConsumer is closing, CoordinatorRequestManager stops looking for coordinator by returning EMPTY in poll() method when closing flag is true.
This prevents commitAsync() and other coordinator-dependent operations from completing, causing close() to hang indefinitely.
Solution:
Remove the closing flag check in poll() method of
CoordinatorRequestManager, so it continues to look for coordinator when needed, even during closing state. This ensures pending coordinator-dependent operations can complete during shutdown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2206233011
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,34 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer();
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ consumer.poll(Duration.ofMillis(500));
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), callback);
+ consumer.close(CloseOptions.timeout(Duration.ofMillis(500)));
Review Comment:
yes, +1 to add assert to ensure the elapsed time is not more than timeout
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2205033179
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,36 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ // Try without looking up the coordinator first
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), callback);
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), callback);
Review Comment:
The idea is that when you start the consumer, the coordinator will also be assigned.
To achieve the test scenario, we will need to `poll` to update metadata after the brokers shutdown.
After that, the coordinator would be `Optional.empty`, then we could verify our test case -
To determine if the consumer can fast fail when committing during close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Mirai1129 (via GitHub)" <gi...@apache.org>.
Mirai1129 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2204475248
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,36 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ // Try without looking up the coordinator first
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), callback);
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), callback);
Review Comment:
@TaiJuWu Thanks for the suggestion. Before the comment, I've tried to remove this one of the `commitAsync()`. But when I remove this line. The exception came to `RetriableCommitFailedException` and said "Offset commit failed with a retriable exception. You should retry committing the latest consumed offsets.". That's why I kept this line in the lastest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs as commitAsync never completes when coordinator is missing [kafka]
Posted by "kuoche1712003 (via GitHub)" <gi...@apache.org>.
kuoche1712003 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2266029745
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -178,6 +178,11 @@ public CommitRequestManager(
*/
@Override
public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
+ if (coordinatorRequestManager.coordinator().isEmpty() && closing) {
+ handleClosingWithoutCoordinator();
+ return EMPTY;
+ }
+
// poll when the coordinator node is known and fatal error is not present
if (coordinatorRequestManager.coordinator().isEmpty()) {
pendingRequests.maybeFailOnCoordinatorFatalError();
Review Comment:
Perhaps it could be written like this for better readability:
```suggestion
pendingRequests.maybeFailOnCoordinatorFatalError();
if(closing && pendingRequests.hasUnsentRequests()){
CommitFailedException exception = new CommitFailedException(
"Failed to commit offsets: Coordinator unknown and consumer is closing");
pendingRequests.drainPendingCommits()
.forEach(request -> request.future().completeExceptionally(exception));
}
```
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CommitRequestManagerTest.java:
##########
@@ -1491,6 +1492,31 @@ private static void assertEmptyPendingRequests(CommitRequestManager commitReques
assertTrue(commitRequestManager.pendingRequests.unsentOffsetCommits.isEmpty());
}
+ @Test
+ public void testPollWithFatalErrorDuringCoordinatorIsEmptyAndClosing() {
+ CommitRequestManager commitRequestManager = create(true, 100);
+ when(coordinatorRequestManager.coordinator()).thenReturn(Optional.of(mockedNode));
+
+ Map<TopicPartition, OffsetAndMetadata> offsets = Collections.singletonMap(new TopicPartition("topic", 1),
+ new OffsetAndMetadata(0));
+
+ var commitFuture = commitRequestManager.commitAsync(offsets);
+
+ commitRequestManager.signalClose();
+ when(coordinatorRequestManager.coordinator()).thenReturn(Optional.empty());
+ when(coordinatorRequestManager.fatalError())
+ .thenReturn(Optional.of(new GroupAuthorizationException("Fatal error")));
+
+ assertEquals(NetworkClientDelegate.PollResult.EMPTY, commitRequestManager.poll(200));
+
+ assertTrue(commitFuture.isCompletedExceptionally());
+
+ ExecutionException exception = assertThrows(ExecutionException.class, commitFuture::get);
+
+ assertInstanceOf(GroupAuthorizationException.class, exception.getCause());
+ assertEquals("Fatal error", exception.getCause().getMessage());
Review Comment:
You might consider using this method.
`TestUtils.assertFutureThrows`
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CommitRequestManagerTest.java:
##########
@@ -1491,6 +1492,31 @@ private static void assertEmptyPendingRequests(CommitRequestManager commitReques
assertTrue(commitRequestManager.pendingRequests.unsentOffsetCommits.isEmpty());
}
+ @Test
+ public void testPollWithFatalErrorDuringCoordinatorIsEmptyAndClosing() {
+ CommitRequestManager commitRequestManager = create(true, 100);
+ when(coordinatorRequestManager.coordinator()).thenReturn(Optional.of(mockedNode));
Review Comment:
Would it be necessary to include this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Mirai1129 (via GitHub)" <gi...@apache.org>.
Mirai1129 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2204978263
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,36 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ // Try without looking up the coordinator first
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), callback);
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), callback);
Review Comment:
Got it! Thank you @frankvicky !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Mirai1129 (via GitHub)" <gi...@apache.org>.
Mirai1129 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2204475248
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,36 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ // Try without looking up the coordinator first
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), callback);
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), callback);
Review Comment:
@TaiJuWu Thanks for the suggestion. Before the comment, I've tried to remove one of the `commitAsync()`. But when I remove this line. The exception came to `RetriableCommitFailedException` and said "Offset commit failed with a retriable exception. You should retry committing the latest consumed offsets.". That's why I kept this line in the lastest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Mirai1129 (via GitHub)" <gi...@apache.org>.
Mirai1129 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2204475248
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,36 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ // Try without looking up the coordinator first
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), callback);
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), callback);
Review Comment:
@TaiJuWu Thanks for the suggestion, before the comment. I've tried to remove this one of the `commitAsync()`. But when I remove this line. The exception came to `RetriableCommitFailedException` and said "Offset commit failed with a retriable exception. You should retry committing the latest consumed offsets.". That's why I kept this line in the lastest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2205011315
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,35 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer();
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ // Try without looking up the coordinator first
+ var callback = new CountConsumerCommitCallback();
Review Comment:
Why this callback need this comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "TaiJuWu (via GitHub)" <gi...@apache.org>.
TaiJuWu commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2203680551
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,37 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ sendRecords(producer, tp1, 3, System.currentTimeMillis());
Review Comment:
Do we need to two partition to check the coordinator is not found?
Or just one is enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Mirai1129 (via GitHub)" <gi...@apache.org>.
Mirai1129 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2205042151
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,36 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ // Try without looking up the coordinator first
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), callback);
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), callback);
Review Comment:
Thank you so much for the explanation @frankvicky !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Mirai1129 (via GitHub)" <gi...@apache.org>.
Mirai1129 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2206217684
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,34 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer();
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
Review Comment:
Thank you so much! I'll add `waitFor()` ensure the brokers are down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2132406509
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -99,7 +99,7 @@ public void signalClose() {
*/
@Override
public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
- if (closing || this.coordinator != null)
Review Comment:
Well, we do need to be careful. There is also code to make sure that finding a coordinator does not block the progress of closing. If you start a consumer when there are no running brokers, close needs to complete promptly. If you stop the brokers when a consumer has been running and then attempt a close, it also needs to complete promptly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2206031972
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,34 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer();
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ consumer.poll(Duration.ofMillis(500));
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), callback);
+ consumer.close(CloseOptions.timeout(Duration.ofMillis(500)));
Review Comment:
Is it possible for `close()` to take more than 500 milliseconds? Should we time the method call and add an assert to ensure it's not more than, say, 1000 milliseconds?
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,34 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer();
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
Review Comment:
I wonder if we need to have some sort of `wiatFor()` check after this to ensure the brokers are down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "TaiJuWu (via GitHub)" <gi...@apache.org>.
TaiJuWu commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2204435607
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,36 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
Review Comment:
The default of acks is already `all` so we can remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2203212279
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -178,6 +178,23 @@ public CommitRequestManager(
*/
@Override
public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
+ if (coordinatorRequestManager.coordinator().isEmpty() && closing) {
+ CommitFailedException exception = new CommitFailedException(
+ "Failed to commit offsets: Coordinator unknown and consumer is closing");
+
+ List<NetworkClientDelegate.UnsentRequest> pendingCommits = pendingRequests.drainPendingCommits();
+
+ // Complete each commit request exceptionally
+ pendingCommits.forEach(request -> {
+ if (request.future() != null) {
+ request.future().completeExceptionally(exception);
+ }
Review Comment:
We don't need a null check here.
Ref:
https://github.com/apache/kafka/blob/986322dc3619fc1725012ac33eebed7db8d8cbbb/clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java#L406
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -178,6 +178,23 @@ public CommitRequestManager(
*/
@Override
public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
+ if (coordinatorRequestManager.coordinator().isEmpty() && closing) {
+ CommitFailedException exception = new CommitFailedException(
+ "Failed to commit offsets: Coordinator unknown and consumer is closing");
+
+ List<NetworkClientDelegate.UnsentRequest> pendingCommits = pendingRequests.drainPendingCommits();
Review Comment:
We could inline this variable.
`pendingRequests.drainPendingCommits().forEach...`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Yunyung (via GitHub)" <gi...@apache.org>.
Yunyung commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2206245015
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,34 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer();
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ consumer.poll(Duration.ofMillis(500));
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), callback);
Review Comment:
Should be `tp` not `tp1`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "TaiJuWu (via GitHub)" <gi...@apache.org>.
TaiJuWu commented on PR #19914:
URL: https://github.com/apache/kafka/pull/19914#issuecomment-3083214277
@Mirai1129 could you merge trunk to fix testMultipleRemoteFetchesInOneFetchRequest()?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2213226100
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +453,40 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer();
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ TestUtils.waitForCondition(() -> cluster.aliveBrokers().isEmpty(), "All brokers should be shut down");
+
+ consumer.poll(Duration.ofMillis(500));
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), callback);
+
+ long startTime = System.currentTimeMillis();
+ consumer.close(CloseOptions.timeout(Duration.ofMillis(500)));
Review Comment:
It doesn't add much value to this test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Yunyung (via GitHub)" <gi...@apache.org>.
Yunyung commented on PR #19914:
URL: https://github.com/apache/kafka/pull/19914#issuecomment-3088725406
> causing close() to hang indefinitely.
Not very accurate, it should be "causing close() to hang until timeout".
> Modified the closing flag check in poll() method of
CoordinatorRequestManager to be more targeted:
CoordinatorRequestManager -> CommitRequestManager
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Mirai1129 (via GitHub)" <gi...@apache.org>.
Mirai1129 commented on PR #19914:
URL: https://github.com/apache/kafka/pull/19914#issuecomment-3032226649
Hi everyone,
Thanks for all the feedback and suggestions! And sorry for late reply. I've updated the PR with the timeout handling solution - when `CommitRequestManager` times out, it will now immediately clear remaining callbacks to prevent system deadlock.
I'm currently working on adding comprehensive integration tests to cover this scenario and will push them shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs as commitAsync never completes when coordinator is missing [kafka]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2346790323
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CommitRequestManagerTest.java:
##########
@@ -1494,6 +1494,27 @@ private static void assertEmptyPendingRequests(CommitRequestManager commitReques
assertTrue(commitRequestManager.pendingRequests.unsentOffsetCommits.isEmpty());
}
+ @Test
+ public void testPollWithFatalErrorDuringCoordinatorIsEmptyAndClosing() {
+ CommitRequestManager commitRequestManager = create(true, 100);
+
+ Map<TopicPartition, OffsetAndMetadata> offsets = Collections.singletonMap(new TopicPartition("topic", 1),
+ new OffsetAndMetadata(0));
Review Comment:
```suggestion
Map<TopicPartition, OffsetAndMetadata> offsets = Map.of(new TopicPartition("topic", 1),
new OffsetAndMetadata(0));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs as commitAsync never completes when coordinator is missing [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2217961135
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -178,6 +178,15 @@ public CommitRequestManager(
*/
@Override
public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
+ if (coordinatorRequestManager.coordinator().isEmpty() && closing) {
+ CommitFailedException exception = new CommitFailedException(
Review Comment:
Should we consider propagating the coordinator error that is causing the empty coordinator?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2203217143
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,37 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ @ClusterTest
+ public void testClearBrokerBeforeConsumerCloses() throws InterruptedException {
+ // This is testing when closing the consumer but commit request has already been sent.
+ // During the closing, the consumer won't find the coordinator anymore.
+
+ // Create offsets topic to ensure coordinator is available during close
+ cluster.createTopic(Topic.GROUP_METADATA_TOPIC_NAME, Integer.parseInt(OFFSETS_TOPIC_PARTITIONS), Short.parseShort(OFFSETS_TOPIC_REPLICATION));
+
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ sendRecords(producer, tp1, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp, tp1));
+
+ // Try without looking up the coordinator first
+ var cb = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ Set<Integer> brokerIds = cluster.brokerIds();
+ brokerIds.forEach(cluster::shutdownBroker);
+
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), cb);
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), cb);
+
+ consumer.close();
Review Comment:
Let's give it a shorter duration to close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs as commitAsync never completes when coordinator is missing [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 merged PR #19914:
URL: https://github.com/apache/kafka/pull/19914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "DL1231 (via GitHub)" <gi...@apache.org>.
DL1231 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2132415250
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -99,7 +99,7 @@ public void signalClose() {
*/
@Override
public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
- if (closing || this.coordinator != null)
+ if (this.coordinator != null)
Review Comment:
Thanks for the PR, left a comment.
Should we add an IT for this scenario: when closing the consumer, commitAsync, and verify that the consumer can shut down properly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2203213066
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,37 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ @ClusterTest
+ public void testClearBrokerBeforeConsumerCloses() throws InterruptedException {
+ // This is testing when closing the consumer but commit request has already been sent.
+ // During the closing, the consumer won't find the coordinator anymore.
+
+ // Create offsets topic to ensure coordinator is available during close
+ cluster.createTopic(Topic.GROUP_METADATA_TOPIC_NAME, Integer.parseInt(OFFSETS_TOPIC_PARTITIONS), Short.parseShort(OFFSETS_TOPIC_REPLICATION));
+
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ sendRecords(producer, tp1, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp, tp1));
+
+ // Try without looking up the coordinator first
+ var cb = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ Set<Integer> brokerIds = cluster.brokerIds();
+ brokerIds.forEach(cluster::shutdownBroker);
+
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), cb);
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), cb);
+
+ consumer.close();
+ assertTrue(cb.lastError.get() instanceof CommitFailedException);
+ assertEquals(0, cb.successCount);
Review Comment:
Could you also check the excpetion error message?
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,37 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ @ClusterTest
+ public void testClearBrokerBeforeConsumerCloses() throws InterruptedException {
+ // This is testing when closing the consumer but commit request has already been sent.
+ // During the closing, the consumer won't find the coordinator anymore.
+
+ // Create offsets topic to ensure coordinator is available during close
+ cluster.createTopic(Topic.GROUP_METADATA_TOPIC_NAME, Integer.parseInt(OFFSETS_TOPIC_PARTITIONS), Short.parseShort(OFFSETS_TOPIC_REPLICATION));
+
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ sendRecords(producer, tp1, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp, tp1));
+
+ // Try without looking up the coordinator first
+ var cb = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ Set<Integer> brokerIds = cluster.brokerIds();
+ brokerIds.forEach(cluster::shutdownBroker);
+
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), cb);
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), cb);
+
+ consumer.close();
+ assertTrue(cb.lastError.get() instanceof CommitFailedException);
Review Comment:
```suggestion
assertTrue(cb.lastError.isPresent());
assertEquals(CommitFailedException.class, cb.lastError.get().getClass());
```
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -178,6 +178,23 @@ public CommitRequestManager(
*/
@Override
public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
+ if (coordinatorRequestManager.coordinator().isEmpty() && closing) {
+ CommitFailedException exception = new CommitFailedException(
+ "Failed to commit offsets: Coordinator unknown and consumer is closing");
+
+ List<NetworkClientDelegate.UnsentRequest> pendingCommits = pendingRequests.drainPendingCommits();
+
+ // Complete each commit request exceptionally
+ pendingCommits.forEach(request -> {
+ if (request.future() != null) {
+ request.future().completeExceptionally(exception);
+ }
+ });
Review Comment:
Is there a chance that the future could complete with a null value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Mirai1129 (via GitHub)" <gi...@apache.org>.
Mirai1129 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2203214845
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -178,6 +178,23 @@ public CommitRequestManager(
*/
@Override
public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
+ if (coordinatorRequestManager.coordinator().isEmpty() && closing) {
+ CommitFailedException exception = new CommitFailedException(
+ "Failed to commit offsets: Coordinator unknown and consumer is closing");
+
+ List<NetworkClientDelegate.UnsentRequest> pendingCommits = pendingRequests.drainPendingCommits();
+
+ // Complete each commit request exceptionally
+ pendingCommits.forEach(request -> {
+ if (request.future() != null) {
+ request.future().completeExceptionally(exception);
+ }
Review Comment:
Thanks for the ref, I see. I'll fix it right now!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Mirai1129 (via GitHub)" <gi...@apache.org>.
Mirai1129 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2205020467
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,35 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer();
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ // Try without looking up the coordinator first
+ var callback = new CountConsumerCommitCallback();
Review Comment:
Actually idk why, cuz I just duplicated `testCommitAsyncCompletedBeforeConsumerCloses` and modified it. But if it doesn't need let me remove it 🥹.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs as commitAsync never completes when coordinator is missing [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2366227345
##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CommitRequestManagerTest.java:
##########
@@ -1494,6 +1494,27 @@ private static void assertEmptyPendingRequests(CommitRequestManager commitReques
assertTrue(commitRequestManager.pendingRequests.unsentOffsetCommits.isEmpty());
}
+ @Test
+ public void testPollWithFatalErrorDuringCoordinatorIsEmptyAndClosing() {
Review Comment:
This unit test only covers the path of `coordinatorRequestManager.coordinator().isEmpty() -> pendingRequests.maybeFailOnCoordinatorFatalError`. Could you please add another one to cover the case of `closing && pendingRequests.hasUnsentRequests()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2203213180
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,37 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ @ClusterTest
+ public void testClearBrokerBeforeConsumerCloses() throws InterruptedException {
+ // This is testing when closing the consumer but commit request has already been sent.
+ // During the closing, the consumer won't find the coordinator anymore.
+
+ // Create offsets topic to ensure coordinator is available during close
+ cluster.createTopic(Topic.GROUP_METADATA_TOPIC_NAME, Integer.parseInt(OFFSETS_TOPIC_PARTITIONS), Short.parseShort(OFFSETS_TOPIC_REPLICATION));
+
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ sendRecords(producer, tp1, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp, tp1));
+
+ // Try without looking up the coordinator first
+ var cb = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ Set<Integer> brokerIds = cluster.brokerIds();
+ brokerIds.forEach(cluster::shutdownBroker);
Review Comment:
```suggestion
cluster.brokerIds().forEach(cluster::shutdownBroker);
```
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,37 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ @ClusterTest
+ public void testClearBrokerBeforeConsumerCloses() throws InterruptedException {
+ // This is testing when closing the consumer but commit request has already been sent.
+ // During the closing, the consumer won't find the coordinator anymore.
+
+ // Create offsets topic to ensure coordinator is available during close
+ cluster.createTopic(Topic.GROUP_METADATA_TOPIC_NAME, Integer.parseInt(OFFSETS_TOPIC_PARTITIONS), Short.parseShort(OFFSETS_TOPIC_REPLICATION));
Review Comment:
We don't need this since the top-level annotation has already created it.
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,37 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ @ClusterTest
+ public void testClearBrokerBeforeConsumerCloses() throws InterruptedException {
+ // This is testing when closing the consumer but commit request has already been sent.
+ // During the closing, the consumer won't find the coordinator anymore.
Review Comment:
Please change it to the method-level javadoc.
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,37 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ @ClusterTest
+ public void testClearBrokerBeforeConsumerCloses() throws InterruptedException {
Review Comment:
This test name is not accurate.
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,37 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ @ClusterTest
+ public void testClearBrokerBeforeConsumerCloses() throws InterruptedException {
+ // This is testing when closing the consumer but commit request has already been sent.
+ // During the closing, the consumer won't find the coordinator anymore.
+
+ // Create offsets topic to ensure coordinator is available during close
+ cluster.createTopic(Topic.GROUP_METADATA_TOPIC_NAME, Integer.parseInt(OFFSETS_TOPIC_PARTITIONS), Short.parseShort(OFFSETS_TOPIC_REPLICATION));
+
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ sendRecords(producer, tp1, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp, tp1));
+
+ // Try without looking up the coordinator first
+ var cb = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ Set<Integer> brokerIds = cluster.brokerIds();
+ brokerIds.forEach(cluster::shutdownBroker);
+
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), cb);
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), cb);
+
+ consumer.close();
+ assertTrue(cb.lastError.get() instanceof CommitFailedException);
Review Comment:
```suggestion
assertTrue(cb.lastError.isPresent());
assertInstanceOf(CommitFailedException.class, cb.lastError.get());
```
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,37 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ @ClusterTest
+ public void testClearBrokerBeforeConsumerCloses() throws InterruptedException {
+ // This is testing when closing the consumer but commit request has already been sent.
+ // During the closing, the consumer won't find the coordinator anymore.
+
+ // Create offsets topic to ensure coordinator is available during close
+ cluster.createTopic(Topic.GROUP_METADATA_TOPIC_NAME, Integer.parseInt(OFFSETS_TOPIC_PARTITIONS), Short.parseShort(OFFSETS_TOPIC_REPLICATION));
+
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ sendRecords(producer, tp1, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp, tp1));
+
+ // Try without looking up the coordinator first
+ var cb = new CountConsumerCommitCallback();
Review Comment:
nit:
```suggestion
var callback = new CountConsumerCommitCallback();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2205033179
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,36 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ // Try without looking up the coordinator first
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), callback);
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), callback);
Review Comment:
The idea is that when you assign the topicPartition to the consumer, the coordinator will also be assigned.
To achieve the test scenario, we will need to `poll` again to update metadata after the brokers shutdown.
After that, the coordinator would be `Optional.empty`, then we could verify our test case -
To determine if the consumer can fast fail when committing during close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Mirai1129 (via GitHub)" <gi...@apache.org>.
Mirai1129 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2205020467
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,35 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer();
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ // Try without looking up the coordinator first
+ var callback = new CountConsumerCommitCallback();
Review Comment:
Actually idk why, cuz I just duplicate `testCommitAsyncCompletedBeforeConsumerCloses` and modified it. But if it doesn't need let me remove it 🥹.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Mirai1129 (via GitHub)" <gi...@apache.org>.
Mirai1129 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2204982828
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,36 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
Review Comment:
I see, thank you so much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Yunyung (via GitHub)" <gi...@apache.org>.
Yunyung commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2213147485
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +453,40 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer();
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ TestUtils.waitForCondition(() -> cluster.aliveBrokers().isEmpty(), "All brokers should be shut down");
+
+ consumer.poll(Duration.ofMillis(500));
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), callback);
+
+ long startTime = System.currentTimeMillis();
+ consumer.close(CloseOptions.timeout(Duration.ofMillis(500)));
Review Comment:
How about
```suggestion
assertDoesNotThrow(() -> consumer.close(CloseOptions.timeout(Duration.ofMillis(1000))));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs as commitAsync never completes when coordinator is missing [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #19914:
URL: https://github.com/apache/kafka/pull/19914#issuecomment-3311655441
@Mirai1129 could you please rebase code to run QA again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2205033179
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,36 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ // Try without looking up the coordinator first
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), callback);
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), callback);
Review Comment:
The idea is that when you start the consumer, the coordinator will also be assigned.
To achieve the test scenario, we will need to `poll` again to update metadata after the brokers shutdown.
After that, the coordinator would be `Optional.empty`, then we could verify our test case -
To determine if the consumer can fast fail when committing during close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "TaiJuWu (via GitHub)" <gi...@apache.org>.
TaiJuWu commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2204435607
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,36 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
Review Comment:
The default of ack is already `all` so we can remove it.
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,36 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ // Try without looking up the coordinator first
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), callback);
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), callback);
Review Comment:
This line can be removed too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2204834123
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -178,6 +178,16 @@ public CommitRequestManager(
*/
@Override
public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
+ if (coordinatorRequestManager.coordinator().isEmpty() && closing) {
+ CommitFailedException exception = new CommitFailedException(
+ "Failed to commit offsets: Coordinator unknown and consumer is closing");
+
+ // Complete each commit request exceptionally
Review Comment:
We could remove this comment.
It doesn't provide additional value.
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,36 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer(Map.of(ProducerConfig.ACKS_CONFIG, "all"));
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ // Try without looking up the coordinator first
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ consumer.commitAsync(Map.of(tp, new OffsetAndMetadata(1L)), callback);
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), callback);
Review Comment:
Your consumer should at least `poll` once before you commit.
This will initiate the heartbeat, establish a connection with the broker, and update the metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs as commitAsync never completes when coordinator is missing [kafka]
Posted by "Mirai1129 (via GitHub)" <gi...@apache.org>.
Mirai1129 commented on PR #19914:
URL: https://github.com/apache/kafka/pull/19914#issuecomment-3319631803
@chia7712 Thanks for the comment, I have added new test in the latest commit!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Mirai1129 (via GitHub)" <gi...@apache.org>.
Mirai1129 commented on PR #19914:
URL: https://github.com/apache/kafka/pull/19914#issuecomment-3088846675
@Yunyung Thank you so much, I have updated the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Mirai1129 (via GitHub)" <gi...@apache.org>.
Mirai1129 commented on PR #19914:
URL: https://github.com/apache/kafka/pull/19914#issuecomment-3084346703
@Yunyung I've updated the solution! Could you please take a look at the changes when you have a moment? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "Mirai1129 (via GitHub)" <gi...@apache.org>.
Mirai1129 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2206219409
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCommitTest.java:
##########
@@ -452,6 +452,34 @@ private void testPositionAndCommit(GroupProtocol groupProtocol) throws Interrupt
}
}
+ /**
+ * This is testing when closing the consumer but commit request has already been sent.
+ * During the closing, the consumer won't find the coordinator anymore.
+ */
+ @ClusterTest
+ public void testCommitAsyncFailsWhenCoordinatorUnavailableDuringClose() throws InterruptedException {
+ try (Producer<byte[], byte[]> producer = cluster.producer();
+ var consumer = createConsumer(GroupProtocol.CONSUMER, false)
+ ) {
+ sendRecords(producer, tp, 3, System.currentTimeMillis());
+ consumer.assign(List.of(tp));
+
+ var callback = new CountConsumerCommitCallback();
+
+ // Close the coordinator before committing because otherwise the commit will fail to find the coordinator.
+ cluster.brokerIds().forEach(cluster::shutdownBroker);
+
+ consumer.poll(Duration.ofMillis(500));
+ consumer.commitAsync(Map.of(tp1, new OffsetAndMetadata(1L)), callback);
+ consumer.close(CloseOptions.timeout(Duration.ofMillis(500)));
Review Comment:
Is this change to avoid the test being incorrect due to the shutdown taking too long then we add an assert to ensure that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19357: AsyncConsumer#close hangs during closing because the commitAsync request never completes due to a missing coordinator [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19914:
URL: https://github.com/apache/kafka/pull/19914#discussion_r2132367537
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -99,7 +99,7 @@ public void signalClose() {
*/
@Override
public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
- if (closing || this.coordinator != null)
Review Comment:
This makes the consumer find the coordinator during closing, right? If the consumer doesn't have a coordinator running, does it make sense to find one during closing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org