You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "m1a2st (via GitHub)" <gi...@apache.org> on 2024/12/05 08:54:45 UTC

[PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

m1a2st opened a new pull request, #18050:
URL: https://github.com/apache/kafka/pull/18050

   Jira: https://issues.apache.org/jira/browse/KAFKA-18034
   
   In (KAFKA-17696)[https://github.com/apache/kafka/pull/17440] we will remove ErrorEvent to pass the metadata error, In this PR we also want to fix that passing the fatal event without using ErrorEvent
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #18050:
URL: https://github.com/apache/kafka/pull/18050#issuecomment-2546167787

   Regarding the concern with the interaction between the request managers,  I totally see how in some cases a 3rd component can help decouple, but that becomes more interesting when we have 2-way interactions, mostly because it also introduces state to maintain and another player in the architecture, so better not to pay that price if not really needed?
   
   The interaction here seems fairly simple to me (1 way only): `coordinatorMgr` exposes coordinator node and fatal error to others (so HB/Commit mgr use it). Also, the dependency already exists before this PR, to access the coordinator node, we're just extending it to access fatal errors.
   
   Those are my first thoughts on that, but of course, happy to hear more about the tracker idea if we think it's a fair trade-off here and I'm not seeing it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1911784734


##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManagerTest.java:
##########
@@ -253,6 +236,21 @@ public void testNetworkTimeout() {
         res2 = coordinatorManager.poll(time.milliseconds());
         assertEquals(1, res2.unsentRequests.size());
     }
+    
+    @Test

Review Comment:
   It's a good idea, I will address it :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18050:
URL: https://github.com/apache/kafka/pull/18050#issuecomment-2591655707

   Note that it said the PR timed out.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1905871090


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -196,6 +198,14 @@ public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
         return new NetworkClientDelegate.PollResult(timeUntilNextPoll, requests);
     }
 
+    private void maybeFailPendingRequestsOnCoordinatorFatalError() {
+        Optional<Throwable> fatalError = coordinatorRequestManager.fatalError();
+        if (fatalError.isPresent()) {
+            pendingRequests.unsentOffsetCommits.forEach(request -> request.future.completeExceptionally(fatalError.get()));
+            pendingRequests.unsentOffsetFetches.forEach(request -> request.future.completeExceptionally(fatalError.get()));

Review Comment:
   Agree, the `clearAll` was only being called from poll, when draining the requests to send https://github.com/apache/kafka/blob/bc3a18c59c638f267d9ebe28105be6b94f659f4f/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java#L193 
   
   If there was a fatal coordinator error, the poll does an early return so we would never call `clearAll` and the failed requests would forever remain in the queue I expect (even after expiration even, because the early return wouldn't even allow to make it to the `failAndRemoveExpiredCommitRequests`). Makes sense @kirktrue? (ln 207 doesn't have a clearAll so just want to make sure we're looking at the same things) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1885087039


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -176,9 +176,15 @@ public CommitRequestManager(
      */
     @Override
     public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
-        // poll only when the coordinator node is known.
-        if (coordinatorRequestManager.coordinator().isEmpty())
+        // poll when the coordinator node is known and fatal errors are not present
+        if (coordinatorRequestManager.coordinator().isEmpty()) {
+            Optional<Throwable> fatalError = coordinatorRequestManager.fatalError();
+            if (fatalError.isPresent()) {
+                pendingRequests.unsentOffsetCommits.forEach(request -> request.future.completeExceptionally(fatalError.get()));
+                pendingRequests.unsentOffsetFetches.forEach(request -> request.future.completeExceptionally(fatalError.get()));
+            }
             return EMPTY;

Review Comment:
   > The CommitRequestManager does not clear the fatal error, but the AbstractHeartbeatRequestManager does. Can you explain and/or add some comments so that others can understand the distinction between these two cases?
   
   I addressed some comments on this, If you have other question, Please let me know.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1885085674


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -200,12 +196,12 @@ private void onFailedResponse(final long currentTimeMs, final Throwable exceptio
         if (exception == Errors.GROUP_AUTHORIZATION_FAILED.exception()) {
             log.debug("FindCoordinator request failed due to authorization error {}", exception.getMessage());
             KafkaException groupAuthorizationException = GroupAuthorizationException.forGroupId(this.groupId);
-            backgroundEventHandler.add(new ErrorEvent(groupAuthorizationException));
+            fatalError = Optional.of(groupAuthorizationException);

Review Comment:
   I think all cases are using the `RequestManager#poll` in `ConsumerNetworkThread#runOnce`, so Im not having any edge case on overwritten the fata error now. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on PR #18050:
URL: https://github.com/apache/kafka/pull/18050#issuecomment-2582785918

   Hello @lianetm, sorry for the late response, Im thinking sbout this [comment](https://github.com/apache/kafka/pull/18050#discussion_r1907514364), I will address this comment ASAP, Thanks for your patience :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on PR #18050:
URL: https://github.com/apache/kafka/pull/18050#issuecomment-2582994673

   Hello @lianetm, if you have free cycle, PTAL, Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1905777105


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -196,6 +198,14 @@ public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
         return new NetworkClientDelegate.PollResult(timeUntilNextPoll, requests);
     }
 
+    private void maybeFailPendingRequestsOnCoordinatorFatalError() {
+        Optional<Throwable> fatalError = coordinatorRequestManager.fatalError();
+        if (fatalError.isPresent()) {
+            pendingRequests.unsentOffsetCommits.forEach(request -> request.future.completeExceptionally(fatalError.get()));
+            pendingRequests.unsentOffsetFetches.forEach(request -> request.future.completeExceptionally(fatalError.get()));

Review Comment:
   I addressed the clearAll() method after @lianetm’s comments. It makes sense to clear all pendingRequests once we’ve completed all unused requests and fetches.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1907490954


##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManagerTest.java:
##########
@@ -191,28 +187,15 @@ public void testBackoffAfterRetriableFailure() {
     }
 
     @Test
-    public void testPropagateAndBackoffAfterFatalError() {
+    public void testBackoffAfterFatalError() {
         CoordinatorRequestManager coordinatorManager = setupCoordinatorManager(GROUP_ID);
         expectFindCoordinatorRequest(coordinatorManager, Errors.GROUP_AUTHORIZATION_FAILED);
 
-        verify(backgroundEventHandler).add(argThat(backgroundEvent -> {
-            if (!(backgroundEvent instanceof ErrorEvent))
-                return false;
-
-            RuntimeException exception = ((ErrorEvent) backgroundEvent).error();
-
-            if (!(exception instanceof GroupAuthorizationException))
-                return false;
-
-            GroupAuthorizationException groupAuthException = (GroupAuthorizationException) exception;
-            return groupAuthException.groupId().equals(GROUP_ID);
-        }));
-
         time.sleep(RETRY_BACKOFF_MS - 1);
-        assertEquals(Collections.emptyList(), coordinatorManager.poll(time.milliseconds()).unsentRequests);

Review Comment:
   I would expect we should keep this line to check that no request is generated if the backoff hasn't expired. 



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManagerTest.java:
##########
@@ -191,28 +187,15 @@ public void testBackoffAfterRetriableFailure() {
     }
 
     @Test
-    public void testPropagateAndBackoffAfterFatalError() {
+    public void testBackoffAfterFatalError() {
         CoordinatorRequestManager coordinatorManager = setupCoordinatorManager(GROUP_ID);
         expectFindCoordinatorRequest(coordinatorManager, Errors.GROUP_AUTHORIZATION_FAILED);
 
-        verify(backgroundEventHandler).add(argThat(backgroundEvent -> {
-            if (!(backgroundEvent instanceof ErrorEvent))
-                return false;
-
-            RuntimeException exception = ((ErrorEvent) backgroundEvent).error();
-
-            if (!(exception instanceof GroupAuthorizationException))
-                return false;
-
-            GroupAuthorizationException groupAuthException = (GroupAuthorizationException) exception;
-            return groupAuthException.groupId().equals(GROUP_ID);
-        }));
-
         time.sleep(RETRY_BACKOFF_MS - 1);
-        assertEquals(Collections.emptyList(), coordinatorManager.poll(time.milliseconds()).unsentRequests);
 
         time.sleep(1);
         assertEquals(1, coordinatorManager.poll(time.milliseconds()).unsentRequests.size());
+        assertEquals(0, coordinatorManager.poll(time.milliseconds()).unsentRequests.size());

Review Comment:
   if we restore the line from the comment above seems we don't need to add this, right?  



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -244,4 +246,12 @@ private void onResponse(
     public Optional<Node> coordinator() {
         return Optional.ofNullable(this.coordinator);
     }
+    
+    public void clearFatalError() {

Review Comment:
   should this be private? we only call it from this same component now



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -186,6 +187,7 @@ private void onSuccessfulResponse(
                 coordinator.port());
         log.info("Discovered group coordinator {}", coordinator);
         coordinatorRequestState.onSuccessfulAttempt(currentTimeMs);
+        clearFatalError();

Review Comment:
   thinking more about this call here, wonder if it may not be enough and should be higher up in the call stack of the response. Ex. in the case of fatal error and then a retriable error, we would not clear the fatal error right? If we want to ensure we always clear/reset the fatal error, maybe we should move it to the first thing to do when we receive a response
   https://github.com/apache/kafka/blob/624dd458099fa93b3fa1e1715b58bbc6d8689857/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java#L116
   
   From there on, it will be set again if there is again a fatal error, or left clear if there is not (successful response, non-fatal error). Makes sense? 
   
   Also, could we add a unit test to cover this, checking how the coordinatorReqMgr clears/reset the fatal error when receiving a successful response, retriable error, or different fatal error? 



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CommitRequestManagerTest.java:
##########
@@ -1498,6 +1498,23 @@ public void testSignalClose() {
         OffsetCommitRequestData data = (OffsetCommitRequestData) res.unsentRequests.get(0).requestBuilder().build().data();
         assertEquals("topic", data.topics().get(0).name());
     }
+    
+    @Test
+    public void testPollWithFatalErrorShouldFailingAllUnsentRequest() {

Review Comment:
   ```suggestion
       public void testPollWithFatalErrorShouldFailAllUnsentRequests() {
   ```



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -1246,6 +1248,16 @@ private List<NetworkClientDelegate.UnsentRequest> drainPendingCommits() {
             clearAll();
             return res;
         }
+
+        private void maybeFailCoordinatorFatalError() {
+            Optional<Throwable> fatalError = coordinatorRequestManager.fatalError();
+            if (fatalError.isPresent()) {

Review Comment:
   ```suggestion
               coordinatorRequestManager.fatalError().ifPresent(throwable -> {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1905888150


##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManagerTest.java:
##########
@@ -194,25 +190,11 @@ public void testBackoffAfterRetriableFailure() {
     public void testPropagateAndBackoffAfterFatalError() {
         CoordinatorRequestManager coordinatorManager = setupCoordinatorManager(GROUP_ID);
         expectFindCoordinatorRequest(coordinatorManager, Errors.GROUP_AUTHORIZATION_FAILED);
-
-        verify(backgroundEventHandler).add(argThat(backgroundEvent -> {
-            if (!(backgroundEvent instanceof ErrorEvent))
-                return false;
-
-            RuntimeException exception = ((ErrorEvent) backgroundEvent).error();
-
-            if (!(exception instanceof GroupAuthorizationException))
-                return false;
-
-            GroupAuthorizationException groupAuthException = (GroupAuthorizationException) exception;
-            return groupAuthException.groupId().equals(GROUP_ID);
-        }));
-
-        time.sleep(RETRY_BACKOFF_MS - 1);

Review Comment:
   I think we should keep this line, and the we don't need to change the assertion below (I would expect the only change in this test is to remove the verify). Also, should we rename it to `testBackoffAfterFatalError`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm merged PR #18050:
URL: https://github.com/apache/kafka/pull/18050


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #18050:
URL: https://github.com/apache/kafka/pull/18050#issuecomment-2587560968

   Cherry-picked to 4.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #18050:
URL: https://github.com/apache/kafka/pull/18050#issuecomment-2582769412

   Hey @m1a2st, just close follow-up on this one given how close we are to the code freeze. There are just 3 comments left ([here](https://github.com/apache/kafka/pull/18050#pullrequestreview-2537671320)), please take a look when you have a chance, let me know if any question/concern (also please merge trunk latest changes). Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1881596024


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -188,12 +184,12 @@ private void onFailedResponse(final long currentTimeMs, final Throwable exceptio
         if (exception == Errors.GROUP_AUTHORIZATION_FAILED.exception()) {
             log.debug("FindCoordinator request failed due to authorization error {}", exception.getMessage());
             KafkaException groupAuthorizationException = GroupAuthorizationException.forGroupId(this.groupId);
-            backgroundEventHandler.add(new ErrorEvent(groupAuthorizationException));
+            fatalError = Optional.of(groupAuthorizationException);
             return;
         }
 
         log.warn("FindCoordinator request failed due to fatal exception", exception);
-        backgroundEventHandler.add(new ErrorEvent(exception));

Review Comment:
    It seems that each manager handles fatal errors differently, so allowing each manager to handle fatal errors in its own way after receiving them would be a better approach. I will draft a version to test this though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on PR #18050:
URL: https://github.com/apache/kafka/pull/18050#issuecomment-2543124548

   Thanks for @kirktrue comments,
   
   > I have a personal preference to avoid different RequestManager classes from calling one another directly. I could imagine introducing a class that's created in the RequestManagers class and then passed down into any RequestManager that needs it. 
   
   Both `AbstractHeartbeatRequestManager` and `CommitRequestManager` now require `CoordinatorRequestManager`, so I’m uncertain whether creating a `Tracker` class would effectively prevent different `RequestManager` classes from directly invoking one another. Additionally, fatal errors include not only `GROUP_AUTHORIZATION` but also other unpredictable exceptions. If we introduce a new class, it would essentially serve as a record. WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1884655709


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -200,12 +196,12 @@ private void onFailedResponse(final long currentTimeMs, final Throwable exceptio
         if (exception == Errors.GROUP_AUTHORIZATION_FAILED.exception()) {
             log.debug("FindCoordinator request failed due to authorization error {}", exception.getMessage());
             KafkaException groupAuthorizationException = GroupAuthorizationException.forGroupId(this.groupId);
-            backgroundEventHandler.add(new ErrorEvent(groupAuthorizationException));
+            fatalError = Optional.of(groupAuthorizationException);

Review Comment:
   Are there any edge cases that could occur if `fatalError` was already set when it's overwritten here? 



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -188,12 +184,12 @@ private void onFailedResponse(final long currentTimeMs, final Throwable exceptio
         if (exception == Errors.GROUP_AUTHORIZATION_FAILED.exception()) {
             log.debug("FindCoordinator request failed due to authorization error {}", exception.getMessage());
             KafkaException groupAuthorizationException = GroupAuthorizationException.forGroupId(this.groupId);
-            backgroundEventHandler.add(new ErrorEvent(groupAuthorizationException));
+            fatalError = Optional.of(groupAuthorizationException);
             return;
         }
 
         log.warn("FindCoordinator request failed due to fatal exception", exception);
-        backgroundEventHandler.add(new ErrorEvent(exception));
+        fatalError = Optional.of(exception);

Review Comment:
   It's cleared in the `AbstractHeartbeatRequestManager` at this point, but I don't know if it should also be cleared elsewhere instead or in addition to `AbstractHeartbeatRequestManager.poll()`.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -176,9 +176,15 @@ public CommitRequestManager(
      */
     @Override
     public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
-        // poll only when the coordinator node is known.
-        if (coordinatorRequestManager.coordinator().isEmpty())
+        // poll when the coordinator node is known and fatal errors are not present
+        if (coordinatorRequestManager.coordinator().isEmpty()) {
+            Optional<Throwable> fatalError = coordinatorRequestManager.fatalError();
+            if (fatalError.isPresent()) {
+                pendingRequests.unsentOffsetCommits.forEach(request -> request.future.completeExceptionally(fatalError.get()));
+                pendingRequests.unsentOffsetFetches.forEach(request -> request.future.completeExceptionally(fatalError.get()));
+            }
             return EMPTY;

Review Comment:
   The `CommitRequestManager` does _not_ clear the fatal error, but the `AbstractHeartbeatRequestManager` does. Can you explain and/or add some comments so that others can understand the distinction between these two cases?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -176,9 +176,15 @@ public CommitRequestManager(
      */
     @Override
     public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
-        // poll only when the coordinator node is known.
-        if (coordinatorRequestManager.coordinator().isEmpty())
+        // poll when the coordinator node is known and fatal errors are not present
+        if (coordinatorRequestManager.coordinator().isEmpty()) {
+            Optional<Throwable> fatalError = coordinatorRequestManager.fatalError();
+            if (fatalError.isPresent()) {
+                pendingRequests.unsentOffsetCommits.forEach(request -> request.future.completeExceptionally(fatalError.get()));
+                pendingRequests.unsentOffsetFetches.forEach(request -> request.future.completeExceptionally(fatalError.get()));
+            }
             return EMPTY;

Review Comment:
   If there's an error, do we not want to also fail any `commitSync()`/`commitAsync()` calls? Or is that handled separately?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1887078953


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractHeartbeatRequestManager.java:
##########
@@ -163,6 +163,11 @@ public abstract class AbstractHeartbeatRequestManager<R extends AbstractResponse
     public NetworkClientDelegate.PollResult poll(long currentTimeMs) {
         if (coordinatorRequestManager.coordinator().isEmpty() || membershipManager().shouldSkipHeartbeat()) {
             membershipManager().onHeartbeatRequestSkipped();
+            // When Consumer#poll, it will need to check Error events without triggering an API event,
+            // also some managers may need to take actions based on the Error events. Thus, the fatal error
+            // lifecycle is end, and the error event is propagated to the event queue.
+            coordinatorRequestManager.getAndClearFatalError()
+                    .ifPresent(fatalError -> backgroundEventHandler.add(new ErrorEvent(fatalError)));

Review Comment:
   what do you think about encapsulating this is a kind of `maybePropagateCoordinatorFatalErrorEvent`. I think it will make the intention much clearer (without needing comments)



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -176,9 +176,18 @@ public CommitRequestManager(
      */
     @Override
     public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
-        // poll only when the coordinator node is known.
-        if (coordinatorRequestManager.coordinator().isEmpty())
+        // poll when the coordinator node is known and fatal errors are not present
+        if (coordinatorRequestManager.coordinator().isEmpty()) {
+            // The fatal error completeExceptionally all pending requests, but application thread may not get the
+            // fatal error When Consumer#poll, because it will need to check Error events without triggering an API event.
+            // So, we won't clear the fatal error here.
+            Optional<Throwable> fatalError = coordinatorRequestManager.fatalError();
+            if (fatalError.isPresent()) {
+                pendingRequests.unsentOffsetCommits.forEach(request -> request.future.completeExceptionally(fatalError.get()));
+                pendingRequests.unsentOffsetFetches.forEach(request -> request.future.completeExceptionally(fatalError.get()));
+            }

Review Comment:
   for clarity, and to find some consistency with how the HBMgr handles coordinator errors, we could encapsulate this in a kind of `maybeFailPendingRequestsOnCoordinatorFatalError`? maybe with that it's clear enough and we don't even need the comments 



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -53,24 +51,22 @@
 public class CoordinatorRequestManager implements RequestManager {
     private static final long COORDINATOR_DISCONNECT_LOGGING_INTERVAL_MS = 60 * 1000;
     private final Logger log;
-    private final BackgroundEventHandler backgroundEventHandler;
     private final String groupId;
 
     private final RequestState coordinatorRequestState;
     private long timeMarkedUnknownMs = -1L; // starting logging a warning only after unable to connect for a while
     private long totalDisconnectedMin = 0;
     private Node coordinator;
+    private Optional<Throwable> fatalError = Optional.empty();

Review Comment:
   here I would suggest a comment to describe this is the last fatal error received, exposed so that managers that require a coordinator can know about it and take whatever action the need (ex. HB manager propagates error event to app thread, commit mgr cancel cached requests)



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -188,12 +184,12 @@ private void onFailedResponse(final long currentTimeMs, final Throwable exceptio
         if (exception == Errors.GROUP_AUTHORIZATION_FAILED.exception()) {
             log.debug("FindCoordinator request failed due to authorization error {}", exception.getMessage());
             KafkaException groupAuthorizationException = GroupAuthorizationException.forGroupId(this.groupId);
-            backgroundEventHandler.add(new ErrorEvent(groupAuthorizationException));
+            fatalError = Optional.of(groupAuthorizationException);
             return;
         }
 
         log.warn("FindCoordinator request failed due to fatal exception", exception);
-        backgroundEventHandler.add(new ErrorEvent(exception));
+        fatalError = Optional.of(exception);

Review Comment:
   I think clearing it on the HB manager should be enough, given that it's the one propagating it to the app thread (that's what we need to make sure happens before clearing it I would say)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on PR #18050:
URL: https://github.com/apache/kafka/pull/18050#issuecomment-2573471899

   >  clearing it after a "transient" fatal error (ie. missing ACLs added) seems conceptually right. 
   
   I completely agree with this perspective. While I believe it will be an edge case. If we addressed this, I think we should notice this comments, it would be another issue. If there are two "transient" fatal error, the first fatal error would be overwritten by second.
   https://github.com/apache/kafka/pull/18050#discussion_r1884655709


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on PR #18050:
URL: https://github.com/apache/kafka/pull/18050#issuecomment-2578191988

   Thanks for @kirktrue and @lianetm review, I addressed all comments, PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1911403443


##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManagerTest.java:
##########
@@ -253,6 +236,21 @@ public void testNetworkTimeout() {
         res2 = coordinatorManager.poll(time.milliseconds());
         assertEquals(1, res2.unsentRequests.size());
     }
+    
+    @Test

Review Comment:
   what about parametrizing this test to cover retriable errors in the following response? 
   
   `@EnumSource(value = Errors.class, names = {"NONE", "COORDINATOR_NOT_AVAILABLE"})`, and pass that error arg on ln 251. 
   
   The behaviour regarding the clear fatal error should be the same when the next response is successful or retriable, so I expect the test should apply to both params without changes



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -1246,6 +1248,16 @@ private List<NetworkClientDelegate.UnsentRequest> drainPendingCommits() {
             clearAll();
             return res;
         }
+
+        private void maybeFailCoordinatorFatalError() {

Review Comment:
   ```suggestion
           private void maybeFailOnCoordinatorFatalError() {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18050:
URL: https://github.com/apache/kafka/pull/18050#issuecomment-2591655321

   This PR didn't pass the tests and trunk (and likely 4.0, but I didn't check) have been broken since
   
   <img width="1555" alt="image" src="https://github.com/user-attachments/assets/9cead107-cad2-46d2-b4be-775f4e19415a" />
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1876144917


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -188,12 +184,12 @@ private void onFailedResponse(final long currentTimeMs, final Throwable exceptio
         if (exception == Errors.GROUP_AUTHORIZATION_FAILED.exception()) {
             log.debug("FindCoordinator request failed due to authorization error {}", exception.getMessage());
             KafkaException groupAuthorizationException = GroupAuthorizationException.forGroupId(this.groupId);
-            backgroundEventHandler.add(new ErrorEvent(groupAuthorizationException));
+            fatalError = Optional.of(groupAuthorizationException);
             return;
         }
 
         log.warn("FindCoordinator request failed due to fatal exception", exception);
-        backgroundEventHandler.add(new ErrorEvent(exception));
+        fatalError = Optional.of(exception);

Review Comment:
   seems we're never clearing this fatal error and we should right? we could consider clearing it when the request completes, or maybe better when a new request is generated on poll? thoughts?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -188,12 +184,12 @@ private void onFailedResponse(final long currentTimeMs, final Throwable exceptio
         if (exception == Errors.GROUP_AUTHORIZATION_FAILED.exception()) {
             log.debug("FindCoordinator request failed due to authorization error {}", exception.getMessage());
             KafkaException groupAuthorizationException = GroupAuthorizationException.forGroupId(this.groupId);
-            backgroundEventHandler.add(new ErrorEvent(groupAuthorizationException));
+            fatalError = Optional.of(groupAuthorizationException);
             return;
         }
 
         log.warn("FindCoordinator request failed due to fatal exception", exception);
-        backgroundEventHandler.add(new ErrorEvent(exception));

Review Comment:
   I expect that removing this here will introduce a gap we need to cover: how are we going to get these coordinator errors from the consumer.poll now? (if no commit in use)
   I see 2 options:
   1. we keep the coordinator mgr propagating its errors via the event queue, just because we do need to know about them in the app thread (from the consumer.poll), without a triggering API event + the coordinator mgr exposes a `fatalError` just to allow other managers in the background thread take internal actions (ex. commit mgr needs to cancel pending requests).
   2. the coord mgr does not propagate via error event anymore, just exposes the coordinator found or fatal error. Other managers that require a coordinator know what to do about it: 
       - commit request mgr uses the coordinator, or cancels all its pending requests if there is a coord fatal error
       - HB mgr uses the coordinator or propagates an error via ErrorEvent if there is a coord fatal error (currently the HB Mgr just skips sending a HB if there is no coordinator, but it could propagate the error too ) https://github.com/apache/kafka/blob/d76238a18fb6a86b4b08bc04918ea1a0ee626517/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractHeartbeatRequestManager.java#L164
   
   These are the only 2 managers that require a known coordinator, so I would expect we cover it all by ensuring that they handle the coordinator fatal error properly. Thoughts? (These are first thoughts, let's shape them)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1879153506


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -188,12 +184,12 @@ private void onFailedResponse(final long currentTimeMs, final Throwable exceptio
         if (exception == Errors.GROUP_AUTHORIZATION_FAILED.exception()) {
             log.debug("FindCoordinator request failed due to authorization error {}", exception.getMessage());
             KafkaException groupAuthorizationException = GroupAuthorizationException.forGroupId(this.groupId);
-            backgroundEventHandler.add(new ErrorEvent(groupAuthorizationException));
+            fatalError = Optional.of(groupAuthorizationException);
             return;
         }
 
         log.warn("FindCoordinator request failed due to fatal exception", exception);
-        backgroundEventHandler.add(new ErrorEvent(exception));

Review Comment:
   > we keep the coordinator mgr propagating its errors via the event queue, just because we do need to know about them in the app thread (from the consumer.poll)
   
   In my opinion, if we remove the event queue propagation method in this PR, it might lead to unexpected behavior in SharedConsumer. If this concern exists, should we consider keeping the event queue propagation method?
   
   > the coord mgr does not propagate via error event anymore, just exposes the coordinator found or fatal error. Other managers that require a coordinator know what to do about it
   
   I'm not entirely clear on this approach, so I'm still thinking it through.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1885068181


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -188,12 +184,12 @@ private void onFailedResponse(final long currentTimeMs, final Throwable exceptio
         if (exception == Errors.GROUP_AUTHORIZATION_FAILED.exception()) {
             log.debug("FindCoordinator request failed due to authorization error {}", exception.getMessage());
             KafkaException groupAuthorizationException = GroupAuthorizationException.forGroupId(this.groupId);
-            backgroundEventHandler.add(new ErrorEvent(groupAuthorizationException));
+            fatalError = Optional.of(groupAuthorizationException);
             return;
         }
 
         log.warn("FindCoordinator request failed due to fatal exception", exception);
-        backgroundEventHandler.add(new ErrorEvent(exception));

Review Comment:
   > the only thing is probably what you mentioned that we need to handle it in the ShareHBMgr too right? (similar to how the ConsumerHBMgr does, sending ErrorEvent to the app thread?).
   
   I wasn’t write clear, which caused some misunderstanding. I was considering whether `AsyncConsumer` and `ShareConsumer` handle fatal errors differently in `poll` method. However, at this point, it seems that moving the error event propagation to `AbstractHeartbeatRequestManager` should not be an issue.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on PR #18050:
URL: https://github.com/apache/kafka/pull/18050#issuecomment-2587254468

   Thanks for @lianetm and @kirktrue review :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1905871090


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -196,6 +198,14 @@ public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
         return new NetworkClientDelegate.PollResult(timeUntilNextPoll, requests);
     }
 
+    private void maybeFailPendingRequestsOnCoordinatorFatalError() {
+        Optional<Throwable> fatalError = coordinatorRequestManager.fatalError();
+        if (fatalError.isPresent()) {
+            pendingRequests.unsentOffsetCommits.forEach(request -> request.future.completeExceptionally(fatalError.get()));
+            pendingRequests.unsentOffsetFetches.forEach(request -> request.future.completeExceptionally(fatalError.get()));

Review Comment:
   Agree, the `clearAll` was only being called from poll, when draining the requests to send https://github.com/apache/kafka/blob/bc3a18c59c638f267d9ebe28105be6b94f659f4f/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java#L193 
   
   If there was a fatal coordinator error, the poll does an early return so we would never call `clearAll` and the failed requests would forever remain in the queue I expect (even after expiration even, because the early return wouldn't even allow to make it to the `failAndRemoveExpiredCommitRequests`). Makes sense @kirktrue? (ln 207 doesn't have a clearAll so just want to make sure we're looking at the same thing) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1901880810


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -196,6 +198,14 @@ public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
         return new NetworkClientDelegate.PollResult(timeUntilNextPoll, requests);
     }
 
+    private void maybeFailPendingRequestsOnCoordinatorFatalError() {
+        Optional<Throwable> fatalError = coordinatorRequestManager.fatalError();
+        if (fatalError.isPresent()) {
+            pendingRequests.unsentOffsetCommits.forEach(request -> request.future.completeExceptionally(fatalError.get()));

Review Comment:
   should we add a debug line here with something like "Failing all pending offset fetch and commit requests due to coordinator fatal error"?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -188,12 +184,12 @@ private void onFailedResponse(final long currentTimeMs, final Throwable exceptio
         if (exception == Errors.GROUP_AUTHORIZATION_FAILED.exception()) {
             log.debug("FindCoordinator request failed due to authorization error {}", exception.getMessage());
             KafkaException groupAuthorizationException = GroupAuthorizationException.forGroupId(this.groupId);
-            backgroundEventHandler.add(new ErrorEvent(groupAuthorizationException));
+            fatalError = Optional.of(groupAuthorizationException);
             return;
         }
 
         log.warn("FindCoordinator request failed due to fatal exception", exception);
-        backgroundEventHandler.add(new ErrorEvent(exception));
+        fatalError = Optional.of(exception);

Review Comment:
   second thoughts on this one, should we clear it also when we receive a successful coordinator response? 
   https://github.com/apache/kafka/blob/9818e9db26fa4eb1242f2dcba384702781e50930/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java#L176
   
   Just for completeness, to have this coordinator mgr not leaving any loose ends. I expect it's really not needed at this point just because of the way we poll all managers (all polled sequentially, HB always polled if there is a groupId, even if not in group), but clearing it after a "transient" fatal error (ie. missing ACLs added) seems conceptually right.  Thoughts?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -196,6 +198,14 @@ public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
         return new NetworkClientDelegate.PollResult(timeUntilNextPoll, requests);
     }
 
+    private void maybeFailPendingRequestsOnCoordinatorFatalError() {
+        Optional<Throwable> fatalError = coordinatorRequestManager.fatalError();
+        if (fatalError.isPresent()) {
+            pendingRequests.unsentOffsetCommits.forEach(request -> request.future.completeExceptionally(fatalError.get()));
+            pendingRequests.unsentOffsetFetches.forEach(request -> request.future.completeExceptionally(fatalError.get()));

Review Comment:
   don't we need to remove the pending requests that we're failing here? `pendingRequests` are usually cleared on poll [clearAll](https://github.com/apache/kafka/blob/64bbdb1a0319044d1fa60e3836ab4052c7a28c12/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java#L1221), but if there is a fatal error we won't make it there right?
   
   If this fix makes sense, let's try to add a unit test to ensure that we don't leave any pending commit request after a fatal error with the coordinator. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1907458576


##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManagerTest.java:
##########
@@ -194,25 +190,11 @@ public void testBackoffAfterRetriableFailure() {
     public void testPropagateAndBackoffAfterFatalError() {
         CoordinatorRequestManager coordinatorManager = setupCoordinatorManager(GROUP_ID);
         expectFindCoordinatorRequest(coordinatorManager, Errors.GROUP_AUTHORIZATION_FAILED);
-
-        verify(backgroundEventHandler).add(argThat(backgroundEvent -> {
-            if (!(backgroundEvent instanceof ErrorEvent))
-                return false;
-
-            RuntimeException exception = ((ErrorEvent) backgroundEvent).error();
-
-            if (!(exception instanceof GroupAuthorizationException))
-                return false;
-
-            GroupAuthorizationException groupAuthException = (GroupAuthorizationException) exception;
-            return groupAuthException.groupId().equals(GROUP_ID);
-        }));
-
-        time.sleep(RETRY_BACKOFF_MS - 1);

Review Comment:
   Sure, I will addressed it 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on PR #18050:
URL: https://github.com/apache/kafka/pull/18050#issuecomment-2591697583

   Thanks for @ijuma, I will take a look which test timeout and make build error


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1904255768


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -196,6 +198,14 @@ public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
         return new NetworkClientDelegate.PollResult(timeUntilNextPoll, requests);
     }
 
+    private void maybeFailPendingRequestsOnCoordinatorFatalError() {
+        Optional<Throwable> fatalError = coordinatorRequestManager.fatalError();
+        if (fatalError.isPresent()) {
+            pendingRequests.unsentOffsetCommits.forEach(request -> request.future.completeExceptionally(fatalError.get()));
+            pendingRequests.unsentOffsetFetches.forEach(request -> request.future.completeExceptionally(fatalError.get()));

Review Comment:
   don't we need to remove the pending requests that we're failing here? `pendingRequests` are usually cleared on poll [clearAll](https://github.com/apache/kafka/blob/64bbdb1a0319044d1fa60e3836ab4052c7a28c12/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java#L1221), but if there is a fatal error we won't make it there right?
   
   If this fix makes sense, let's try to add a unit test to ensure that we don't leave any pending commit request that fails after a fatal error with the coordinator. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #18050:
URL: https://github.com/apache/kafka/pull/18050#issuecomment-2592756672

   Thanks! Reverted in 4.0 while we stabilize this. Totally my bad missing the output, probably focusing on the tests
   
   @m1a2st ping me if I can help with the investigation. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1904717131


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -196,6 +198,16 @@ public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
         return new NetworkClientDelegate.PollResult(timeUntilNextPoll, requests);
     }
 
+    private void maybeFailPendingRequestsOnCoordinatorFatalError() {
+        Optional<Throwable> fatalError = coordinatorRequestManager.fatalError();
+        if (fatalError.isPresent()) {
+            log.warn("Failing all unset commit requests and offset fetches because of coordinator fatal error. ", fatalError.get());

Review Comment:
   ```suggestion
               log.warn("Failing all unsent commit requests and offset fetches because of coordinator fatal error. ", fatalError.get());
   ```



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -196,6 +198,14 @@ public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
         return new NetworkClientDelegate.PollResult(timeUntilNextPoll, requests);
     }
 
+    private void maybeFailPendingRequestsOnCoordinatorFatalError() {
+        Optional<Throwable> fatalError = coordinatorRequestManager.fatalError();
+        if (fatalError.isPresent()) {
+            pendingRequests.unsentOffsetCommits.forEach(request -> request.future.completeExceptionally(fatalError.get()));
+            pendingRequests.unsentOffsetFetches.forEach(request -> request.future.completeExceptionally(fatalError.get()));

Review Comment:
   > don't we need to remove the pending requests that we're failing here? `pendingRequests` are usually cleared on poll [clearAll]
   
   Isn't that what `clearAll()` on line 207 does?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractHeartbeatRequestManager.java:
##########
@@ -263,6 +264,11 @@ public void resetPollTimer(final long pollMs) {
         pollTimer.reset(maxPollIntervalMs);
     }
 
+    private void maybePropagateCoordinatorFatalErrorEvent() {
+        coordinatorRequestManager.getAndClearFatalError()
+                .ifPresent(fatalError -> backgroundEventHandler.add(new ErrorEvent(fatalError)));
+    }
+

Review Comment:
   My apologies—it's still not clear from looking at the code why the `AbstractHeartbeatRequestManager` clears the fatal error but `CommitRequestManager` doesn't. Can you provide some context for the different ways of interacting with the error? Thanks!



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -196,6 +198,16 @@ public NetworkClientDelegate.PollResult poll(final long currentTimeMs) {
         return new NetworkClientDelegate.PollResult(timeUntilNextPoll, requests);
     }
 
+    private void maybeFailPendingRequestsOnCoordinatorFatalError() {

Review Comment:
   What about moving the entire `maybeFailPendingRequestsOnCoordinatorFatalError()` method into the inner `PendingRequests` class directly? That way the outer class doesn't have to worry about the details of how to fail them, only that it has to call `pendingRequests.maybeFailOnCoordinatorFatalError()`?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -188,12 +184,12 @@ private void onFailedResponse(final long currentTimeMs, final Throwable exceptio
         if (exception == Errors.GROUP_AUTHORIZATION_FAILED.exception()) {
             log.debug("FindCoordinator request failed due to authorization error {}", exception.getMessage());
             KafkaException groupAuthorizationException = GroupAuthorizationException.forGroupId(this.groupId);
-            backgroundEventHandler.add(new ErrorEvent(groupAuthorizationException));
+            fatalError = Optional.of(groupAuthorizationException);
             return;
         }
 
         log.warn("FindCoordinator request failed due to fatal exception", exception);
-        backgroundEventHandler.add(new ErrorEvent(exception));
+        fatalError = Optional.of(exception);

Review Comment:
   Clearing the fatal error on a successful response from the coordinator seems _intuitively_ correct, but I could easily be wrong.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -53,24 +51,27 @@
 public class CoordinatorRequestManager implements RequestManager {
     private static final long COORDINATOR_DISCONNECT_LOGGING_INTERVAL_MS = 60 * 1000;
     private final Logger log;
-    private final BackgroundEventHandler backgroundEventHandler;
     private final String groupId;
 
     private final RequestState coordinatorRequestState;
     private long timeMarkedUnknownMs = -1L; // starting logging a warning only after unable to connect for a while
     private long totalDisconnectedMin = 0;
     private Node coordinator;
+    // Hold the last fatal error received. It is exposed so that managers requiring a coordinator can access it and take 

Review Comment:
   Super nit picky, but can we change the term _last_ to either _latest_ or _most recent_?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1884523029


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManager.java:
##########
@@ -188,12 +184,12 @@ private void onFailedResponse(final long currentTimeMs, final Throwable exceptio
         if (exception == Errors.GROUP_AUTHORIZATION_FAILED.exception()) {
             log.debug("FindCoordinator request failed due to authorization error {}", exception.getMessage());
             KafkaException groupAuthorizationException = GroupAuthorizationException.forGroupId(this.groupId);
-            backgroundEventHandler.add(new ErrorEvent(groupAuthorizationException));
+            fatalError = Optional.of(groupAuthorizationException);
             return;
         }
 
         log.warn("FindCoordinator request failed due to fatal exception", exception);
-        backgroundEventHandler.add(new ErrorEvent(exception));

Review Comment:
   ack. With the approach of letting each manager that requires a coordinator handle the error as it needs, the only thing is probably what you mentioned that we need to handle it in the ShareHBMgr too right? (similar to how the ConsumerHBMgr does, sending ErrorEvent to the app thread?). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18034: CommitRequestManager should fail pending requests on fatal coordinator errors [kafka]

Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #18050:
URL: https://github.com/apache/kafka/pull/18050#discussion_r1905888150


##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CoordinatorRequestManagerTest.java:
##########
@@ -194,25 +190,11 @@ public void testBackoffAfterRetriableFailure() {
     public void testPropagateAndBackoffAfterFatalError() {
         CoordinatorRequestManager coordinatorManager = setupCoordinatorManager(GROUP_ID);
         expectFindCoordinatorRequest(coordinatorManager, Errors.GROUP_AUTHORIZATION_FAILED);
-
-        verify(backgroundEventHandler).add(argThat(backgroundEvent -> {
-            if (!(backgroundEvent instanceof ErrorEvent))
-                return false;
-
-            RuntimeException exception = ((ErrorEvent) backgroundEvent).error();
-
-            if (!(exception instanceof GroupAuthorizationException))
-                return false;
-
-            GroupAuthorizationException groupAuthException = (GroupAuthorizationException) exception;
-            return groupAuthException.groupId().equals(GROUP_ID);
-        }));
-
-        time.sleep(RETRY_BACKOFF_MS - 1);

Review Comment:
   I think we should keep this line, and the we don't need to change the assertion below (I would expect the only change in this test is to remove the verify). Should we rename it to `testBackoffAfterFatalError`?



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CommitRequestManagerTest.java:
##########
@@ -1481,6 +1482,23 @@ public void testSignalClose() {
         OffsetCommitRequestData data = (OffsetCommitRequestData) res.unsentRequests.get(0).requestBuilder().build().data();
         assertEquals("topic", data.topics().get(0).name());
     }
+    
+    @Test
+    public void testPollWithFatalErrorShouldFailingAllUnsetRequest() {

Review Comment:
   typo Unsent



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractHeartbeatRequestManager.java:
##########
@@ -263,6 +264,11 @@ public void resetPollTimer(final long pollMs) {
         pollTimer.reset(maxPollIntervalMs);
     }
 
+    private void maybePropagateCoordinatorFatalErrorEvent() {
+        coordinatorRequestManager.getAndClearFatalError()
+                .ifPresent(fatalError -> backgroundEventHandler.add(new ErrorEvent(fatalError)));
+    }
+

Review Comment:
   good question. Actually, if my other [comment](https://github.com/apache/kafka/pull/18050/files#r1904292495) makes sense to you all, maybe we don't need to clear the error in the `AbstractHeartbeatRequestManager`?
   
   We would end up with a simpler alternative where it's the `coordinatorReqMgr` the sole responsible for maintaining the fatal coordinator error (sets it when it receives a fatal error, clears it when it receives a successful response). HBMgr and CommitMgr only read the error to take actions specific the them, never clear it. Would that work?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org