You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by li...@apache.org on 2026/08/06 12:43:14 UTC

(kafka) branch trunk updated: KAFKA-20765: Fix OffsetFetch stale-epoch retry spinning in dedup loop (#22747)

This is an automated email from the ASF dual-hosted git repository.

lianetm pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git


The following commit(s) were added to refs/heads/trunk by this push:
     new fe886479351 KAFKA-20765: Fix OffsetFetch stale-epoch retry spinning in dedup loop (#22747)
fe886479351 is described below

commit fe886479351b712c43bbe6c440be7c836f26cb1a
Author: Alieh Saeedi <10...@users.noreply.github.com>
AuthorDate: Thu Aug 6 14:43:07 2026 +0200

    KAFKA-20765: Fix OffsetFetch stale-epoch retry spinning in dedup loop (#22747)
    
    Fixes [KAFKA-20765](https://issues.apache.org/jira/browse/KAFKA-20765).
    
    ## The issue
    
    In the async consumer (`AsyncKafkaConsumer`), when an in-flight
    OffsetFetch fails with `STALE_MEMBER_EPOCH` and a refreshed member epoch
    is already known, `CommitRequestManager` correctly classifies the error
    as retriable — but the retry can enter a tight, non-progressing loop
    instead of being re-sent:
    
    1. Two callers fetch committed offsets for the same partitions. The
    second request is deduplicated by
    `PendingRequests.addOffsetFetchRequest` and *chained* onto the first
    (in-flight) one.
    2. The in-flight request fails with `STALE_MEMBER_EPOCH` (e.g. another
    member joined and the coordinator bumped the group epoch while the
    request was in flight — routine under the KIP-1071 streams protocol
    during staggered startup).
    3. The chained request's completion callback runs *before* the original
    request is removed from the in-flight buffer (CompletableFuture
    dependents run LIFO, and buffer removal is itself a completion
    callback). Its retry is handed back to `addOffsetFetchRequest`, which
    finds the original request still in the buffer — `sameRequest()`
    compares only the requested partitions — and chains the retry onto it.
    4. But that request's future is **already completed** (with the same
    stale-epoch error), so chaining completes the retry's future immediately
    and synchronously. The error handler sees `STALE_MEMBER_EPOCH` again,
    retries again, chains again… a synchronous spin that only ends when the
    stack blows up.
    
    Consequences (all observed in a system test run and reproduced in the
    unit test):
    - `WARN "A duplicated, inflight, request was identified, but unable to
    find it in the outbound buffer"` repeated hundreds of times within
    milliseconds,
    - `java.lang.StackOverflowError` surfacing in SLF4J's `toString()` of
    `OffsetFetchRequestState`,
    - no OffsetFetch with the new epoch ever goes on the wire from this path
    (the epoch is stamped at send time — the one stage the loop never
    reaches),
    - the application-level future is never completed (orphaned).
    
    In Kafka Streams with the streams rebalance protocol (KIP-1071) this
    blocks the stream thread in committed-offset initialization for the full
    `default.api.timeout.ms`, exceeds `max.poll.interval.ms`, and gets the
    member evicted (observed in `streams_broker_down_resilience_test` with
    staggered startup). Plain `group.protocol=consumer` apps are exposed to
    the same spin; KIP-1071 just triggers the preconditions far more easily.
    
    ## The fix
    
    In `PendingRequests.addOffsetFetchRequest`, requests whose future is
    already completed are no longer considered duplicates. A completed
    request cannot deliver a result to anyone; it only appears in the
    buffers transiently while its completion callbacks run. With the check
    in place, the retry falls through to being enqueued and re-sent, picking
    up the current member epoch at send time.
    
    Also included:
    - A unit test that deterministically reproduces the loop (fails without
    the fix with the exact production signatures) and verifies the retry is
    sent with the new epoch and completes both callers' futures.
    - Log cleanup: the stale-epoch ERROR no longer claims the request
    "cannot be retried" when a retry is about to happen (only logged when
    the member has no epoch anymore), and the per-chained-caller
    "duplicated, inflight" WARN is now an accurate debug message (it fires
    for every deduplicated request, success or failure — expected, not an
    anomaly).
    
    ## Testing
    
    `./gradlew :clients:test --tests
    "org.apache.kafka.clients.consumer.internals.CommitRequestManagerTest"`
    — all pass, including the new
    `testDuplicatedOffsetFetchFailsWithStaleEpochAndRetriesWithNewEpoch`.
    
    🤖 Generated with [Claude Code](https://claude.com/claude-code)
    
    Reviewers: Lianet Magrans <lm...@confluent.io>
    
    ---------
    
    Co-authored-by: Claude Fable 5 <no...@anthropic.com>
---
 .../consumer/internals/CommitRequestManager.java   | 31 +++++++----
 .../internals/CommitRequestManagerTest.java        | 60 ++++++++++++++++++++++
 2 files changed, 82 insertions(+), 9 deletions(-)

diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java
index 27fcd53607f..a287f935799 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java
+++ b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java
@@ -554,8 +554,12 @@ public class CommitRequestManager implements RequestManager, MemberStateListener
         currentResult.whenComplete((res, error) -> {
             boolean inflightRemoved = pendingRequests.inflightOffsetFetches.remove(fetchRequest);
             if (!inflightRemoved) {
-                log.warn("A duplicated, inflight, request was identified, but unable to find it in the " +
-                    "outbound buffer: {}", fetchRequest);
+                // A completed request may legitimately not be in the in-flight buffer for a few
+                // reasons: it was deduplicated and chained onto an existing request (so it was never
+                // added to the buffers), or it completed before it was ever sent (e.g. while there
+                // was no coordinator available). In all these cases there is nothing to remove here.
+                log.debug("Completed offset fetch request was not found in the in-flight buffer: {}",
+                    fetchRequest);
             }
 
             // Group-level error
@@ -1223,9 +1227,14 @@ public class CommitRequestManager implements RequestManager, MemberStateListener
                     " anymore.", responseError);
                 future.completeExceptionally(exception);
             } else if (responseError == Errors.STALE_MEMBER_EPOCH) {
-                log.error("OffsetFetch failed with {} and the consumer is not part " +
-                    "of the group anymore (it probably left the group, got fenced" +
-                    " or failed). The request cannot be retried and will fail.", responseError);
+                if (memberInfo.memberEpoch.isPresent()) {
+                    log.debug("OffsetFetch failed with {}. The member is still in the group, so the " +
+                        "request can be retried with the latest member epoch as long as it has not expired.", responseError);
+                } else {
+                    log.error("OffsetFetch failed with {} and the consumer is not part " +
+                        "of the group anymore (it probably left the group, got fenced" +
+                        " or failed). The request cannot be retried and will fail.", responseError);
+                }
                 future.completeExceptionally(exception);
             } else if (responseError == Errors.NOT_COORDINATOR || responseError == Errors.COORDINATOR_NOT_AVAILABLE) {
                 // Re-discover the coordinator and retry
@@ -1395,17 +1404,21 @@ public class CommitRequestManager implements RequestManager, MemberStateListener
         }
 
         /**
-         * <p>Adding an offset fetch request to the outgoing buffer.  If the same request was made, we chain the future
-         * to the existing one.
+         * <p>Adding an offset fetch request to the outgoing buffer.  If the same request was made and has not
+         * completed yet, we chain the future to the existing one.
          *
          * <p>If the request is new, it invokes a callback to remove itself from the {@code inflightOffsetFetches}
          * upon completion.
          */
         private CompletableFuture<OffsetFetchResult> addOffsetFetchRequest(final OffsetFetchRequestState request) {
+            // A request that already completed cannot deliver a result anymore, but may still appear in the
+            // buffers while its completion callbacks run (removal from the buffer is itself one of those
+            // callbacks). Chaining onto it would complete the new request immediately with the stale outcome
+            // instead of sending it (e.g. re-failing the retry of a STALE_MEMBER_EPOCH error in a tight loop).
             Optional<OffsetFetchRequestState> dupe =
-                    unsentOffsetFetches.stream().filter(r -> r.sameRequest(request)).findAny();
+                    unsentOffsetFetches.stream().filter(r -> r.sameRequest(request) && !r.future.isDone()).findAny();
             Optional<OffsetFetchRequestState> inflight =
-                    inflightOffsetFetches.stream().filter(r -> r.sameRequest(request)).findAny();
+                    inflightOffsetFetches.stream().filter(r -> r.sameRequest(request) && !r.future.isDone()).findAny();
 
             if (dupe.isPresent() || inflight.isPresent()) {
                 log.debug("Duplicated unsent offset fetch request found for partitions: {}", request.requestedPartitions);
diff --git a/clients/src/test/java/org/apache/kafka/clients/consumer/internals/CommitRequestManagerTest.java b/clients/src/test/java/org/apache/kafka/clients/consumer/internals/CommitRequestManagerTest.java
index e3a2a506096..426b55833ed 100644
--- a/clients/src/test/java/org/apache/kafka/clients/consumer/internals/CommitRequestManagerTest.java
+++ b/clients/src/test/java/org/apache/kafka/clients/consumer/internals/CommitRequestManagerTest.java
@@ -1238,6 +1238,66 @@ public class CommitRequestManagerTest {
         assertEquals(memberId, reqData.groups().get(0).memberId());
     }
 
+    // Same as testSyncOffsetFetchFailsWithStaleEpochAndRetriesWithNewEpoch, but with a
+    // duplicated fetch for the same partitions chained onto the in-flight request when the
+    // STALE_MEMBER_EPOCH error is received. The retry of the chained request must not be
+    // deduplicated against the already-completed in-flight request: chaining onto a completed
+    // future fails it again immediately and re-triggers the retry in a tight synchronous loop
+    // that never sends a request with the new epoch and never completes the callers' futures
+    // (KAFKA-20765).
+    @Test
+    public void testDuplicatedOffsetFetchFailsWithStaleEpochAndRetriesWithNewEpoch() {
+        CommitRequestManager commitRequestManager = create(false, 100);
+        Set<TopicPartition> partitions = Collections.singleton(new TopicPartition("t1", 0));
+        when(coordinatorRequestManager.coordinator()).thenReturn(Optional.of(mockedNode));
+
+        // Two callers fetch offsets for the same partitions; the second request is deduplicated
+        // and chained onto the first.
+        long deadlineMs = time.milliseconds() + defaultApiTimeoutMs;
+        CompletableFuture<CommitRequestManager.OffsetFetchResult> firstResult =
+            commitRequestManager.fetchOffsets(partitions, deadlineMs);
+        CompletableFuture<CommitRequestManager.OffsetFetchResult> secondResult =
+            commitRequestManager.fetchOffsets(partitions, deadlineMs);
+
+        // A single deduplicated request goes on the wire.
+        NetworkClientDelegate.PollResult res = commitRequestManager.poll(time.milliseconds());
+        assertEquals(1, res.unsentRequests.size());
+
+        // Mock member has a new valid epoch, so STALE_MEMBER_EPOCH is retriable.
+        int newEpoch = 8;
+        String memberId = "member1";
+        commitRequestManager.onMemberEpochUpdated(Optional.of(newEpoch), memberId);
+
+        // Receive error when member already has a newer member epoch. Request should be retried.
+        res.unsentRequests.get(0).handler().onComplete(
+            buildOffsetFetchClientResponse(res.unsentRequests.get(0), partitions, Errors.STALE_MEMBER_EPOCH));
+
+        // The failed request should be removed from the in-flight buffer, a retry should be
+        // enqueued, and the callers' futures should still be waiting for the retry's outcome.
+        assertEquals(0, commitRequestManager.pendingRequests.inflightOffsetFetches.size());
+        assertEquals(1, commitRequestManager.pendingRequests.unsentOffsetFetches.size());
+        assertFalse(firstResult.isDone());
+        assertFalse(secondResult.isDone());
+
+        // The deduplicated retry is chained onto the original as a fresh request, so it is sent on
+        // the next poll with no backoff, carrying the latest member ID and epoch.
+        res = commitRequestManager.poll(time.milliseconds());
+        assertEquals(1, res.unsentRequests.size());
+        OffsetFetchRequestData reqData =
+            (OffsetFetchRequestData) res.unsentRequests.get(0).requestBuilder().build().data();
+        assertEquals(1, reqData.groups().size());
+        assertEquals(newEpoch, reqData.groups().get(0).memberEpoch());
+        assertEquals(memberId, reqData.groups().get(0).memberId());
+
+        // A successful response should complete both callers' futures.
+        res.unsentRequests.get(0).handler().onComplete(
+            buildOffsetFetchClientResponse(res.unsentRequests.get(0), partitions, Errors.NONE));
+        assertTrue(firstResult.isDone());
+        assertFalse(firstResult.isCompletedExceptionally());
+        assertTrue(secondResult.isDone());
+        assertFalse(secondResult.isCompletedExceptionally());
+    }
+
     // This should be the case of an OffsetFetch that fails because the member is not in the
     // group anymore (left the group, failed with fatal error, or got fenced). In that case the
     // request should fail without retry.