[jira] [Reopened] (KAFKA-21255) Streams heartbeat retry responses can omit assignments after response loss

["Lucas Brutschy (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lucas Brutschy reopened KAFKA-21255:
------------------------------------

> Streams heartbeat retry responses can omit assignments after response loss
> --------------------------------------------------------------------------
>
>                 Key: KAFKA-21255
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21255
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: sanghyeok An
>            Assignee: sanghyeok An
>            Priority: Major
>              Labels: group-coordinator, streams
>
> When a Streams group coordinator applies an assignment change but the heartbeat response is lost, the server and client can retain different assignments.
> After a heartbeat failure, the client retries with its current active, standby, and warmup tasks. However, the server has already applied the assignment change while processing the previous request. If processing the retry produces no further assignment change, the server omits the task fields from its response. Consequently, the assignment that the client missed is not delivered again.
> The following scenarios illustrate the issue. These examples use only active tasks; the client reports empty lists ({{{}[]{}}}) for standby and warmup tasks when retrying.
>  
> *Scenario 1: Losing an assignment response that advances the member epoch*
> The client owns task {{[0]}} at member epoch {{{}1{}}}. While processing a heartbeat, the server advances the member epoch to {{2}} and assigns tasks {{{}[0, 1]{}}}, but the response is lost.
> The client retries with its existing epoch {{1}} and owned task {{{}[0]{}}}. Since the server has already recorded epoch {{2}} and assignment {{{}[0, 1]{}}}, the retry response returns epoch {{2}} but omits the task fields.
> The client updates its epoch to {{2}} while continuing to own only task {{{}[0]{}}}. The server considers task {{1}} assigned, but the client does not process it. This discrepancy can persist even while subsequent heartbeats succeed.
>  
> *Scenario 2: Losing a revocation response within the same member epoch*
> The client owns tasks {{[0, 1]}} at member epoch {{{}1{}}}, and its new target assignment is {{{}[0]{}}}. The server transitions the member to {{{}UNREVOKED_TASKS{}}}, waiting for task {{1}} to be released, and responds with epoch {{1}} and the remaining task {{{}[0]{}}}. This response is lost.
> Because the client did not receive the revocation instruction, it retries with epoch {{1}} and owned tasks {{{}[0, 1]{}}}. The server remains in the same state and omits the task fields because its assignment has not changed.
> The client continues to own task {{1}} and cannot acknowledge its revocation. Moving that task to another member is delayed. When the existing rebalance timeout expires, the member is removed and must rejoin.
>  
> *Scenario 3: Losing an additional assignment within the same member epoch*
> The client has already reached member epoch {{{}2{}}}, but the member is in {{{}UNRELEASED_TASKS{}}}, waiting for another member to release a task. Once the previous owner releases it, the server assigns the task without changing member epoch {{{}2{}}}. This response is lost.
> The client retries with epoch {{2}} and the tasks it owned before the additional assignment. Since the server has already recorded the assignment, it omits the task fields from the retry response.
> The server and client have the same member epoch but different assignments. The client does not process the task that the server considers assigned, and successful heartbeats alone may not resolve the discrepancy.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)