[jira] [Updated] (KAFKA-21300) Streams group task ownership index is not rolled back, blocking valid task assignments

["Matthias J. Sax (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-21300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias J. Sax updated KAFKA-21300:
------------------------------------
    Labels:   (was: group-coordinator streams)

> Streams group task ownership index is not rolled back, blocking valid task assignments
> --------------------------------------------------------------------------------------
>
>                 Key: KAFKA-21300
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21300
>             Project: Kafka
>          Issue Type: Bug
>          Components: group-coordinator, streams
>            Reporter: sanghyeok An
>            Assignee: sanghyeok An
>            Priority: Minor
>
> When the Streams group coordinator rolls back a task assignment change, member assignments and the process ownership index for standby and warm-up tasks can become inconsistent.
> In addition to per-member assignments, the coordinator maintains a set of process IDs that own each task. This index prevents the same task from being assigned to different members within the same process in conflicting roles.
> Although the ownership index is stored in a {{{}TimelineHashMap{}}}, its values are ordinary {{HashSet}} instances that are mutated in place. Reverting to a snapshot therefore does not restore the previous contents of these sets. Member assignments are restored correctly, but the index may retain ownership from a rolled-back assignment or lose ownership that should have been restored.
> A representative scenario involves a log append failure while adding a standby owner to a task that already has one:
>  # P1 owns active task T, P3 owns standby task T, and P2 has no assigned tasks.
>  # Increasing the number of standby replicas produces a target assignment that also assigns standby T to P2.
>  # While processing P2’s heartbeat, the coordinator replays the current-assignment record in memory, changing the standby ownership index to {{{}{P3, P2}{}}}.
>  # The log append fails and the coordinator rolls back. P2’s assignment becomes empty again, but P2 remains in the ownership index.
>  # If P1 and P3 subsequently leave and P2 needs to receive active T, the coordinator blocks the assignment because it believes another member in process P2 still owns standby T.
> No actual owner exists to release the task. The member can continue sending successful heartbeats while retaining an empty assignment and remaining in {{{}UNRELEASED_TASKS{}}}. For an application with a single task, this can halt input processing.
> Removing the original member and rejoining with the same process ID may not resolve the issue. Since the original member’s assignment is empty after rollback, member removal does not clean up the stale ownership entry. Restarting an application with persistent stores while preserving its state directory can reuse the same process ID and meet this condition.
> The reverse inconsistency can also occur. If a standby or warm-up owner is removed and that change is rolled back, the member’s assignment is restored, but its process ID may remain absent from the ownership index. The check that should prevent another member in the same process from receiving the active task can then incorrectly allow the assignment.
> The impact can persist after such a rollback while the same coordinator state remains loaded. Reconstructing coordinator state from persisted member records also rebuilds the ownership index.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)