[jira] [Updated] (KAFKA-21169) Streams group coordinator may trigger unnecessary rebalances on every heartbeat when a custom assignor returns empty task sets

["Matthias J. Sax (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-21169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias J. Sax updated KAFKA-21169:
------------------------------------
    Labels:   (was: group-coordinator streams)

> Streams group coordinator may trigger unnecessary rebalances on every heartbeat when a custom assignor returns empty task sets
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-21169
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21169
>             Project: Kafka
>          Issue Type: Bug
>          Components: group-coordinator, streams
>            Reporter: sanghyeok An
>            Assignee: sanghyeok An
>            Priority: Minor
>
> Custom task assignors may use different representations for a subtopology with no assigned tasks. Some implementations omit the subtopology entry, while others include it with an empty task set:
>  
> {code:java}
> // No standby tasks: omit the subtopology entry.
> Map.of()
> // No standby tasks: include the subtopology with an empty set.
> Map.of("subtopology-1", Set.of()){code}
> Both representations assign zero standby tasks. The public MemberAssignment API does not explicitly require empty subtopology entries to be omitted, and the coordinator accepts and persists them.
> However, the target assignment preserves empty entries, while reconciliation omits them from the current assignment. TasksTuple.sameTasks() treats these representations as different. As a result, an otherwise stable Streams group can trigger an unnecessary assignment refinement on every subsequent heartbeat, even though its actual task assignment has not changed.
> This advances the group, target assignment, and member epochs and generates additional coordinator records. Reconciliation leaves the same representation difference in place, so the next heartbeat triggers another update.
> Here, an unnecessary rebalance means an epoch and assignment update in the coordinator. In the reproduction, the assignor was called only once; the subsequent updates did not require rerunning it. Task restarts or data loss have not been demonstrated.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)