[jira] [Assigned] (KAFKA-20719) "Streams" group sticky assignor not sticky at startup

["Matthias J. Sax (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-20719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias J. Sax reassigned KAFKA-20719:
---------------------------------------

    Assignee: Ziyun Fu

> "Streams" group sticky assignor not sticky at startup
> -----------------------------------------------------
>
>                 Key: KAFKA-20719
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20719
>             Project: Kafka
>          Issue Type: Bug
>          Components: group-coordinator, streams
>    Affects Versions: 4.2.0
>            Reporter: Matthias J. Sax
>            Assignee: Ziyun Fu
>            Priority: Critical
>
> For "streams" groups (KIP-1071), the sticky-assignor is not sticky at startup of a new KS instance/client.
> In the "classic" protocol, a newly starting client encodes what task directories it finds in its local state directory as part of the rebalance metadata. This allows the "classic" client-side sticky assignor to re-assign tasks to previous clients, meeting the goal of being sticky.
> In contrast, for "streams" group no such information is encoded. A starting instance correctly sends empty previous assignment on its initial "join group heartbeat" (with epoch=0). While the group coordinator does pass in the "current/previous assignment" into the sticky assignor, new clients/threads get a new memberId, and thus previous state cannot be re-assigned in a sticky manner on startup. (Note, static membership might address this, but it's also not yet supported for "streams" group: https://issues.apache.org/jira/browse/KAFKA-20169)
> To fix this issue, we should encode offset information(derived from the local task offset checkpoint) via `taskOffsetSum` field in the "join group heartbeat", and update the sticky-assignor to use this data.
> This ticket is blocked by https://issues.apache.org/jira/browse/KAFKA-20116 which does add support for `taskOffsetSum` client and broker side already, to support warmup tasks. After KAFKA-20116 was completed, this ticket would need to fix the sticky-assignor itself to use the newly available data.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)