[jira] [Created] (KAFKA-20170) Support topology updates for streams groups without requiring a new group

["Lucas Brutschy (Jira)" <ji...@apache.org>]

Lucas Brutschy created KAFKA-20170:
--------------------------------------

             Summary: Support topology updates for streams groups without requiring a new group
                 Key: KAFKA-20170
                 URL: https://issues.apache.org/jira/browse/KAFKA-20170
             Project: Kafka
          Issue Type: Task
          Components: streams, group-coordinator
            Reporter: Lucas Brutschy


The streams rebalance protocol \(KIP\-1071\) does not currently support topology updates. If a topology is changed significantly \(e.g., by adding new source topics or changing the number of subtopologies\), users must create a new streams group. This is a significant limitation for applications that need to evolve over time.

As specified in KIP\-1071, topology updates should be supported through explicit versioning via a topology.epoch configuration. When a member joins with a different topology, the broker should compare epochs: if the epoch is the same but metadata differs, return STREAMS\_INVALID\_TOPOLOGY\_EPOCH; if the epoch is lower, return STREAMS\_TOPOLOGY\_FENCED; if the epoch is bumped by one, accept and update the group topology. Members with stale topologies should receive a STALE\_TOPOLOGY status in heartbeat responses but continue processing with their current tasks until upgraded. This enables rolling upgrades of the topology across application instances.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)