You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "TengYao Chi (Jira)" <ji...@apache.org> on 2026/06/15 16:48:00 UTC

[jira] [Created] (KAFKA-20696) Periodic plugin.deleteTopology cleanup for naturally-expired streams groups

TengYao Chi created KAFKA-20696:
-----------------------------------

             Summary: Periodic plugin.deleteTopology cleanup for naturally-expired streams groups
                 Key: KAFKA-20696
                 URL: https://issues.apache.org/jira/browse/KAFKA-20696
             Project: Kafka
          Issue Type: Sub-task
            Reporter: TengYao Chi


Periodic topology description cleanup task: when a plugin is configured, the broker runs a cleanup cycle every offsets.retention.check.interval.ms to release plugin-side state for naturally-expired streams groups before the offset-expiration sweep tombstones them.

Cycle: fan out a read-only eligibility query across the broker's hosted __consumer_offsets partitions (isEmpty && allOffsetsExpired && StoredDescriptionTopologyEpoch != -1), then for each eligible group call plugin.deleteTopology and, on success, clear StoredDescriptionTopologyEpoch via a metadata record conditional on the epoch the cycle observed (no-op if a concurrent setTopology has advanced it). Plugin failures leave the field set; the same group is retried on the next cycle. Single-flight: cycles don't overlap.

Gate the existing offset-expiration sweep: when a plugin is configured, streams groups must satisfy StoredDescriptionTopologyEpoch == -1 before the sweep is allowed to tombstone them, so plugin-side state is always released ahead of the in-memory record. Without a plugin the sweep behaves as today; operators disabling a previously-configured plugin are responsible for plugin-side cleanup out of band.

Lifecycle: timer starts when a plugin is configured at startup, stops on shutdown. Cleanup interval is offsets.retention.check.interval.ms.

Metrics under kafka.server:type=group-coordinator-metrics: streams-group-topology-description-cleanup-cycle-\{rate,count} and streams-group-topology-description-cleanup-eligible-\{rate,count}. The per-group plugin.deleteTopology outcome reuses the delete-\{success,error} sensors introduced by KAFKA-20623.

Tests: cycle behaviour (single-flight, no-op without plugin, per-group plugin.deleteTopology + conditional clear), the offset-expiration-sweep gate, and a broker integration test where a streams group expires naturally and the plugin sees deleteTopology before the group is tombstoned.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)