[I] Kafka gateway: list offsets-only groups as Empty, and serve DeleteGroups and OffsetDelete [iggy]

["krishvishal (via GitHub)" <gi...@apache.org>]

krishvishal opened a new issue, #4511:
URL: https://github.com/apache/iggy/issues/4511

   Follow-up to #3548.
   
   ## Summary
   
   #3548 added ListGroups (16) and DescribeGroups (15) from the in-memory coordinator. Three gaps remain:
   
   - A group with committed offsets and no live members is not listed, and DescribeGroups reports it `Dead`. Kafka lists it as `Empty`. A gateway restart also empties the list.
   - DeleteGroups (42) and OffsetDelete (47) are not served.
   - `client_id` and `client_host` are empty, so `kafka-consumer-groups --describe` shows no consumer hosts.
   
   Who hits it: lag exporters (kafka_exporter, kminion, kafka-lag-exporter and the OpenTelemetry Kafka metrics receiver), UIs, and `kafka-consumer-groups --list`, `--delete` and `--delete-offsets`. When the consumers stop, a lag alert loses its group. That is the moment the lag starts to grow. Azure Event Hubs had the same gap, and it broke `kafka-consumer-groups` and lag exporters.
   
   ## Scope
   
   - [ ] List and describe a group that holds offsets in Iggy groups named `kafka.cg.<group>` as `Empty`
   - [ ] Limit the cost of that scan. It takes one Iggy call per topic, and kafka-ui calls ListGroups every 30 seconds
   - [ ] DeleteGroups v0-3: delete `kafka.cg.<group>` on each topic. While members are live, answer `NON_EMPTY_GROUP` (68). If nothing exists, answer `GROUP_ID_NOT_FOUND` (69)
   - [ ] OffsetDelete v0-1: delete the offsets of the named partitions. Answer `GROUP_SUBSCRIBED_TO_TOPIC` (86) while a live member consumes the topic
   - [ ] Keep each member's client id and peer address
   - [ ] Update `gateways/kafka/docs/CONSUMER_GROUPS.md`, including its `TODO(#3542)` and the line that says OffsetFetch is not implemented
   
   ## Acceptance criteria
   
   - [ ] After its consumers stop, a group still shows as `Empty` in `kafka-consumer-groups --list` and `--describe --state`
   - [ ] `kafka-consumer-groups --delete` and `--delete-offsets` work, and refuse while the group is live
   - [ ] kafka_exporter reports lag for a group with no live members
   
   ## References
   
   - `gateways/kafka/docs/CONSUMER_GROUPS.md`
   - [ConsumerGroupCommand.java:84-113](https://github.com/apache/kafka/blob/b15258c7398889248d7fbf2a2218cc01346ebf92/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L84-L113)
   - kafka-ui [ReactiveAdminClient.java](https://github.com/kafbat/kafka-ui/blob/0c481c0ba60aabe942387e82963f2e0cc1352543/api/src/main/java/io/kafbat/ui/service/ReactiveAdminClient.java), kafka_exporter [kafka_exporter.go](https://github.com/danielqsj/kafka_exporter/blob/v1.10.0/kafka_exporter.go)
   - Azure Event Hubs: https://github.com/Azure/azure-event-hubs-for-kafka/issues/100, https://github.com/Azure/azure-event-hubs-for-kafka/issues/143
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org