[jira] [Resolved] (KAFKA-6287) Inconsistent protocol type for empty consumer groups

["Jason Gustafson (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-6287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Gustafson resolved KAFKA-6287.
------------------------------------
    Resolution: Fixed

> Inconsistent protocol type for empty consumer groups
> ----------------------------------------------------
>
>                 Key: KAFKA-6287
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6287
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 1.0.0
>            Reporter: Ryan Leslie
>            Assignee: Jason Gustafson
>            Priority: Minor
>             Fix For: 1.0.1
>
>
> When a consumer is created for a new group, the group metadata's protocol type is set to 'consumer' and this is stored both in __consumer_offsets as well as in the coordinator's local cache.
> If the consumer leaves the group and the group becomes empty, ListGroups requests will continue to show the group as type 'consumer', and as such kafka-consumer-groups.sh will show it via --list.
> However, if the coordinator (broker) node is killed and a new coordinator is elected, when the GroupMetadataManager loads the group from __consumer_offsets into its cache, it will not set the protocolType if there are no active consumers. As a result, the group's protocolType will now become the empty string (UNKNOWN_PROTOCOL_TYPE), and kafka-consumer-groups.sh will no longer show the group.
> Ideally bouncing a broker should not result in the group's protocol changing. protocolType can be set in GroupMetadataManager.readGroupMessageValue() to always reflect what's present in the persistent metadata (__consumer_offsets) regardless if there are active members.
> In general, things can get confusing when distinguishing between 'consumer' and non-consumer groups. For example, DescribeGroups and OffsetFetchRequest does not filter on protocol type, so it's possible for kafka-consumer-groups.sh to describe groups (--describe) without actually being able to list them. In the protocol guide, OffsetFetchRequest / OffsetCommitRequest have their parameters listed as 'ConsumerGroup', but in reality these can be used for groups of unknown type as well. For instance, a consumer group can be copied by finding a coordinator (GroupCoordinatorRequest / FindCoordinatorRequest) and sending an OffsetCommitRequest. The group will be auto-created with an empty protocol. Although this is arguably correct, the group will now exist but not be  a proper 'consumer' group until later when there is a JoinGroupRequest. Again, this can be confusing as far as categorization / visibility of the group is concerned. A group can instead be copied more directly by creating a consumer and calling commitSync (as kafka-consumer-groups.sh), but this does involve extra connections / requests and so is a little slower when trying to keep a large number of groups in sync in real-time across clusters.
> If we want to make it easier to keep consumer groups consistent, options include allowing groups to be explicitly created with a protocol type instead of only lazily, or for groups created outside of JoinGroupRequest the coordinator can gain a broker config to set a default protocol type for groups. This would default to 'consumer'. This sort of setting might be confusing for users though, since implicitly created groups is certainly not the norm.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)