[jira] [Created] (KAFKA-9043) Problem committing offsets when using consumer subscribe and assign method on different topics with same group.id

["Michael Arndt (Jira)" <ji...@apache.org>]

Michael Arndt created KAFKA-9043:
------------------------------------

             Summary: Problem committing offsets when using consumer subscribe and assign method on different topics with same group.id
                 Key: KAFKA-9043
                 URL: https://issues.apache.org/jira/browse/KAFKA-9043
             Project: Kafka
          Issue Type: Bug
          Components: consumer
    Affects Versions: 2.3.0
         Environment: not relevant
            Reporter: Michael Arndt


I am using a consumer with group.id "myGroup" on an *_empty_* topic "election" with 24 partitions in subscribe mode. The poll loop is a noop and is for rebalancing only. On new assigments (only the partitions are relevant) I start two consumers per assigned partition on other topics ("foo" and "bar") using assignments on a specific partition/topic.

Summary:

consumer1: election-0, election-1 (subscribe)
consumer2: foo-0 (assign)
consumer3: bar-0 (assign)
consumer4: foo-1 (assign)
consumer5: bar-1 (assign)

The reason for this setup is a join on time and a certain key in the topics foo and bar, where the content in one partition is complete for the key.

However, consumer1 never commits since the topic is empty and just for rebalancing issues that will guarantee that I will always read from the same partitions from foo and bar in this service instance.

When trying to commit consumer2-4 (they are using the same group-id as consumer1) the ClientCoordinator reports an error telling the consumer is not responsible for this topic and partition. This is an error raised at the server.

When using different group-ids for consumer1 and 2-4 the commit works fine.

This behaviour is somehow reasonable since the server only knows about the subscription, but since assignment and subscription run on different topics it's at least a lack in documentation.

As this will also happen when using assign and subscribe from different clients on differnt topis one would not be able to use a common group-id for identification issues (e.g. team related; useful when working with ACLs in an environment where on kafka serves many teams and you want to limit the number of group ids managed in the ACLs).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)