Unable to put more than 3 consumers in a single process

[Lahiru Chandima <la...@gmail.com>]

 Hi,

I am trying to place multiple kafka consumers of a single consumer group in
the same process. But, I am unable to put more than 3 consumers in the same
process.The topic which the consumer group tries to read has 10
partitions.When I try to attach the 4th consumer, it leaves the group after
the first poll, with below message. No partitions get assigned to the
consumer.

[ForkJoinPool.commonPool-worker-3] INFO
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer
clientId=consumer-group65-4, groupId=group65] Member
consumer-group65-4-b148f7ec-b852-46ed-8514-c3af3bd78395 sending LeaveGroup
request to coordinator my-cluster:9092 (id: 2147483645 rack: null) due to
the consumer is being closed

What is the reason for this? Is there any way to put more than 3 consumers
in the same process?

Thank you
-- 
Lahiru Chandima

Re: Unable to put more than 3 consumers in a single process

[Sophie Blee-Goldman <so...@confluent.io.INVALID>]

Hey Lahiru,

As the log message describes, the member is leaving the group because the
consumer
has been closed by your process/application. You need to track down where
in your
application code the Consumer#close method is being called, and why.

There's no technical limit on how many consumers can be in the same
process. Of course
there is a practical limit, eg if you try to start up millions of consumers
then they may struggle
to stay in the group for example due to the millions of background
heartbeat threads competing
for cpu and network resources. But I see no reason why a modern machine
would be unable
to handle just 4 consumers, and anyways the log message you provided makes
it clear that
the consumer is only leaving the group due to being manually closed.

Hope that helps,
Sophie

On Thu, May 6, 2021 at 7:29 AM Lahiru Chandima <la...@gmail.com>
wrote:

>  Hi,
>
> I am trying to place multiple kafka consumers of a single consumer group in
> the same process. But, I am unable to put more than 3 consumers in the same
> process.The topic which the consumer group tries to read has 10
> partitions.When I try to attach the 4th consumer, it leaves the group after
> the first poll, with below message. No partitions get assigned to the
> consumer.
>
> [ForkJoinPool.commonPool-worker-3] INFO
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer
> clientId=consumer-group65-4, groupId=group65] Member
> consumer-group65-4-b148f7ec-b852-46ed-8514-c3af3bd78395 sending LeaveGroup
> request to coordinator my-cluster:9092 (id: 2147483645 rack: null) due to
> the consumer is being closed
>
> What is the reason for this? Is there any way to put more than 3 consumers
> in the same process?
>
> Thank you
> --
> Lahiru Chandima
>