Re: Some of the Kafka consumers are not getting the messages from the broker

[Karts <ka...@gmail.com>]

unsubscribe

On Wed, Aug 21, 2019 at 2:23 AM sampath kumar <sa...@gmail.com> wrote:

> Hi,
>
> Using Broker 5.3.0, new consumers(Consumers managed by brokers). Brokers
> are deployed in a Kubernetes environment
>
> Number of brokers : 3, Number of 3 Zookeeper setup
>
> One of the Topic "inventory.request" we have 3 replication, with insync
> replicas configured as 2 and partition count is 1024
>
> We have 20 instances of microservice subscribe to the above topic, each
> instance will have 48 consumers registered as a group "agent.group.inv"
>
> Issue :
>
> Here some times a couple of the consumers suddenly stopped receiving the
> request, and lag seems to keep increasing. Only option to recover it
> restart the consumers and invokes rebalancing
>
> ``` agent.group.inv inventory.request 543        17423           17612
>       189             agent19.inv.35-6e6eb252-8d26-489b-8d7f-53b25f182f30
>     /10.200.187.103 agent19.inv.35 ```
>
> we checked the thread dump of the consumer, the consumer keeps performing
> polling and assigned with partitions, However not receiving the any
> messages
>
> ``` "inventory.request-agent19.inv.35" #499 prio=1 os_prio=4
> tid=0x00007f88a855b000 nid=0x389 runnable [0x00007f87e8be6000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000000aa502730> (a sun.nio.ch.Util$3)
>         - locked <0x00000000aa5026b0> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000aa502668> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at
> org.apache.kafka.common.network.Selector.select(Selector.java:794)
>         at org.apache.kafka.common.network.Selector.poll(Selector.java:467)
>         at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:539)
>         at
>
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
>         at
>
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
>         at
>
> org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1281)
>         at
>
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1225)
>         at
>
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159)
> ```
>
>
> No errors are observed consumer client, brokers and also resource issue not
> seen.
>
> Can you please help us in identifying the root cause for this consumer
> client behavior?
>
> Please let me know if any other details required?
>
> --
> Regards,
> Sampath
>