Kafka partitions are piled up for Consumer

["Ghosh, Achintya (Contractor)" <Ac...@comcast.com>]

Hi,

If we have some high response in backend( let's say database high response), the few of the partitions messages are piled up. Once the backend gets normal still those partitions are not processing any message and it creates a huge lag.

Any idea why it happened? We thought once the backend gets back to normal those partitions will be processing but its not happening. Somehow some threads are stuck.

Thanks

RE: Kafka partitions are piled up for Consumer

["Ghosh, Achintya (Contractor)" <Ac...@comcast.com>]

This is not the case for us. Anyone has any other idea please? How we can put timeout for Kafka thread?

Thanks!

-----Original Message-----
From: Naanu Bora [mailto:naanubora@gmail.com] 
Sent: Thursday, July 06, 2017 1:41 PM
To: users@kafka.apache.org
Subject: Re: Kafka partitions are piled up for Consumer

I am observing the same with our .10.2 cluster where consumers are hanged for those partitions where the current offset's data gets deleted due to retention. It looks like a bug to me.

Thanks!

On Jul 6, 2017 9:23 AM, "Ghosh, Achintya (Contractor)" < Achintya_Ghosh@comcast.com> wrote:

> Hi,
>
> If we have some high response in backend( let's say database high 
> response), the few of the partitions messages are piled up. Once the 
> backend gets normal still those partitions are not processing any 
> message and it creates a huge lag.
>
> Any idea why it happened? We thought once the backend gets back to 
> normal those partitions will be processing but its not happening. 
> Somehow some threads are stuck.
>
> Thanks
>
>

Re: Kafka partitions are piled up for Consumer

[Naanu Bora <na...@gmail.com>]

I am observing the same with our .10.2 cluster where consumers are hanged
for those partitions where the current offset's data gets deleted due to
retention. It looks like a bug to me.

Thanks!

On Jul 6, 2017 9:23 AM, "Ghosh, Achintya (Contractor)" <
Achintya_Ghosh@comcast.com> wrote:

> Hi,
>
> If we have some high response in backend( let's say database high
> response), the few of the partitions messages are piled up. Once the
> backend gets normal still those partitions are not processing any message
> and it creates a huge lag.
>
> Any idea why it happened? We thought once the backend gets back to normal
> those partitions will be processing but its not happening. Somehow some
> threads are stuck.
>
> Thanks
>
>