You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Garvit Sharma <en...@gmail.com> on 2019/06/26 13:53:53 UTC
Kafka Topic Partition Consumer Lags
Hi All,
I can see huge consumer lag in a few partitions of Kafka topic. I need to
know the root cause of this issue.
Please let me know, how to proceed.
Below is sample consumer lag data :
[image: image.png]
Thanks,
Re: Kafka Topic Partition Consumer Lags
Posted by Steve Howard <st...@confluent.io>.
Hi Garvit,
Are the consumers java? If so, can you take a thread dump every five
seconds for 30 seconds total for the affected consumer JVM?
Thanks,
Steve
On Thu, Jun 27, 2019 at 12:00 AM Garvit Sharma <en...@gmail.com> wrote:
> I don't think that is the case. The lag is huge ~10^5 records.
>
> On Thu, Jun 27, 2019 at 9:13 AM Srinath C <sr...@gmail.com> wrote:
>
> > Ok Garvit I still don't see the image but based on these inputs you
> > provided I'm thinking that the possible scenario could be that between
> two
> > polls from the consumer:
> > (a) the number of records added to the partitions already consumed in
> > previous poll is 500 or more (max.poll.records)
> > or
> > (b) the size of records in the partitions already consumed in previous
> poll
> > is around ~50Mb (fetch.max.bytes)
> >
> > Regards,
> > Srinath.
> >
> >
> > On Thu, Jun 27, 2019 at 7:41 AM Garvit Sharma <en...@gmail.com>
> > wrote:
> >
> > > Hi Srinath,
> > >
> > > I have attached the image.
> > >
> > > The partitions belong to the same topic only. I have not explicitly set
> > > max.partition.fetch.bytes or fetch.max.bytes or max.poll.records so it
> > > should take the default values.
> > >
> > > Let me know.
> > >
> > > Thanks,
> > >
> > > On Thu, Jun 27, 2019 at 7:11 AM Srinath C <sr...@gmail.com> wrote:
> > >
> > >> Hi Garvit,
> > >>
> > >> Am unable to see the image you attached for some reason and am not
> able
> > to
> > >> see if the partitions are in the same topic or in different topics.
> > >> Check if any of max.partition.fetch.bytes or fetch.max.bytes or
> > >> max.poll.records configured in your consumer is causing the behaviour.
> > >>
> > >> Regards,
> > >> Srinath
> > >>
> > >>
> > >> On Thu, Jun 27, 2019 at 5:43 AM Peter Bukowinski <pm...@gmail.com>
> > >> wrote:
> > >>
> > >> > Is there a correlation between the lagging partitions and the
> consumer
> > >> > assigned to them?
> > >> >
> > >> > > On Jun 26, 2019, at 4:25 PM, Garvit Sharma <en...@gmail.com>
> > >> wrote:
> > >> > >
> > >> > > Can anyone please help me with this.
> > >> > >
> > >> > > On Wed, Jun 26, 2019 at 8:56 PM Garvit Sharma <
> eng.garvit@gmail.com
> > >
> > >> > wrote:
> > >> > >
> > >> > >> Hey Steve,
> > >> > >>
> > >> > >> I have checked, count of messages on all the partitions are same.
> > >> > >>
> > >> > >> I am still exploring an approach using which the root cause could
> > be
> > >> > >> determined.
> > >> > >>
> > >> > >> Thanks,
> > >> > >>
> > >> > >> On Wed, Jun 26, 2019 at 8:07 PM Garvit Sharma <
> > eng.garvit@gmail.com>
> > >> > >> wrote:
> > >> > >>
> > >> > >>> I am not sure about that. Is there a way to analyse that ?
> > >> > >>>
> > >> > >>> On Wed, Jun 26, 2019 at 7:35 PM Steve Howard <
> > >> > steve.howard@confluent.io>
> > >> > >>> wrote:
> > >> > >>>
> > >> > >>>> Hi Garvit,
> > >> > >>>>
> > >> > >>>> Are the slow partitions "hot", i.e., receiving a lot more
> > messages
> > >> > than
> > >> > >>>> others?
> > >> > >>>>
> > >> > >>>> Thanks,
> > >> > >>>>
> > >> > >>>> Steve
> > >> > >>>>
> > >> > >>>> On Wed, Jun 26, 2019, 9:56 AM Garvit Sharma <
> > eng.garvit@gmail.com
> > >> > wrote:
> > >> > >>>>
> > >> > >>>>> Just to add more details, these consumers are processing the
> > Kafka
> > >> > >>>> events
> > >> > >>>>> and writing to DB(fast write guaranteed).
> > >> > >>>>>
> > >> > >>>>> On Wed, Jun 26, 2019 at 7:23 PM Garvit Sharma <
> > >> eng.garvit@gmail.com>
> > >> > >>>>> wrote:
> > >> > >>>>>
> > >> > >>>>>> Hi All,
> > >> > >>>>>>
> > >> > >>>>>> I can see huge consumer lag in a few partitions of Kafka
> > topic. I
> > >> > >>>> need to
> > >> > >>>>>> know the root cause of this issue.
> > >> > >>>>>>
> > >> > >>>>>> Please let me know, how to proceed.
> > >> > >>>>>>
> > >> > >>>>>> Below is sample consumer lag data :
> > >> > >>>>>>
> > >> > >>>>>> [image: image.png]
> > >> > >>>>>>
> > >> > >>>>>> Thanks,
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>
> > >> > >>>>
> > >> > >>>
> > >> >
> > >> >
> > >>
> > >
> >
>
Re: Kafka Topic Partition Consumer Lags
Posted by Garvit Sharma <en...@gmail.com>.
I don't think that is the case. The lag is huge ~10^5 records.
On Thu, Jun 27, 2019 at 9:13 AM Srinath C <sr...@gmail.com> wrote:
> Ok Garvit I still don't see the image but based on these inputs you
> provided I'm thinking that the possible scenario could be that between two
> polls from the consumer:
> (a) the number of records added to the partitions already consumed in
> previous poll is 500 or more (max.poll.records)
> or
> (b) the size of records in the partitions already consumed in previous poll
> is around ~50Mb (fetch.max.bytes)
>
> Regards,
> Srinath.
>
>
> On Thu, Jun 27, 2019 at 7:41 AM Garvit Sharma <en...@gmail.com>
> wrote:
>
> > Hi Srinath,
> >
> > I have attached the image.
> >
> > The partitions belong to the same topic only. I have not explicitly set
> > max.partition.fetch.bytes or fetch.max.bytes or max.poll.records so it
> > should take the default values.
> >
> > Let me know.
> >
> > Thanks,
> >
> > On Thu, Jun 27, 2019 at 7:11 AM Srinath C <sr...@gmail.com> wrote:
> >
> >> Hi Garvit,
> >>
> >> Am unable to see the image you attached for some reason and am not able
> to
> >> see if the partitions are in the same topic or in different topics.
> >> Check if any of max.partition.fetch.bytes or fetch.max.bytes or
> >> max.poll.records configured in your consumer is causing the behaviour.
> >>
> >> Regards,
> >> Srinath
> >>
> >>
> >> On Thu, Jun 27, 2019 at 5:43 AM Peter Bukowinski <pm...@gmail.com>
> >> wrote:
> >>
> >> > Is there a correlation between the lagging partitions and the consumer
> >> > assigned to them?
> >> >
> >> > > On Jun 26, 2019, at 4:25 PM, Garvit Sharma <en...@gmail.com>
> >> wrote:
> >> > >
> >> > > Can anyone please help me with this.
> >> > >
> >> > > On Wed, Jun 26, 2019 at 8:56 PM Garvit Sharma <eng.garvit@gmail.com
> >
> >> > wrote:
> >> > >
> >> > >> Hey Steve,
> >> > >>
> >> > >> I have checked, count of messages on all the partitions are same.
> >> > >>
> >> > >> I am still exploring an approach using which the root cause could
> be
> >> > >> determined.
> >> > >>
> >> > >> Thanks,
> >> > >>
> >> > >> On Wed, Jun 26, 2019 at 8:07 PM Garvit Sharma <
> eng.garvit@gmail.com>
> >> > >> wrote:
> >> > >>
> >> > >>> I am not sure about that. Is there a way to analyse that ?
> >> > >>>
> >> > >>> On Wed, Jun 26, 2019 at 7:35 PM Steve Howard <
> >> > steve.howard@confluent.io>
> >> > >>> wrote:
> >> > >>>
> >> > >>>> Hi Garvit,
> >> > >>>>
> >> > >>>> Are the slow partitions "hot", i.e., receiving a lot more
> messages
> >> > than
> >> > >>>> others?
> >> > >>>>
> >> > >>>> Thanks,
> >> > >>>>
> >> > >>>> Steve
> >> > >>>>
> >> > >>>> On Wed, Jun 26, 2019, 9:56 AM Garvit Sharma <
> eng.garvit@gmail.com
> >> > wrote:
> >> > >>>>
> >> > >>>>> Just to add more details, these consumers are processing the
> Kafka
> >> > >>>> events
> >> > >>>>> and writing to DB(fast write guaranteed).
> >> > >>>>>
> >> > >>>>> On Wed, Jun 26, 2019 at 7:23 PM Garvit Sharma <
> >> eng.garvit@gmail.com>
> >> > >>>>> wrote:
> >> > >>>>>
> >> > >>>>>> Hi All,
> >> > >>>>>>
> >> > >>>>>> I can see huge consumer lag in a few partitions of Kafka
> topic. I
> >> > >>>> need to
> >> > >>>>>> know the root cause of this issue.
> >> > >>>>>>
> >> > >>>>>> Please let me know, how to proceed.
> >> > >>>>>>
> >> > >>>>>> Below is sample consumer lag data :
> >> > >>>>>>
> >> > >>>>>> [image: image.png]
> >> > >>>>>>
> >> > >>>>>> Thanks,
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>
> >> > >>>>
> >> > >>>
> >> >
> >> >
> >>
> >
>
Re: Kafka Topic Partition Consumer Lags
Posted by Srinath C <sr...@gmail.com>.
Ok Garvit I still don't see the image but based on these inputs you
provided I'm thinking that the possible scenario could be that between two
polls from the consumer:
(a) the number of records added to the partitions already consumed in
previous poll is 500 or more (max.poll.records)
or
(b) the size of records in the partitions already consumed in previous poll
is around ~50Mb (fetch.max.bytes)
Regards,
Srinath.
On Thu, Jun 27, 2019 at 7:41 AM Garvit Sharma <en...@gmail.com> wrote:
> Hi Srinath,
>
> I have attached the image.
>
> The partitions belong to the same topic only. I have not explicitly set
> max.partition.fetch.bytes or fetch.max.bytes or max.poll.records so it
> should take the default values.
>
> Let me know.
>
> Thanks,
>
> On Thu, Jun 27, 2019 at 7:11 AM Srinath C <sr...@gmail.com> wrote:
>
>> Hi Garvit,
>>
>> Am unable to see the image you attached for some reason and am not able to
>> see if the partitions are in the same topic or in different topics.
>> Check if any of max.partition.fetch.bytes or fetch.max.bytes or
>> max.poll.records configured in your consumer is causing the behaviour.
>>
>> Regards,
>> Srinath
>>
>>
>> On Thu, Jun 27, 2019 at 5:43 AM Peter Bukowinski <pm...@gmail.com>
>> wrote:
>>
>> > Is there a correlation between the lagging partitions and the consumer
>> > assigned to them?
>> >
>> > > On Jun 26, 2019, at 4:25 PM, Garvit Sharma <en...@gmail.com>
>> wrote:
>> > >
>> > > Can anyone please help me with this.
>> > >
>> > > On Wed, Jun 26, 2019 at 8:56 PM Garvit Sharma <en...@gmail.com>
>> > wrote:
>> > >
>> > >> Hey Steve,
>> > >>
>> > >> I have checked, count of messages on all the partitions are same.
>> > >>
>> > >> I am still exploring an approach using which the root cause could be
>> > >> determined.
>> > >>
>> > >> Thanks,
>> > >>
>> > >> On Wed, Jun 26, 2019 at 8:07 PM Garvit Sharma <en...@gmail.com>
>> > >> wrote:
>> > >>
>> > >>> I am not sure about that. Is there a way to analyse that ?
>> > >>>
>> > >>> On Wed, Jun 26, 2019 at 7:35 PM Steve Howard <
>> > steve.howard@confluent.io>
>> > >>> wrote:
>> > >>>
>> > >>>> Hi Garvit,
>> > >>>>
>> > >>>> Are the slow partitions "hot", i.e., receiving a lot more messages
>> > than
>> > >>>> others?
>> > >>>>
>> > >>>> Thanks,
>> > >>>>
>> > >>>> Steve
>> > >>>>
>> > >>>> On Wed, Jun 26, 2019, 9:56 AM Garvit Sharma <eng.garvit@gmail.com
>> > wrote:
>> > >>>>
>> > >>>>> Just to add more details, these consumers are processing the Kafka
>> > >>>> events
>> > >>>>> and writing to DB(fast write guaranteed).
>> > >>>>>
>> > >>>>> On Wed, Jun 26, 2019 at 7:23 PM Garvit Sharma <
>> eng.garvit@gmail.com>
>> > >>>>> wrote:
>> > >>>>>
>> > >>>>>> Hi All,
>> > >>>>>>
>> > >>>>>> I can see huge consumer lag in a few partitions of Kafka topic. I
>> > >>>> need to
>> > >>>>>> know the root cause of this issue.
>> > >>>>>>
>> > >>>>>> Please let me know, how to proceed.
>> > >>>>>>
>> > >>>>>> Below is sample consumer lag data :
>> > >>>>>>
>> > >>>>>> [image: image.png]
>> > >>>>>>
>> > >>>>>> Thanks,
>> > >>>>>>
>> > >>>>>>
>> > >>>>>
>> > >>>>
>> > >>>
>> >
>> >
>>
>
Re: Kafka Topic Partition Consumer Lags
Posted by Garvit Sharma <en...@gmail.com>.
Hi Srinath,
I have attached the image.
The partitions belong to the same topic only. I have not explicitly set
max.partition.fetch.bytes or fetch.max.bytes or max.poll.records so it
should take the default values.
Let me know.
Thanks,
On Thu, Jun 27, 2019 at 7:11 AM Srinath C <sr...@gmail.com> wrote:
> Hi Garvit,
>
> Am unable to see the image you attached for some reason and am not able to
> see if the partitions are in the same topic or in different topics.
> Check if any of max.partition.fetch.bytes or fetch.max.bytes or
> max.poll.records configured in your consumer is causing the behaviour.
>
> Regards,
> Srinath
>
>
> On Thu, Jun 27, 2019 at 5:43 AM Peter Bukowinski <pm...@gmail.com> wrote:
>
> > Is there a correlation between the lagging partitions and the consumer
> > assigned to them?
> >
> > > On Jun 26, 2019, at 4:25 PM, Garvit Sharma <en...@gmail.com>
> wrote:
> > >
> > > Can anyone please help me with this.
> > >
> > > On Wed, Jun 26, 2019 at 8:56 PM Garvit Sharma <en...@gmail.com>
> > wrote:
> > >
> > >> Hey Steve,
> > >>
> > >> I have checked, count of messages on all the partitions are same.
> > >>
> > >> I am still exploring an approach using which the root cause could be
> > >> determined.
> > >>
> > >> Thanks,
> > >>
> > >> On Wed, Jun 26, 2019 at 8:07 PM Garvit Sharma <en...@gmail.com>
> > >> wrote:
> > >>
> > >>> I am not sure about that. Is there a way to analyse that ?
> > >>>
> > >>> On Wed, Jun 26, 2019 at 7:35 PM Steve Howard <
> > steve.howard@confluent.io>
> > >>> wrote:
> > >>>
> > >>>> Hi Garvit,
> > >>>>
> > >>>> Are the slow partitions "hot", i.e., receiving a lot more messages
> > than
> > >>>> others?
> > >>>>
> > >>>> Thanks,
> > >>>>
> > >>>> Steve
> > >>>>
> > >>>> On Wed, Jun 26, 2019, 9:56 AM Garvit Sharma <eng.garvit@gmail.com
> > wrote:
> > >>>>
> > >>>>> Just to add more details, these consumers are processing the Kafka
> > >>>> events
> > >>>>> and writing to DB(fast write guaranteed).
> > >>>>>
> > >>>>> On Wed, Jun 26, 2019 at 7:23 PM Garvit Sharma <
> eng.garvit@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hi All,
> > >>>>>>
> > >>>>>> I can see huge consumer lag in a few partitions of Kafka topic. I
> > >>>> need to
> > >>>>>> know the root cause of this issue.
> > >>>>>>
> > >>>>>> Please let me know, how to proceed.
> > >>>>>>
> > >>>>>> Below is sample consumer lag data :
> > >>>>>>
> > >>>>>> [image: image.png]
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> >
> >
>
Re: Kafka Topic Partition Consumer Lags
Posted by Srinath C <sr...@gmail.com>.
Hi Garvit,
Am unable to see the image you attached for some reason and am not able to
see if the partitions are in the same topic or in different topics.
Check if any of max.partition.fetch.bytes or fetch.max.bytes or
max.poll.records configured in your consumer is causing the behaviour.
Regards,
Srinath
On Thu, Jun 27, 2019 at 5:43 AM Peter Bukowinski <pm...@gmail.com> wrote:
> Is there a correlation between the lagging partitions and the consumer
> assigned to them?
>
> > On Jun 26, 2019, at 4:25 PM, Garvit Sharma <en...@gmail.com> wrote:
> >
> > Can anyone please help me with this.
> >
> > On Wed, Jun 26, 2019 at 8:56 PM Garvit Sharma <en...@gmail.com>
> wrote:
> >
> >> Hey Steve,
> >>
> >> I have checked, count of messages on all the partitions are same.
> >>
> >> I am still exploring an approach using which the root cause could be
> >> determined.
> >>
> >> Thanks,
> >>
> >> On Wed, Jun 26, 2019 at 8:07 PM Garvit Sharma <en...@gmail.com>
> >> wrote:
> >>
> >>> I am not sure about that. Is there a way to analyse that ?
> >>>
> >>> On Wed, Jun 26, 2019 at 7:35 PM Steve Howard <
> steve.howard@confluent.io>
> >>> wrote:
> >>>
> >>>> Hi Garvit,
> >>>>
> >>>> Are the slow partitions "hot", i.e., receiving a lot more messages
> than
> >>>> others?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Steve
> >>>>
> >>>> On Wed, Jun 26, 2019, 9:56 AM Garvit Sharma <eng.garvit@gmail.com
> wrote:
> >>>>
> >>>>> Just to add more details, these consumers are processing the Kafka
> >>>> events
> >>>>> and writing to DB(fast write guaranteed).
> >>>>>
> >>>>> On Wed, Jun 26, 2019 at 7:23 PM Garvit Sharma <en...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi All,
> >>>>>>
> >>>>>> I can see huge consumer lag in a few partitions of Kafka topic. I
> >>>> need to
> >>>>>> know the root cause of this issue.
> >>>>>>
> >>>>>> Please let me know, how to proceed.
> >>>>>>
> >>>>>> Below is sample consumer lag data :
> >>>>>>
> >>>>>> [image: image.png]
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
>
>
Re: Kafka Topic Partition Consumer Lags
Posted by Peter Bukowinski <pm...@gmail.com>.
Is there a correlation between the lagging partitions and the consumer assigned to them?
> On Jun 26, 2019, at 4:25 PM, Garvit Sharma <en...@gmail.com> wrote:
>
> Can anyone please help me with this.
>
> On Wed, Jun 26, 2019 at 8:56 PM Garvit Sharma <en...@gmail.com> wrote:
>
>> Hey Steve,
>>
>> I have checked, count of messages on all the partitions are same.
>>
>> I am still exploring an approach using which the root cause could be
>> determined.
>>
>> Thanks,
>>
>> On Wed, Jun 26, 2019 at 8:07 PM Garvit Sharma <en...@gmail.com>
>> wrote:
>>
>>> I am not sure about that. Is there a way to analyse that ?
>>>
>>> On Wed, Jun 26, 2019 at 7:35 PM Steve Howard <st...@confluent.io>
>>> wrote:
>>>
>>>> Hi Garvit,
>>>>
>>>> Are the slow partitions "hot", i.e., receiving a lot more messages than
>>>> others?
>>>>
>>>> Thanks,
>>>>
>>>> Steve
>>>>
>>>> On Wed, Jun 26, 2019, 9:56 AM Garvit Sharma <eng.garvit@gmail.com wrote:
>>>>
>>>>> Just to add more details, these consumers are processing the Kafka
>>>> events
>>>>> and writing to DB(fast write guaranteed).
>>>>>
>>>>> On Wed, Jun 26, 2019 at 7:23 PM Garvit Sharma <en...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I can see huge consumer lag in a few partitions of Kafka topic. I
>>>> need to
>>>>>> know the root cause of this issue.
>>>>>>
>>>>>> Please let me know, how to proceed.
>>>>>>
>>>>>> Below is sample consumer lag data :
>>>>>>
>>>>>> [image: image.png]
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
Re: Kafka Topic Partition Consumer Lags
Posted by Garvit Sharma <en...@gmail.com>.
Can anyone please help me with this.
On Wed, Jun 26, 2019 at 8:56 PM Garvit Sharma <en...@gmail.com> wrote:
> Hey Steve,
>
> I have checked, count of messages on all the partitions are same.
>
> I am still exploring an approach using which the root cause could be
> determined.
>
> Thanks,
>
> On Wed, Jun 26, 2019 at 8:07 PM Garvit Sharma <en...@gmail.com>
> wrote:
>
>> I am not sure about that. Is there a way to analyse that ?
>>
>> On Wed, Jun 26, 2019 at 7:35 PM Steve Howard <st...@confluent.io>
>> wrote:
>>
>>> Hi Garvit,
>>>
>>> Are the slow partitions "hot", i.e., receiving a lot more messages than
>>> others?
>>>
>>> Thanks,
>>>
>>> Steve
>>>
>>> On Wed, Jun 26, 2019, 9:56 AM Garvit Sharma <eng.garvit@gmail.com wrote:
>>>
>>> > Just to add more details, these consumers are processing the Kafka
>>> events
>>> > and writing to DB(fast write guaranteed).
>>> >
>>> > On Wed, Jun 26, 2019 at 7:23 PM Garvit Sharma <en...@gmail.com>
>>> > wrote:
>>> >
>>> >> Hi All,
>>> >>
>>> >> I can see huge consumer lag in a few partitions of Kafka topic. I
>>> need to
>>> >> know the root cause of this issue.
>>> >>
>>> >> Please let me know, how to proceed.
>>> >>
>>> >> Below is sample consumer lag data :
>>> >>
>>> >> [image: image.png]
>>> >>
>>> >> Thanks,
>>> >>
>>> >>
>>> >
>>>
>>
Re: Kafka Topic Partition Consumer Lags
Posted by Garvit Sharma <en...@gmail.com>.
Hey Steve,
I have checked, count of messages on all the partitions are same.
I am still exploring an approach using which the root cause could be
determined.
Thanks,
On Wed, Jun 26, 2019 at 8:07 PM Garvit Sharma <en...@gmail.com> wrote:
> I am not sure about that. Is there a way to analyse that ?
>
> On Wed, Jun 26, 2019 at 7:35 PM Steve Howard <st...@confluent.io>
> wrote:
>
>> Hi Garvit,
>>
>> Are the slow partitions "hot", i.e., receiving a lot more messages than
>> others?
>>
>> Thanks,
>>
>> Steve
>>
>> On Wed, Jun 26, 2019, 9:56 AM Garvit Sharma <eng.garvit@gmail.com wrote:
>>
>> > Just to add more details, these consumers are processing the Kafka
>> events
>> > and writing to DB(fast write guaranteed).
>> >
>> > On Wed, Jun 26, 2019 at 7:23 PM Garvit Sharma <en...@gmail.com>
>> > wrote:
>> >
>> >> Hi All,
>> >>
>> >> I can see huge consumer lag in a few partitions of Kafka topic. I need
>> to
>> >> know the root cause of this issue.
>> >>
>> >> Please let me know, how to proceed.
>> >>
>> >> Below is sample consumer lag data :
>> >>
>> >> [image: image.png]
>> >>
>> >> Thanks,
>> >>
>> >>
>> >
>>
>
Re: Kafka Topic Partition Consumer Lags
Posted by Garvit Sharma <en...@gmail.com>.
I am not sure about that. Is there a way to analyse that ?
On Wed, Jun 26, 2019 at 7:35 PM Steve Howard <st...@confluent.io>
wrote:
> Hi Garvit,
>
> Are the slow partitions "hot", i.e., receiving a lot more messages than
> others?
>
> Thanks,
>
> Steve
>
> On Wed, Jun 26, 2019, 9:56 AM Garvit Sharma <eng.garvit@gmail.com wrote:
>
> > Just to add more details, these consumers are processing the Kafka events
> > and writing to DB(fast write guaranteed).
> >
> > On Wed, Jun 26, 2019 at 7:23 PM Garvit Sharma <en...@gmail.com>
> > wrote:
> >
> >> Hi All,
> >>
> >> I can see huge consumer lag in a few partitions of Kafka topic. I need
> to
> >> know the root cause of this issue.
> >>
> >> Please let me know, how to proceed.
> >>
> >> Below is sample consumer lag data :
> >>
> >> [image: image.png]
> >>
> >> Thanks,
> >>
> >>
> >
>
Re: Kafka Topic Partition Consumer Lags
Posted by Steve Howard <st...@confluent.io>.
Hi Garvit,
Are the slow partitions "hot", i.e., receiving a lot more messages than
others?
Thanks,
Steve
On Wed, Jun 26, 2019, 9:56 AM Garvit Sharma <eng.garvit@gmail.com wrote:
> Just to add more details, these consumers are processing the Kafka events
> and writing to DB(fast write guaranteed).
>
> On Wed, Jun 26, 2019 at 7:23 PM Garvit Sharma <en...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I can see huge consumer lag in a few partitions of Kafka topic. I need to
>> know the root cause of this issue.
>>
>> Please let me know, how to proceed.
>>
>> Below is sample consumer lag data :
>>
>> [image: image.png]
>>
>> Thanks,
>>
>>
>
Re: Kafka Topic Partition Consumer Lags
Posted by Garvit Sharma <en...@gmail.com>.
Just to add more details, these consumers are processing the Kafka events
and writing to DB(fast write guaranteed).
On Wed, Jun 26, 2019 at 7:23 PM Garvit Sharma <en...@gmail.com> wrote:
> Hi All,
>
> I can see huge consumer lag in a few partitions of Kafka topic. I need to
> know the root cause of this issue.
>
> Please let me know, how to proceed.
>
> Below is sample consumer lag data :
>
> [image: image.png]
>
> Thanks,
>
>