You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ashutosh singh <ge...@gmail.com> on 2020/06/20 08:31:16 UTC
_consumer_offsets topic produce rate is dam high
Hi Guys,
Hope you all are doing well.
All of sudden I see very high throughput for _consumer_offsets topic. it
is around 30-40K per second. What could be the reason for such a high
rate ? Do I need to be concerned around this ?
[image: image.png]
This happens for an hour and after that it goes down. Now it is happening
once or twice daily.
I have 8 node cluster , 1000+ topics and 644 consumer groups.
All nodes have almost equal number of lead partition across nodes and
almost equal partition on all nodes.
Kafka version : 2.1.1
If you see above graph, there are no other topics where messages are more
than 300 Message/sec. only _consumer_offsets is having so high through
put.
I know this is internal topic ans store metadata of topics and consumer
information. But I don't see anything abnormal in my cluster then why
_consumer_offset topic is going crazy. What is going on here ?
Any help will be appreciated.
--
Thanx & Regard
Ashutosh Singh
08151945559
Re: _consumer_offsets topic produce rate is dam high
Posted by Ashutosh singh <ge...@gmail.com>.
Thank Karolis and Liam for your help.
Karolis, there are 600+ consumers , How do I find which consumers have
this kind of crazy configuration? Consumers are maintain by development
team and so is there a way I can identify ?
@Liam,
I did run above command and grep the output for a particular topic. I am
not sure what to figure out from the output. I have attached the sample
output. Could you please let me know how to interpret this or any
documentation to help to understand ?
On Thu, Jun 25, 2020 at 3:16 AM Liam Clarke-Hutchinson <
liam.clarke@adscale.co.nz> wrote:
> ...the classpath is required to make the formatter available to the console
> producer consumer...
>
> On Thu, Jun 25, 2020 at 9:44 AM Liam Clarke-Hutchinson <
> liam.clarke@adscale.co.nz> wrote:
>
> > To investigate, you can read the messages on the topic to figure out how
> > consumer groups are committing. They're stored in a binary format, but
> you
> > can make them human readable using a formatter. Here's what I did last
> time
> > to do this - the classpath is required to make the formatter available to
> > the console producer, but you could do this in code with Kafka as a
> > dependency:
> >
> > export CLASSPATH=/home/development/kafka_2.12-2.4.1/libs/* \
> > && /home/development/kafka_2.12-2.4.1/bin/kafka-console-consumer.sh \
> > --bootstrap-server kafka01:9092 \
> > --topic __consumer_offsets \
> > --from-beginning \
> > --formatter
> "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" >
> offsets.txt
> >
> > In case the email formatting garbles the above, here it is in a gist
> also <
> https://gist.github.com/LiamClarkeFMG/4a0f038a8d1782b1348e34b3ec471c21>.
> You can then parse and analyse the output data using your preferred
> scripting language.
> >
> > Hope that helps,
> >
> > Kind regards,
> >
> > Liam Clarke-Hutchinson
> >
> >
> >
> >
> > On Thu, Jun 25, 2020 at 3:25 AM Karolis Pocius
> > <ka...@sentiance.com.invalid> wrote:
> >
> >> Check if any of your consumers have auto commit turned off and instead
> >> commit after processing each message.
> >>
> >> Also, even if all consumers are using auto commit, maybe some of them
> have
> >> the interval set to something crazy low like 1 ms.
> >>
> >> On Sat, Jun 20, 2020 at 8:31 PM Ashutosh singh <ge...@gmail.com>
> >> wrote:
> >>
> >> > Hi Guys,
> >> >
> >> > Hope you all are doing well.
> >> >
> >> > All of sudden I see very high throughput for _consumer_offsets topic.
> >> it
> >> > is around 30-40K per second. What could be the reason for such a
> high
> >> > rate ? Do I need to be concerned around this ?
> >> >
> >> > [image: image.png]
> >> >
> >> >
> >> > This happens for an hour and after that it goes down. Now it is
> >> happening
> >> > once or twice daily.
> >> >
> >> > I have 8 node cluster , 1000+ topics and 644 consumer groups.
> >> > All nodes have almost equal number of lead partition across nodes and
> >> > almost equal partition on all nodes.
> >> >
> >> > Kafka version : 2.1.1
> >> >
> >> > If you see above graph, there are no other topics where messages are
> >> more
> >> > than 300 Message/sec. only _consumer_offsets is having so high
> through
> >> > put.
> >> >
> >> > I know this is internal topic ans store metadata of topics and
> consumer
> >> > information. But I don't see anything abnormal in my cluster then why
> >> > _consumer_offset topic is going crazy. What is going on here ?
> >> >
> >> > Any help will be appreciated.
> >> >
> >> > --
> >> > Thanx & Regard
> >> > Ashutosh Singh
> >> > 08151945559
> >> >
> >> >
> >>
> >
>
--
Thanx & Regard
Ashutosh Singh
08151945559
Re: _consumer_offsets topic produce rate is dam high
Posted by Liam Clarke-Hutchinson <li...@adscale.co.nz>.
...the classpath is required to make the formatter available to the console
producer consumer...
On Thu, Jun 25, 2020 at 9:44 AM Liam Clarke-Hutchinson <
liam.clarke@adscale.co.nz> wrote:
> To investigate, you can read the messages on the topic to figure out how
> consumer groups are committing. They're stored in a binary format, but you
> can make them human readable using a formatter. Here's what I did last time
> to do this - the classpath is required to make the formatter available to
> the console producer, but you could do this in code with Kafka as a
> dependency:
>
> export CLASSPATH=/home/development/kafka_2.12-2.4.1/libs/* \
> && /home/development/kafka_2.12-2.4.1/bin/kafka-console-consumer.sh \
> --bootstrap-server kafka01:9092 \
> --topic __consumer_offsets \
> --from-beginning \
> --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" > offsets.txt
>
> In case the email formatting garbles the above, here it is in a gist also <https://gist.github.com/LiamClarkeFMG/4a0f038a8d1782b1348e34b3ec471c21>. You can then parse and analyse the output data using your preferred scripting language.
>
> Hope that helps,
>
> Kind regards,
>
> Liam Clarke-Hutchinson
>
>
>
>
> On Thu, Jun 25, 2020 at 3:25 AM Karolis Pocius
> <ka...@sentiance.com.invalid> wrote:
>
>> Check if any of your consumers have auto commit turned off and instead
>> commit after processing each message.
>>
>> Also, even if all consumers are using auto commit, maybe some of them have
>> the interval set to something crazy low like 1 ms.
>>
>> On Sat, Jun 20, 2020 at 8:31 PM Ashutosh singh <ge...@gmail.com>
>> wrote:
>>
>> > Hi Guys,
>> >
>> > Hope you all are doing well.
>> >
>> > All of sudden I see very high throughput for _consumer_offsets topic.
>> it
>> > is around 30-40K per second. What could be the reason for such a high
>> > rate ? Do I need to be concerned around this ?
>> >
>> > [image: image.png]
>> >
>> >
>> > This happens for an hour and after that it goes down. Now it is
>> happening
>> > once or twice daily.
>> >
>> > I have 8 node cluster , 1000+ topics and 644 consumer groups.
>> > All nodes have almost equal number of lead partition across nodes and
>> > almost equal partition on all nodes.
>> >
>> > Kafka version : 2.1.1
>> >
>> > If you see above graph, there are no other topics where messages are
>> more
>> > than 300 Message/sec. only _consumer_offsets is having so high through
>> > put.
>> >
>> > I know this is internal topic ans store metadata of topics and consumer
>> > information. But I don't see anything abnormal in my cluster then why
>> > _consumer_offset topic is going crazy. What is going on here ?
>> >
>> > Any help will be appreciated.
>> >
>> > --
>> > Thanx & Regard
>> > Ashutosh Singh
>> > 08151945559
>> >
>> >
>>
>
Re: _consumer_offsets topic produce rate is dam high
Posted by Liam Clarke-Hutchinson <li...@adscale.co.nz>.
To investigate, you can read the messages on the topic to figure out how
consumer groups are committing. They're stored in a binary format, but you
can make them human readable using a formatter. Here's what I did last time
to do this - the classpath is required to make the formatter available to
the console producer, but you could do this in code with Kafka as a
dependency:
export CLASSPATH=/home/development/kafka_2.12-2.4.1/libs/* \
&& /home/development/kafka_2.12-2.4.1/bin/kafka-console-consumer.sh \
--bootstrap-server kafka01:9092 \
--topic __consumer_offsets \
--from-beginning \
--formatter
"kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter"
> offsets.txt
In case the email formatting garbles the above, here it is in a gist
also <https://gist.github.com/LiamClarkeFMG/4a0f038a8d1782b1348e34b3ec471c21>.
You can then parse and analyse the output data using your preferred
scripting language.
Hope that helps,
Kind regards,
Liam Clarke-Hutchinson
On Thu, Jun 25, 2020 at 3:25 AM Karolis Pocius
<ka...@sentiance.com.invalid> wrote:
> Check if any of your consumers have auto commit turned off and instead
> commit after processing each message.
>
> Also, even if all consumers are using auto commit, maybe some of them have
> the interval set to something crazy low like 1 ms.
>
> On Sat, Jun 20, 2020 at 8:31 PM Ashutosh singh <ge...@gmail.com> wrote:
>
> > Hi Guys,
> >
> > Hope you all are doing well.
> >
> > All of sudden I see very high throughput for _consumer_offsets topic.
> it
> > is around 30-40K per second. What could be the reason for such a high
> > rate ? Do I need to be concerned around this ?
> >
> > [image: image.png]
> >
> >
> > This happens for an hour and after that it goes down. Now it is
> happening
> > once or twice daily.
> >
> > I have 8 node cluster , 1000+ topics and 644 consumer groups.
> > All nodes have almost equal number of lead partition across nodes and
> > almost equal partition on all nodes.
> >
> > Kafka version : 2.1.1
> >
> > If you see above graph, there are no other topics where messages are more
> > than 300 Message/sec. only _consumer_offsets is having so high through
> > put.
> >
> > I know this is internal topic ans store metadata of topics and consumer
> > information. But I don't see anything abnormal in my cluster then why
> > _consumer_offset topic is going crazy. What is going on here ?
> >
> > Any help will be appreciated.
> >
> > --
> > Thanx & Regard
> > Ashutosh Singh
> > 08151945559
> >
> >
>
Re: _consumer_offsets topic produce rate is dam high
Posted by Karolis Pocius <ka...@sentiance.com.INVALID>.
Check if any of your consumers have auto commit turned off and instead
commit after processing each message.
Also, even if all consumers are using auto commit, maybe some of them have
the interval set to something crazy low like 1 ms.
On Sat, Jun 20, 2020 at 8:31 PM Ashutosh singh <ge...@gmail.com> wrote:
> Hi Guys,
>
> Hope you all are doing well.
>
> All of sudden I see very high throughput for _consumer_offsets topic. it
> is around 30-40K per second. What could be the reason for such a high
> rate ? Do I need to be concerned around this ?
>
> [image: image.png]
>
>
> This happens for an hour and after that it goes down. Now it is happening
> once or twice daily.
>
> I have 8 node cluster , 1000+ topics and 644 consumer groups.
> All nodes have almost equal number of lead partition across nodes and
> almost equal partition on all nodes.
>
> Kafka version : 2.1.1
>
> If you see above graph, there are no other topics where messages are more
> than 300 Message/sec. only _consumer_offsets is having so high through
> put.
>
> I know this is internal topic ans store metadata of topics and consumer
> information. But I don't see anything abnormal in my cluster then why
> _consumer_offset topic is going crazy. What is going on here ?
>
> Any help will be appreciated.
>
> --
> Thanx & Regard
> Ashutosh Singh
> 08151945559
>
>