You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by prateek arora <pr...@gmail.com> on 2016/06/01 20:49:23 UTC
Kafka broker slow down when consumer try to fetch large messages from topic
I have 4 node kafka broker with following configuration :
Default Number of Partitions : num.partitions : 1
Default Replication Factor : default.replication.factor : 1
Maximum Message Size : message.max.bytes : 10 MB
Replica Maximum Fetch Size : replica.fetch.max.bytes : 10 MB
Right now I have 4 topic with 1 partition and 1 replication factor .
"Topic Name" : "Broker Id" : "Total Messages Received Across Kafka Broker"
: "Total Bytes Received Across Kafka Broker"
Topic 1 - Leader Kafka Broker 1 : 4.67 Message/Second : 1.6 MB/second
Topic 2 - Leader Kafka Broker 2 : 4.78 Message/Second : 4.1 MB/second
Topic 3 - Leader Kafka Broker 1 : 4.83 Message/Second : 1.6 MB/second
Topic 4 - Leader Kafka Broker 3 : 4.8 Message/Second : 4.3 MB/second
Message consist of <String , Array[Byte] >.
when consumer tried to read message from "Topic 2" Kafka Broker rate of
message receiving slow down from 4.77 message/second to 3.12
message/second , after some time try to goes up .
I also attached screenshot of "Total Messages Received Across Kafka Broker"
and "Total Bytes Received Across Kafka Broker" for topic "Topic 2" .
can someone explain why it is happen and how to solve it ?
Regards
Prateek
Re: Kafka broker slow down when consumer try to fetch large messages
from topic
Posted by R Krishna <kr...@gmail.com>.
Prateek, hope you looked at compression?
On Thu, Jun 2, 2016 at 10:26 AM, Tom Crayford <tc...@heroku.com> wrote:
> The article says ideal is about 10KB, which holds up well with what we've
> seen in practice as well.
>
> On Thu, Jun 2, 2016 at 6:25 PM, prateek arora <pr...@gmail.com>
> wrote:
>
> > Hi
> > Thanks for the information .
> >
> > I have one question :
> >
> > Right now in my scenario maximum message size is around 800KB . did we
> > consider these messages in large size categories , because article told
> > about 10-100 MB data .
> >
> > Regards
> > Prateek
> >
> >
> >
> >
> >
> >
> > On Thu, Jun 2, 2016 at 6:54 AM, Tom Crayford <tc...@heroku.com>
> wrote:
> >
> > > Hi there,
> > >
> > > Firstly, a note that Kafka isn't really designed for this kind of large
> > > message. http://ingest.tips/2015/01/21/handling-large-messages-kafka/
> > > covers a lot of tips around this use case however, and covers some
> tuning
> > > that will likely improve your usage.
> > >
> > > In particular, I expect tuning up fetch.message.max.bytes on the
> consumer
> > > to help out a lot here.
> > >
> > > Generally though, doing large messages will lead to very low throughput
> > and
> > > lots of stability issues, as noted in that article. We run thousands of
> > > clusters in production, and typically recommend folk keep message sizes
> > > down to the few tens of KB for most use cases.
> > >
> > > Thanks
> > >
> > > Tom Crayford
> > > Heroku Kafka
> > >
> > > On Wed, Jun 1, 2016 at 9:49 PM, prateek arora <
> > prateek.arora2k6@gmail.com>
> > > wrote:
> > >
> > > > I have 4 node kafka broker with following configuration :
> > > >
> > > > Default Number of Partitions : num.partitions : 1
> > > > Default Replication Factor : default.replication.factor : 1
> > > > Maximum Message Size : message.max.bytes : 10 MB
> > > > Replica Maximum Fetch Size : replica.fetch.max.bytes : 10 MB
> > > >
> > > >
> > > > Right now I have 4 topic with 1 partition and 1 replication factor .
> > > >
> > > > "Topic Name" : "Broker Id" : "Total Messages Received Across Kafka
> > > > Broker" : "Total Bytes Received Across Kafka Broker"
> > > > Topic 1 - Leader Kafka Broker 1 : 4.67 Message/Second : 1.6
> > MB/second
> > > > Topic 2 - Leader Kafka Broker 2 : 4.78 Message/Second : 4.1
> > MB/second
> > > > Topic 3 - Leader Kafka Broker 1 : 4.83 Message/Second : 1.6
> > > MB/second
> > > > Topic 4 - Leader Kafka Broker 3 : 4.8 Message/Second : 4.3
> > > MB/second
> > > >
> > > > Message consist of <String , Array[Byte] >.
> > > >
> > > >
> > > > when consumer tried to read message from "Topic 2" Kafka Broker rate
> > of
> > > > message receiving slow down from 4.77 message/second to 3.12
> > > > message/second , after some time try to goes up .
> > > >
> > > > I also attached screenshot of "Total Messages Received Across Kafka
> > > > Broker" and "Total Bytes Received Across Kafka Broker" for topic
> > "Topic
> > > > 2" .
> > > >
> > > > can someone explain why it is happen and how to solve it ?
> > > >
> > > > Regards
> > > > Prateek
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
--
Radha Krishna, Proddaturi
253-234-5657
Re: Kafka broker slow down when consumer try to fetch large messages
from topic
Posted by Tom Crayford <tc...@heroku.com>.
The article says ideal is about 10KB, which holds up well with what we've
seen in practice as well.
On Thu, Jun 2, 2016 at 6:25 PM, prateek arora <pr...@gmail.com>
wrote:
> Hi
> Thanks for the information .
>
> I have one question :
>
> Right now in my scenario maximum message size is around 800KB . did we
> consider these messages in large size categories , because article told
> about 10-100 MB data .
>
> Regards
> Prateek
>
>
>
>
>
>
> On Thu, Jun 2, 2016 at 6:54 AM, Tom Crayford <tc...@heroku.com> wrote:
>
> > Hi there,
> >
> > Firstly, a note that Kafka isn't really designed for this kind of large
> > message. http://ingest.tips/2015/01/21/handling-large-messages-kafka/
> > covers a lot of tips around this use case however, and covers some tuning
> > that will likely improve your usage.
> >
> > In particular, I expect tuning up fetch.message.max.bytes on the consumer
> > to help out a lot here.
> >
> > Generally though, doing large messages will lead to very low throughput
> and
> > lots of stability issues, as noted in that article. We run thousands of
> > clusters in production, and typically recommend folk keep message sizes
> > down to the few tens of KB for most use cases.
> >
> > Thanks
> >
> > Tom Crayford
> > Heroku Kafka
> >
> > On Wed, Jun 1, 2016 at 9:49 PM, prateek arora <
> prateek.arora2k6@gmail.com>
> > wrote:
> >
> > > I have 4 node kafka broker with following configuration :
> > >
> > > Default Number of Partitions : num.partitions : 1
> > > Default Replication Factor : default.replication.factor : 1
> > > Maximum Message Size : message.max.bytes : 10 MB
> > > Replica Maximum Fetch Size : replica.fetch.max.bytes : 10 MB
> > >
> > >
> > > Right now I have 4 topic with 1 partition and 1 replication factor .
> > >
> > > "Topic Name" : "Broker Id" : "Total Messages Received Across Kafka
> > > Broker" : "Total Bytes Received Across Kafka Broker"
> > > Topic 1 - Leader Kafka Broker 1 : 4.67 Message/Second : 1.6
> MB/second
> > > Topic 2 - Leader Kafka Broker 2 : 4.78 Message/Second : 4.1
> MB/second
> > > Topic 3 - Leader Kafka Broker 1 : 4.83 Message/Second : 1.6
> > MB/second
> > > Topic 4 - Leader Kafka Broker 3 : 4.8 Message/Second : 4.3
> > MB/second
> > >
> > > Message consist of <String , Array[Byte] >.
> > >
> > >
> > > when consumer tried to read message from "Topic 2" Kafka Broker rate
> of
> > > message receiving slow down from 4.77 message/second to 3.12
> > > message/second , after some time try to goes up .
> > >
> > > I also attached screenshot of "Total Messages Received Across Kafka
> > > Broker" and "Total Bytes Received Across Kafka Broker" for topic
> "Topic
> > > 2" .
> > >
> > > can someone explain why it is happen and how to solve it ?
> > >
> > > Regards
> > > Prateek
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
>
Re: Kafka broker slow down when consumer try to fetch large messages
from topic
Posted by prateek arora <pr...@gmail.com>.
Hi
Thanks for the information .
I have one question :
Right now in my scenario maximum message size is around 800KB . did we
consider these messages in large size categories , because article told
about 10-100 MB data .
Regards
Prateek
On Thu, Jun 2, 2016 at 6:54 AM, Tom Crayford <tc...@heroku.com> wrote:
> Hi there,
>
> Firstly, a note that Kafka isn't really designed for this kind of large
> message. http://ingest.tips/2015/01/21/handling-large-messages-kafka/
> covers a lot of tips around this use case however, and covers some tuning
> that will likely improve your usage.
>
> In particular, I expect tuning up fetch.message.max.bytes on the consumer
> to help out a lot here.
>
> Generally though, doing large messages will lead to very low throughput and
> lots of stability issues, as noted in that article. We run thousands of
> clusters in production, and typically recommend folk keep message sizes
> down to the few tens of KB for most use cases.
>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Wed, Jun 1, 2016 at 9:49 PM, prateek arora <pr...@gmail.com>
> wrote:
>
> > I have 4 node kafka broker with following configuration :
> >
> > Default Number of Partitions : num.partitions : 1
> > Default Replication Factor : default.replication.factor : 1
> > Maximum Message Size : message.max.bytes : 10 MB
> > Replica Maximum Fetch Size : replica.fetch.max.bytes : 10 MB
> >
> >
> > Right now I have 4 topic with 1 partition and 1 replication factor .
> >
> > "Topic Name" : "Broker Id" : "Total Messages Received Across Kafka
> > Broker" : "Total Bytes Received Across Kafka Broker"
> > Topic 1 - Leader Kafka Broker 1 : 4.67 Message/Second : 1.6 MB/second
> > Topic 2 - Leader Kafka Broker 2 : 4.78 Message/Second : 4.1 MB/second
> > Topic 3 - Leader Kafka Broker 1 : 4.83 Message/Second : 1.6
> MB/second
> > Topic 4 - Leader Kafka Broker 3 : 4.8 Message/Second : 4.3
> MB/second
> >
> > Message consist of <String , Array[Byte] >.
> >
> >
> > when consumer tried to read message from "Topic 2" Kafka Broker rate of
> > message receiving slow down from 4.77 message/second to 3.12
> > message/second , after some time try to goes up .
> >
> > I also attached screenshot of "Total Messages Received Across Kafka
> > Broker" and "Total Bytes Received Across Kafka Broker" for topic "Topic
> > 2" .
> >
> > can someone explain why it is happen and how to solve it ?
> >
> > Regards
> > Prateek
> >
> >
> >
> >
> >
> >
> >
>
Re: Kafka broker slow down when consumer try to fetch large messages
from topic
Posted by Tom Crayford <tc...@heroku.com>.
Hi there,
Firstly, a note that Kafka isn't really designed for this kind of large
message. http://ingest.tips/2015/01/21/handling-large-messages-kafka/
covers a lot of tips around this use case however, and covers some tuning
that will likely improve your usage.
In particular, I expect tuning up fetch.message.max.bytes on the consumer
to help out a lot here.
Generally though, doing large messages will lead to very low throughput and
lots of stability issues, as noted in that article. We run thousands of
clusters in production, and typically recommend folk keep message sizes
down to the few tens of KB for most use cases.
Thanks
Tom Crayford
Heroku Kafka
On Wed, Jun 1, 2016 at 9:49 PM, prateek arora <pr...@gmail.com>
wrote:
> I have 4 node kafka broker with following configuration :
>
> Default Number of Partitions : num.partitions : 1
> Default Replication Factor : default.replication.factor : 1
> Maximum Message Size : message.max.bytes : 10 MB
> Replica Maximum Fetch Size : replica.fetch.max.bytes : 10 MB
>
>
> Right now I have 4 topic with 1 partition and 1 replication factor .
>
> "Topic Name" : "Broker Id" : "Total Messages Received Across Kafka
> Broker" : "Total Bytes Received Across Kafka Broker"
> Topic 1 - Leader Kafka Broker 1 : 4.67 Message/Second : 1.6 MB/second
> Topic 2 - Leader Kafka Broker 2 : 4.78 Message/Second : 4.1 MB/second
> Topic 3 - Leader Kafka Broker 1 : 4.83 Message/Second : 1.6 MB/second
> Topic 4 - Leader Kafka Broker 3 : 4.8 Message/Second : 4.3 MB/second
>
> Message consist of <String , Array[Byte] >.
>
>
> when consumer tried to read message from "Topic 2" Kafka Broker rate of
> message receiving slow down from 4.77 message/second to 3.12
> message/second , after some time try to goes up .
>
> I also attached screenshot of "Total Messages Received Across Kafka
> Broker" and "Total Bytes Received Across Kafka Broker" for topic "Topic
> 2" .
>
> can someone explain why it is happen and how to solve it ?
>
> Regards
> Prateek
>
>
>
>
>
>
>