You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Nazario Parsacala <do...@gmail.com> on 2016/07/21 17:07:42 UTC
Maximum number of producers per topic per broker
Kafka Users,
Are there any settings that limits the number of producers per topic per broker ..? I am experimenting with a single broker with around 500 producers and this works fine .. but increasing this further to 600 , 700 producers and suddenly the kafka broker stops functioning .. CPU utilization shoots up and no data is being received by the consumers. If we bring bring the producer back to 500 , the broker starts functioning fine again.
The thread dump on the broker seems to point to the threads being blocked on the SSL side ..?
For experimentation purpose we have increased already the following :
# The number of threads handling network requests
#num.network.threads=3
num.network.threads=128
# The number of threads doing disk I/O
#num.io.threads=8
num.io.threads=256
But no dice ..
BTW I am using kafka_2.10-0.9.0.1
Re: Maximum number of producers per topic per broker
Posted by Ismael Juma <is...@gmail.com>.
Hi,
It would be great if you could file a JIRA with additional details
regarding the issue so that we could investigate if something could be done
about it.
Ismael
On 24 Jul 2016 17:32, "Dodong Juan" <do...@gmail.com> wrote:
> Just found out what's was causing this. Which is quite dangerous. The
> additional 100 or 200 server had the wrong ssl certificate on them causing
> it to creat some kind of DOS on the broker.
>
> That was the reason why I was seeing that the thread dump was more
> pointing at the Ssl side.
>
> Sent from some kind of phone
>
> > On Jul 23, 2016, at 8:41 PM, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
> >
> > There's no strict limit on the number of producers. If you're hitting
> some
> > CPU limit, perhaps you are simply overloading the broker? 6 or 700
> brokers
> > doesn't sound that bad, but if they are producing too much data then of
> > course eventually the broker will become overwhelmed. How much total data
> > is being produced?
> >
> > -Ewen
> >
> > On Thu, Jul 21, 2016 at 10:07 AM, Nazario Parsacala <
> dodongjuan@gmail.com>
> > wrote:
> >
> >> Kafka Users,
> >>
> >> Are there any settings that limits the number of producers per topic per
> >> broker ..? I am experimenting with a single broker with around 500
> >> producers and this works fine .. but increasing this further to 600 ,
> 700
> >> producers and suddenly the kafka broker stops functioning .. CPU
> >> utilization shoots up and no data is being received by the consumers.
> If
> >> we bring bring the producer back to 500 , the broker starts functioning
> >> fine again.
> >>
> >> The thread dump on the broker seems to point to the threads being
> blocked
> >> on the SSL side ..?
> >> For experimentation purpose we have increased already the following :
> >>
> >> # The number of threads handling network requests
> >> #num.network.threads=3
> >> num.network.threads=128
> >>
> >> # The number of threads doing disk I/O
> >> #num.io.threads=8
> >> num.io.threads=256
> >>
> >> But no dice ..
> >>
> >> BTW I am using kafka_2.10-0.9.0.1
> >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
>
Re: Maximum number of producers per topic per broker
Posted by Dodong Juan <do...@gmail.com>.
Just found out what's was causing this. Which is quite dangerous. The additional 100 or 200 server had the wrong ssl certificate on them causing it to creat some kind of DOS on the broker.
That was the reason why I was seeing that the thread dump was more pointing at the Ssl side.
Sent from some kind of phone
> On Jul 23, 2016, at 8:41 PM, Ewen Cheslack-Postava <ew...@confluent.io> wrote:
>
> There's no strict limit on the number of producers. If you're hitting some
> CPU limit, perhaps you are simply overloading the broker? 6 or 700 brokers
> doesn't sound that bad, but if they are producing too much data then of
> course eventually the broker will become overwhelmed. How much total data
> is being produced?
>
> -Ewen
>
> On Thu, Jul 21, 2016 at 10:07 AM, Nazario Parsacala <do...@gmail.com>
> wrote:
>
>> Kafka Users,
>>
>> Are there any settings that limits the number of producers per topic per
>> broker ..? I am experimenting with a single broker with around 500
>> producers and this works fine .. but increasing this further to 600 , 700
>> producers and suddenly the kafka broker stops functioning .. CPU
>> utilization shoots up and no data is being received by the consumers. If
>> we bring bring the producer back to 500 , the broker starts functioning
>> fine again.
>>
>> The thread dump on the broker seems to point to the threads being blocked
>> on the SSL side ..?
>> For experimentation purpose we have increased already the following :
>>
>> # The number of threads handling network requests
>> #num.network.threads=3
>> num.network.threads=128
>>
>> # The number of threads doing disk I/O
>> #num.io.threads=8
>> num.io.threads=256
>>
>> But no dice ..
>>
>> BTW I am using kafka_2.10-0.9.0.1
>
>
>
>
> --
> Thanks,
> Ewen
Re: Maximum number of producers per topic per broker
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
There's no strict limit on the number of producers. If you're hitting some
CPU limit, perhaps you are simply overloading the broker? 6 or 700 brokers
doesn't sound that bad, but if they are producing too much data then of
course eventually the broker will become overwhelmed. How much total data
is being produced?
-Ewen
On Thu, Jul 21, 2016 at 10:07 AM, Nazario Parsacala <do...@gmail.com>
wrote:
> Kafka Users,
>
> Are there any settings that limits the number of producers per topic per
> broker ..? I am experimenting with a single broker with around 500
> producers and this works fine .. but increasing this further to 600 , 700
> producers and suddenly the kafka broker stops functioning .. CPU
> utilization shoots up and no data is being received by the consumers. If
> we bring bring the producer back to 500 , the broker starts functioning
> fine again.
>
> The thread dump on the broker seems to point to the threads being blocked
> on the SSL side ..?
> For experimentation purpose we have increased already the following :
>
> # The number of threads handling network requests
> #num.network.threads=3
> num.network.threads=128
>
> # The number of threads doing disk I/O
> #num.io.threads=8
> num.io.threads=256
>
> But no dice ..
>
> BTW I am using kafka_2.10-0.9.0.1
--
Thanks,
Ewen