You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Tao Feng <fe...@gmail.com> on 2015/08/15 18:19:42 UTC

thread that handle client request in Kafka brokers

Hi All,

I was told that only single thread handle client producer request in Kafka
broker which may potentially be a performance problem with request queue-up
if we have many small requests. I am wondering which part of the code I
should read to understand the above logic

Thanks,
-Tao

Re: thread that handle client request in Kafka brokers

Posted by Tao Feng <fe...@gmail.com>.

Thanks Ben. I think what I heard is that in broker socketServer it only has
one thread(acceptor?) to handle new connections which may potentially
become a bottleneck if many small requests come in.
-Tao

On Mon, Aug 17, 2015 at 7:19 AM, Ben Stopford <be...@confluent.io> wrote:

> Hi Tao
>
> This is unlikely to be a problem. The producer is threadsafe (see here <
> http://kafka.apache.org/082/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html>)
> so you can happily share it between your pool of message producers. Kafka
> also provides a range of facilities for improving parallelism in your pipe.
> These include batching messages for bulk transfer, along with callbacks to
> provide acknowledgement. Kafka obviously provides a great degree of
> parallelism across brokers...
>
> Hope that helps.
>
> B
>
>
>
> > On 15 Aug 2015, at 17:19, Tao Feng <fe...@gmail.com> wrote:
> >
> > Hi All,
> >
> > I was told that only single thread handle client producer request in
> Kafka
> > broker which may potentially be a performance problem with request
> queue-up
> > if we have many small requests. I am wondering which part of the code I
> > should read to understand the above logic
> >
> > Thanks,
> > -Tao
>
>

Re: thread that handle client request in Kafka brokers

Posted by Ben Stopford <be...@confluent.io>.

Hi Tao

This is unlikely to be a problem. The producer is threadsafe (see here <http://kafka.apache.org/082/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html>) so you can happily share it between your pool of message producers. Kafka also provides a range of facilities for improving parallelism in your pipe. These include batching messages for bulk transfer, along with callbacks to provide acknowledgement. Kafka obviously provides a great degree of parallelism across brokers...

Hope that helps. 

B

> On 15 Aug 2015, at 17:19, Tao Feng <fe...@gmail.com> wrote:
> 
> Hi All,
> 
> I was told that only single thread handle client producer request in Kafka
> broker which may potentially be a performance problem with request queue-up
> if we have many small requests. I am wondering which part of the code I
> should read to understand the above logic
> 
> Thanks,
> -Tao