You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Shushant Arora <sh...@gmail.com> on 2015/06/29 18:38:49 UTC
no of threads per async producer
Hi
Does kafka async producer creates thread(ProducerSendThread) in producer
memory based on no number of partitions or brokers in kafka cluster to
which it will write.
If my cluster had 1000 partitions does each producer will have 1000 threads
running always ?
Re: no of threads per async producer
Posted by Guozhang Wang <wa...@gmail.com>.
That is right. And to determine whether you need multiple producers or not
you can check io-wait-ratio on the producer, if it is close to 0 that means
the ioThread is busy sending and hence you need multiple instances.
Guozhang
On Tue, Jun 30, 2015 at 9:43 AM, Shushant Arora <sh...@gmail.com>
wrote:
> In new java Producer is there only one ioThread irrespective of no of
> brokers which send message from inmemory buffer to brokers sequentially?
>
> So if user need parallelism we should have multiple producer instances ?
>
> On Tue, Jun 30, 2015 at 10:02 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
>
> > ProducerSendThread and SyncProducer all come from the old Scala
> producer, I
> > thought you meant that producer not the new Java producer?
> >
> > Guozhang
> >
> > On Tue, Jun 30, 2015 at 9:09 AM, Shushant Arora <
> shushantarora09@gmail.com
> > >
> > wrote:
> >
> > > According to code of org.apache.kafka.clients.producer.KafkaProducer.
> > >
> > > It has 1 ioThread
> > >
> > > private final Thread
> > > <
> > >
> >
> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/Thread.java#Thread
> > > >
> > > ioThread;
> > >
> > > String <
> > >
> >
> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/String.java#String
> > > >
> > > ioThreadName = "kafka-producer-network-thread" + (clientId.length
> > > <
> > >
> >
> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/String.java#String.length%28%29
> > > >()
> > > > 0 ? " | " + clientId : "");
> > >
> > > 210 <
> > >
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#210
> > > >
> > >
> > > <
> > >
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#
> > > >
> > >
> > > this.ioThread
> > > <
> > >
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0ioThread
> > > >
> > > = new KafkaThread
> > > <
> > >
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/common/utils/KafkaThread.java#KafkaThread
> > > >(ioThreadName,
> > > this.sender <
> > >
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0sender
> > > >,
> > > true);
> > >
> > > 211 <
> > >
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#211
> > > >
> > >
> > > <
> > >
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#
> > > >
> > >
> > > this.ioThread
> > > <
> > >
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0ioThread
> > > >.start
> > > <
> > >
> >
> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/Thread.java#Thread.start%28%29
> > > >();
> > >
> > > 212 <
> > >
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#212
> > > >
> > >
> > >
> > > is ioThread is the one responsible for sending message from inmemory
> > buffer
> > > to kafka broker . So where is SyncProducer got created and is
> > SyncProducer
> > > responsible for sending message from inmemory buffer to kafka broker
> and
> > > where is that got created?
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Jun 30, 2015 at 9:16 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > > > One ProducerSendThread is always the case for producer, but inside
> the
> > > > thread there is one SyncProducer for each destination broker. I think
> > > that
> > > > blog may mis-understand the design a bit.
> > > >
> > > > Guozhang
> > > >
> > > > On Tue, Jun 30, 2015 at 1:45 AM, Shushant Arora <
> > > shushantarora09@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > According to
> > > > > https://engineering.gnip.com/kafka-async-producer/
> > > > > There is one ProducerSendThread per broker. Is it changed from
> > 0.8.2 ?
> > > > to
> > > > > use single send thread irrespective of broker or partitions.
> > > > >
> > > > >
> > > > > On Mon, Jun 29, 2015 at 11:27 PM, Guozhang Wang <
> wangguoz@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Shushant,
> > > > > >
> > > > > > The number of ProducerSendThread will be always one, regardless
> of
> > > the
> > > > > > number of destination brokers, or the number of partitions.
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Mon, Jun 29, 2015 at 9:38 AM, Shushant Arora <
> > > > > shushantarora09@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > Does kafka async producer creates thread(ProducerSendThread) in
> > > > > producer
> > > > > > > memory based on no number of partitions or brokers in kafka
> > cluster
> > > > to
> > > > > > > which it will write.
> > > > > > >
> > > > > > > If my cluster had 1000 partitions does each producer will have
> > 1000
> > > > > > threads
> > > > > > > running always ?
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: no of threads per async producer
Posted by Shushant Arora <sh...@gmail.com>.
In new java Producer is there only one ioThread irrespective of no of
brokers which send message from inmemory buffer to brokers sequentially?
So if user need parallelism we should have multiple producer instances ?
On Tue, Jun 30, 2015 at 10:02 PM, Guozhang Wang <wa...@gmail.com> wrote:
> ProducerSendThread and SyncProducer all come from the old Scala producer, I
> thought you meant that producer not the new Java producer?
>
> Guozhang
>
> On Tue, Jun 30, 2015 at 9:09 AM, Shushant Arora <shushantarora09@gmail.com
> >
> wrote:
>
> > According to code of org.apache.kafka.clients.producer.KafkaProducer.
> >
> > It has 1 ioThread
> >
> > private final Thread
> > <
> >
> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/Thread.java#Thread
> > >
> > ioThread;
> >
> > String <
> >
> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/String.java#String
> > >
> > ioThreadName = "kafka-producer-network-thread" + (clientId.length
> > <
> >
> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/String.java#String.length%28%29
> > >()
> > > 0 ? " | " + clientId : "");
> >
> > 210 <
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#210
> > >
> >
> > <
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#
> > >
> >
> > this.ioThread
> > <
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0ioThread
> > >
> > = new KafkaThread
> > <
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/common/utils/KafkaThread.java#KafkaThread
> > >(ioThreadName,
> > this.sender <
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0sender
> > >,
> > true);
> >
> > 211 <
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#211
> > >
> >
> > <
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#
> > >
> >
> > this.ioThread
> > <
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0ioThread
> > >.start
> > <
> >
> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/Thread.java#Thread.start%28%29
> > >();
> >
> > 212 <
> >
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#212
> > >
> >
> >
> > is ioThread is the one responsible for sending message from inmemory
> buffer
> > to kafka broker . So where is SyncProducer got created and is
> SyncProducer
> > responsible for sending message from inmemory buffer to kafka broker and
> > where is that got created?
> >
> >
> >
> >
> >
> >
> > On Tue, Jun 30, 2015 at 9:16 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > One ProducerSendThread is always the case for producer, but inside the
> > > thread there is one SyncProducer for each destination broker. I think
> > that
> > > blog may mis-understand the design a bit.
> > >
> > > Guozhang
> > >
> > > On Tue, Jun 30, 2015 at 1:45 AM, Shushant Arora <
> > shushantarora09@gmail.com
> > > >
> > > wrote:
> > >
> > > > According to
> > > > https://engineering.gnip.com/kafka-async-producer/
> > > > There is one ProducerSendThread per broker. Is it changed from
> 0.8.2 ?
> > > to
> > > > use single send thread irrespective of broker or partitions.
> > > >
> > > >
> > > > On Mon, Jun 29, 2015 at 11:27 PM, Guozhang Wang <wa...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Shushant,
> > > > >
> > > > > The number of ProducerSendThread will be always one, regardless of
> > the
> > > > > number of destination brokers, or the number of partitions.
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Mon, Jun 29, 2015 at 9:38 AM, Shushant Arora <
> > > > shushantarora09@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > Does kafka async producer creates thread(ProducerSendThread) in
> > > > producer
> > > > > > memory based on no number of partitions or brokers in kafka
> cluster
> > > to
> > > > > > which it will write.
> > > > > >
> > > > > > If my cluster had 1000 partitions does each producer will have
> 1000
> > > > > threads
> > > > > > running always ?
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: no of threads per async producer
Posted by Guozhang Wang <wa...@gmail.com>.
ProducerSendThread and SyncProducer all come from the old Scala producer, I
thought you meant that producer not the new Java producer?
Guozhang
On Tue, Jun 30, 2015 at 9:09 AM, Shushant Arora <sh...@gmail.com>
wrote:
> According to code of org.apache.kafka.clients.producer.KafkaProducer.
>
> It has 1 ioThread
>
> private final Thread
> <
> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/Thread.java#Thread
> >
> ioThread;
>
> String <
> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/String.java#String
> >
> ioThreadName = "kafka-producer-network-thread" + (clientId.length
> <
> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/String.java#String.length%28%29
> >()
> > 0 ? " | " + clientId : "");
>
> 210 <
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#210
> >
>
> <
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#
> >
>
> this.ioThread
> <
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0ioThread
> >
> = new KafkaThread
> <
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/common/utils/KafkaThread.java#KafkaThread
> >(ioThreadName,
> this.sender <
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0sender
> >,
> true);
>
> 211 <
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#211
> >
>
> <
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#
> >
>
> this.ioThread
> <
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0ioThread
> >.start
> <
> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/Thread.java#Thread.start%28%29
> >();
>
> 212 <
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#212
> >
>
>
> is ioThread is the one responsible for sending message from inmemory buffer
> to kafka broker . So where is SyncProducer got created and is SyncProducer
> responsible for sending message from inmemory buffer to kafka broker and
> where is that got created?
>
>
>
>
>
>
> On Tue, Jun 30, 2015 at 9:16 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > One ProducerSendThread is always the case for producer, but inside the
> > thread there is one SyncProducer for each destination broker. I think
> that
> > blog may mis-understand the design a bit.
> >
> > Guozhang
> >
> > On Tue, Jun 30, 2015 at 1:45 AM, Shushant Arora <
> shushantarora09@gmail.com
> > >
> > wrote:
> >
> > > According to
> > > https://engineering.gnip.com/kafka-async-producer/
> > > There is one ProducerSendThread per broker. Is it changed from 0.8.2 ?
> > to
> > > use single send thread irrespective of broker or partitions.
> > >
> > >
> > > On Mon, Jun 29, 2015 at 11:27 PM, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > >
> > > > Hi Shushant,
> > > >
> > > > The number of ProducerSendThread will be always one, regardless of
> the
> > > > number of destination brokers, or the number of partitions.
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Jun 29, 2015 at 9:38 AM, Shushant Arora <
> > > shushantarora09@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > Does kafka async producer creates thread(ProducerSendThread) in
> > > producer
> > > > > memory based on no number of partitions or brokers in kafka cluster
> > to
> > > > > which it will write.
> > > > >
> > > > > If my cluster had 1000 partitions does each producer will have 1000
> > > > threads
> > > > > running always ?
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: no of threads per async producer
Posted by Shushant Arora <sh...@gmail.com>.
According to code of org.apache.kafka.clients.producer.KafkaProducer.
It has 1 ioThread
private final Thread
<http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/Thread.java#Thread>
ioThread;
String <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/String.java#String>
ioThreadName = "kafka-producer-network-thread" + (clientId.length
<http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/String.java#String.length%28%29>()
> 0 ? " | " + clientId : "");
210 <http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#210>
<http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#>
this.ioThread
<http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0ioThread>
= new KafkaThread
<http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/common/utils/KafkaThread.java#KafkaThread>(ioThreadName,
this.sender <http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0sender>,
true);
211 <http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#211>
<http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#>
this.ioThread
<http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0ioThread>.start
<http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/Thread.java#Thread.start%28%29>();
212 <http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#212>
is ioThread is the one responsible for sending message from inmemory buffer
to kafka broker . So where is SyncProducer got created and is SyncProducer
responsible for sending message from inmemory buffer to kafka broker and
where is that got created?
On Tue, Jun 30, 2015 at 9:16 PM, Guozhang Wang <wa...@gmail.com> wrote:
> One ProducerSendThread is always the case for producer, but inside the
> thread there is one SyncProducer for each destination broker. I think that
> blog may mis-understand the design a bit.
>
> Guozhang
>
> On Tue, Jun 30, 2015 at 1:45 AM, Shushant Arora <shushantarora09@gmail.com
> >
> wrote:
>
> > According to
> > https://engineering.gnip.com/kafka-async-producer/
> > There is one ProducerSendThread per broker. Is it changed from 0.8.2 ?
> to
> > use single send thread irrespective of broker or partitions.
> >
> >
> > On Mon, Jun 29, 2015 at 11:27 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> >
> > > Hi Shushant,
> > >
> > > The number of ProducerSendThread will be always one, regardless of the
> > > number of destination brokers, or the number of partitions.
> > >
> > > Guozhang
> > >
> > > On Mon, Jun 29, 2015 at 9:38 AM, Shushant Arora <
> > shushantarora09@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi
> > > >
> > > > Does kafka async producer creates thread(ProducerSendThread) in
> > producer
> > > > memory based on no number of partitions or brokers in kafka cluster
> to
> > > > which it will write.
> > > >
> > > > If my cluster had 1000 partitions does each producer will have 1000
> > > threads
> > > > running always ?
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: no of threads per async producer
Posted by Guozhang Wang <wa...@gmail.com>.
One ProducerSendThread is always the case for producer, but inside the
thread there is one SyncProducer for each destination broker. I think that
blog may mis-understand the design a bit.
Guozhang
On Tue, Jun 30, 2015 at 1:45 AM, Shushant Arora <sh...@gmail.com>
wrote:
> According to
> https://engineering.gnip.com/kafka-async-producer/
> There is one ProducerSendThread per broker. Is it changed from 0.8.2 ? to
> use single send thread irrespective of broker or partitions.
>
>
> On Mon, Jun 29, 2015 at 11:27 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
>
> > Hi Shushant,
> >
> > The number of ProducerSendThread will be always one, regardless of the
> > number of destination brokers, or the number of partitions.
> >
> > Guozhang
> >
> > On Mon, Jun 29, 2015 at 9:38 AM, Shushant Arora <
> shushantarora09@gmail.com
> > >
> > wrote:
> >
> > > Hi
> > >
> > > Does kafka async producer creates thread(ProducerSendThread) in
> producer
> > > memory based on no number of partitions or brokers in kafka cluster to
> > > which it will write.
> > >
> > > If my cluster had 1000 partitions does each producer will have 1000
> > threads
> > > running always ?
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: no of threads per async producer
Posted by Shushant Arora <sh...@gmail.com>.
According to
https://engineering.gnip.com/kafka-async-producer/
There is one ProducerSendThread per broker. Is it changed from 0.8.2 ? to
use single send thread irrespective of broker or partitions.
On Mon, Jun 29, 2015 at 11:27 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Shushant,
>
> The number of ProducerSendThread will be always one, regardless of the
> number of destination brokers, or the number of partitions.
>
> Guozhang
>
> On Mon, Jun 29, 2015 at 9:38 AM, Shushant Arora <shushantarora09@gmail.com
> >
> wrote:
>
> > Hi
> >
> > Does kafka async producer creates thread(ProducerSendThread) in producer
> > memory based on no number of partitions or brokers in kafka cluster to
> > which it will write.
> >
> > If my cluster had 1000 partitions does each producer will have 1000
> threads
> > running always ?
> >
>
>
>
> --
> -- Guozhang
>
Re: no of threads per async producer
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Shushant,
The number of ProducerSendThread will be always one, regardless of the
number of destination brokers, or the number of partitions.
Guozhang
On Mon, Jun 29, 2015 at 9:38 AM, Shushant Arora <sh...@gmail.com>
wrote:
> Hi
>
> Does kafka async producer creates thread(ProducerSendThread) in producer
> memory based on no number of partitions or brokers in kafka cluster to
> which it will write.
>
> If my cluster had 1000 partitions does each producer will have 1000 threads
> running always ?
>
--
-- Guozhang