You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Mhaskar, Tushar" <tm...@paypal.com.INVALID> on 2016/02/02 20:49:06 UTC
Multiple producer threads in Kafka 0.9 MirrorMaker.
Hi All,
I see that the —num.producers is removed from the Kafka 0.9 MirrorMaker. Why is this?
How can we create multiple producer threads for publishing in MirrorMaker?
Thanks,
Tushar
Re: Multiple producer threads in Kafka 0.9 MirrorMaker.
Posted by Samuel Zhou <zh...@gmail.com>.
I also have the concern that only one producer thread per MM will cause
performance issue, in some case that there are thousands of streams and
just one producer thread, the producer will be a bottleneck. If there are
hundreds of streams and ten MM(so we have 10 producer thread), more JVM
will be started and it will eat up memory. So increasing # of producer this
way may not be a good approach.
KIP-3 explains why we need only one producer per MM process, I think using
only one producer per MM may not be the best approach to solve data loss
issue. Does anyone consider the performance issue or make the similar or
higher throughput(compare to 0.9) with just one MM process?
Thanks,
Samuel
On Wed, Feb 3, 2016 at 9:43 AM, Alex Loddengaard <al...@confluent.io> wrote:
> Hi Tao,
>
> You're right -- thanks for correcting me. Only one producer is created, and
> --num.streams controls the number of consumer streams. As such, to get more
> producers you'd need to run multiple instances of MirrorMaker.
>
> Thanks, Tao!
>
> Alex
>
> On Tue, Feb 2, 2016 at 7:20 PM, tao xiao <xi...@gmail.com> wrote:
>
> > Hi Alex,
> >
> > I doubt this is the case for producer. In 0.9 mirror maker there is only
> > one producer created regardless of --num.streams. --num.streams only
> > controls how many consumers are created.
> >
> >
> >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/MirrorMaker.scala#L201
> >
> >
> > On Wed, 3 Feb 2016 at 09:53 Alex Loddengaard <al...@confluent.io> wrote:
> >
> > > Hi Tushar,
> > >
> > > In 0.9, MirrorMaker is now configured with a number of streams, not a
> > > number of consumers and producers like it was in 0.8. Each stream has
> one
> > > producer and one consumer. You can configure the number of streams
> > > with --num.streams. So setting --num.streams 10 will create 10 producer
> > > threads (and 10 consumers in the same threads).
> > >
> > > Alex
> > >
> > > On Tue, Feb 2, 2016 at 11:49 AM, Mhaskar, Tushar <
> > > tmhaskar@paypal.com.invalid> wrote:
> > >
> > > > Hi All,
> > > >
> > > > I see that the —num.producers is removed from the Kafka 0.9
> > MirrorMaker.
> > > > Why is this?
> > > > How can we create multiple producer threads for publishing in
> > > MirrorMaker?
> > > >
> > > > Thanks,
> > > > Tushar
> > > >
> > >
> > >
> > >
> > > --
> > > *Alex Loddengaard | **Solutions Architect | Confluent*
> > > *Download Apache Kafka and Confluent Platform:
> www.confluent.io/download
> > > <http://www.confluent.io/download>*
> > >
> >
>
>
>
> --
> *Alex Loddengaard | **Solutions Architect | Confluent*
> *Download Apache Kafka and Confluent Platform: www.confluent.io/download
> <http://www.confluent.io/download>*
>
Re: Multiple producer threads in Kafka 0.9 MirrorMaker.
Posted by Alex Loddengaard <al...@confluent.io>.
Hi Tao,
You're right -- thanks for correcting me. Only one producer is created, and
--num.streams controls the number of consumer streams. As such, to get more
producers you'd need to run multiple instances of MirrorMaker.
Thanks, Tao!
Alex
On Tue, Feb 2, 2016 at 7:20 PM, tao xiao <xi...@gmail.com> wrote:
> Hi Alex,
>
> I doubt this is the case for producer. In 0.9 mirror maker there is only
> one producer created regardless of --num.streams. --num.streams only
> controls how many consumers are created.
>
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/MirrorMaker.scala#L201
>
>
> On Wed, 3 Feb 2016 at 09:53 Alex Loddengaard <al...@confluent.io> wrote:
>
> > Hi Tushar,
> >
> > In 0.9, MirrorMaker is now configured with a number of streams, not a
> > number of consumers and producers like it was in 0.8. Each stream has one
> > producer and one consumer. You can configure the number of streams
> > with --num.streams. So setting --num.streams 10 will create 10 producer
> > threads (and 10 consumers in the same threads).
> >
> > Alex
> >
> > On Tue, Feb 2, 2016 at 11:49 AM, Mhaskar, Tushar <
> > tmhaskar@paypal.com.invalid> wrote:
> >
> > > Hi All,
> > >
> > > I see that the —num.producers is removed from the Kafka 0.9
> MirrorMaker.
> > > Why is this?
> > > How can we create multiple producer threads for publishing in
> > MirrorMaker?
> > >
> > > Thanks,
> > > Tushar
> > >
> >
> >
> >
> > --
> > *Alex Loddengaard | **Solutions Architect | Confluent*
> > *Download Apache Kafka and Confluent Platform: www.confluent.io/download
> > <http://www.confluent.io/download>*
> >
>
--
*Alex Loddengaard | **Solutions Architect | Confluent*
*Download Apache Kafka and Confluent Platform: www.confluent.io/download
<http://www.confluent.io/download>*
Re: Multiple producer threads in Kafka 0.9 MirrorMaker.
Posted by tao xiao <xi...@gmail.com>.
Hi Alex,
I doubt this is the case for producer. In 0.9 mirror maker there is only
one producer created regardless of --num.streams. --num.streams only
controls how many consumers are created.
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/MirrorMaker.scala#L201
On Wed, 3 Feb 2016 at 09:53 Alex Loddengaard <al...@confluent.io> wrote:
> Hi Tushar,
>
> In 0.9, MirrorMaker is now configured with a number of streams, not a
> number of consumers and producers like it was in 0.8. Each stream has one
> producer and one consumer. You can configure the number of streams
> with --num.streams. So setting --num.streams 10 will create 10 producer
> threads (and 10 consumers in the same threads).
>
> Alex
>
> On Tue, Feb 2, 2016 at 11:49 AM, Mhaskar, Tushar <
> tmhaskar@paypal.com.invalid> wrote:
>
> > Hi All,
> >
> > I see that the —num.producers is removed from the Kafka 0.9 MirrorMaker.
> > Why is this?
> > How can we create multiple producer threads for publishing in
> MirrorMaker?
> >
> > Thanks,
> > Tushar
> >
>
>
>
> --
> *Alex Loddengaard | **Solutions Architect | Confluent*
> *Download Apache Kafka and Confluent Platform: www.confluent.io/download
> <http://www.confluent.io/download>*
>
Re: Multiple producer threads in Kafka 0.9 MirrorMaker.
Posted by Alex Loddengaard <al...@confluent.io>.
Hi Tushar,
In 0.9, MirrorMaker is now configured with a number of streams, not a
number of consumers and producers like it was in 0.8. Each stream has one
producer and one consumer. You can configure the number of streams
with --num.streams. So setting --num.streams 10 will create 10 producer
threads (and 10 consumers in the same threads).
Alex
On Tue, Feb 2, 2016 at 11:49 AM, Mhaskar, Tushar <
tmhaskar@paypal.com.invalid> wrote:
> Hi All,
>
> I see that the —num.producers is removed from the Kafka 0.9 MirrorMaker.
> Why is this?
> How can we create multiple producer threads for publishing in MirrorMaker?
>
> Thanks,
> Tushar
>
--
*Alex Loddengaard | **Solutions Architect | Confluent*
*Download Apache Kafka and Confluent Platform: www.confluent.io/download
<http://www.confluent.io/download>*