Re: Fw: How to measure performance of Mirror Maker

[Otis Gospodnetic <ot...@gmail.com>]

Hi,

I don't know much about MM yet, but what sort of MM performance are you
looking for?  If MM acts as just another producer for another Kafka cluster
(just a guess!) then you could look at the metrics of the consuming cluster
to see how fast it's ingesting....  but if MM works differently, then
something else is needed.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Tue, Mar 17, 2015 at 5:46 PM, Saladi Naidu <naidusp2002@yahoo.com.invalid
> wrote:

> Any suggestions on how to measure throughput of the Mirror Maker Naidu
> Saladi
>
>     ----- Forwarded Message -----
>   From: Saladi Naidu <na...@yahoo.com>
>  To: "users@kafka.apache.org" <us...@kafka.apache.org>
>  Sent: Monday, March 16, 2015 10:31 PM
>  Subject: How to measure performance of Mirror Maker
>
> We have three Kafka clusters deployed in 3 DC's, each one having their own
> topics. We are using Mirror Maker to keep all the three clusters up to date
> with continuous  replication using Mirror Maker. We used Perf producer and
> Perf consumer to conduct basic testing, all seems to work great.
> Our 2 DC's are in US, 50 miles a part well connected by MAN fiber, our 3rd
> DC is in Europe, which is connected with dedicated links to 2 DC's in US,
> round trip latency is around 300 ms.
> When we ran performance test, our client node where perf test scripts ran
> are in US DC's, I have observed very low through put (almost 1/100th),  I
> understand some part is due to  NW latency. Same will be applied to Mirror
> Maker as well right?
> Are there any tools to measure Mirror Maker performance in general and
> specifically in remote DC?    Naidu Saladi
>
>
>
>

Re: Fw: How to measure performance of Mirror Maker

[Daniel Compton <da...@gmail.com>]

>From memory, MirrorMaker is just using Kafka Producers and Consumers to
send the data from one DC to the other. So the meaningful performance
metric I would be looking at is how far behind your mirror queues are from
your source queues. Your other performance metrics are going to be very
dependent on what's going on in the source queue.

For stress testing, I would create a constant source of Kafka messages, and
increase the rate by 50% every 10 minutes until the MM can no longer keep
up. Your network link throughput is almost certainly going to be the
bottleneck.

On Wed, 22 Apr 2015 at 8:30 am Otis Gospodnetic <ot...@gmail.com>
wrote:

> Hi,
>
> I don't know much about MM yet, but what sort of MM performance are you
> looking for?  If MM acts as just another producer for another Kafka cluster
> (just a guess!) then you could look at the metrics of the consuming cluster
> to see how fast it's ingesting....  but if MM works differently, then
> something else is needed.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Tue, Mar 17, 2015 at 5:46 PM, Saladi Naidu
> <naidusp2002@yahoo.com.invalid
> > wrote:
>
> > Any suggestions on how to measure throughput of the Mirror Maker Naidu
> > Saladi
> >
> >     ----- Forwarded Message -----
> >   From: Saladi Naidu <na...@yahoo.com>
> >  To: "users@kafka.apache.org" <us...@kafka.apache.org>
> >  Sent: Monday, March 16, 2015 10:31 PM
> >  Subject: How to measure performance of Mirror Maker
> >
> > We have three Kafka clusters deployed in 3 DC's, each one having their
> own
> > topics. We are using Mirror Maker to keep all the three clusters up to
> date
> > with continuous  replication using Mirror Maker. We used Perf producer
> and
> > Perf consumer to conduct basic testing, all seems to work great.
> > Our 2 DC's are in US, 50 miles a part well connected by MAN fiber, our
> 3rd
> > DC is in Europe, which is connected with dedicated links to 2 DC's in US,
> > round trip latency is around 300 ms.
> > When we ran performance test, our client node where perf test scripts ran
> > are in US DC's, I have observed very low through put (almost 1/100th),  I
> > understand some part is due to  NW latency. Same will be applied to
> Mirror
> > Maker as well right?
> > Are there any tools to measure Mirror Maker performance in general and
> > specifically in remote DC?    Naidu Saladi
> >
> >
> >
> >
>