MirrorMaker 2 throttling

[Péter Sinóros-Szabó <pe...@transferwise.com.INVALID>]

Hi,

I'd like to throttle the mirroring process when I start Mirror Maker 2 at
the first time, so it starts to pull all the messages that exists on the
source cluster. I'd like to do this only to avoid putting too much traffic
on the source cluster that may slow down existing production client on it.

I tried several quota setups on both the source and destination clusters,
both none of them worked.
- it either did not have any affect
- or slowed down the mirroring but also cause issues like
ERROR WorkerSourceTask{id=MirrorHeartbeatConnector-0} Failed to flush,
timed out while waiting for producer to flush outstanding 115 messages

Is there a good practice on how to initialize/bootstrap a MirrorMaker
cluster on an existing Kafka cluster?

Cheers,
Peter

Re: MirrorMaker 2 throttling

[Péter Sinóros-Szabó <pe...@transferwise.com.INVALID>]

Hey,

yes, at first it specified client.id in the config (I was not sure if which
one is needed):

source->destination.producer.client.id = "mm2"

source->destination.consumer.client.id = "mm2"

source.producer.client.id = "mm2"

source.consumer.client.id = "mm2"

destination.producer.client.id = "mm2"

destination.consumer.client.id = "mm2"

and set throttling this way:

/opt/app/kafka/kafka/bin/kafka-configs.sh --zookeeper test-zk:2181 --alter
--add-config consumer_byte_rate=200000 --entity-type clients --entity-name
mm2

but it did not seem to work and I saw in the logs that the client id is
generated based on the above config and it is actually in the form
of: consumer-source-mm2-1, consumer-source-mm2-2, ...   so I tried that as
well but without luck, maybe I just did not set the throttle low enough?
So I tried setting the throttle for all clients using the default setting
on the destination cluster:

/opt/app/kafka/kafka/bin/kafka-configs.sh --zookeeper test-zk-backup:2181
--alter --add-config producer_byte_rate=200000 --entity-type clients
--entity-default

that's when I observed that if I set the throttle value to a low enough
value that really slows down the mirroring traffic (i checked network
utilization), then I started to see "Failed to flush" error messages.

So if anyone has a good practice on how it is done, please share.

Peter



On Wed, 8 Jan 2020 at 17:34, Ryanne Dolan <ry...@gmail.com> wrote:

> Peter, have you tried overriding the client ID used by MM2's consumers?
> Otherwise, the client IDs are dynamic, which would make it difficult to
> throttle using quotas.
>
> Ryanne
>
> On Wed, Jan 8, 2020, 10:12 AM Péter Sinóros-Szabó
> <pe...@transferwise.com.invalid> wrote:
>
> > Hi,
> >
> > I'd like to throttle the mirroring process when I start Mirror Maker 2 at
> > the first time, so it starts to pull all the messages that exists on the
> > source cluster. I'd like to do this only to avoid putting too much
> traffic
> > on the source cluster that may slow down existing production client on
> it.
> >
> > I tried several quota setups on both the source and destination clusters,
> > both none of them worked.
> > - it either did not have any affect
> > - or slowed down the mirroring but also cause issues like
> > ERROR WorkerSourceTask{id=MirrorHeartbeatConnector-0} Failed to flush,
> > timed out while waiting for producer to flush outstanding 115 messages
> >
> > Is there a good practice on how to initialize/bootstrap a MirrorMaker
> > cluster on an existing Kafka cluster?
> >
> > Cheers,
> > Peter
> >
>


-- 
 - Sini

Re: MirrorMaker 2 throttling

[Ryanne Dolan <ry...@gmail.com>]

Peter, have you tried overriding the client ID used by MM2's consumers?
Otherwise, the client IDs are dynamic, which would make it difficult to
throttle using quotas.

Ryanne

On Wed, Jan 8, 2020, 10:12 AM Péter Sinóros-Szabó
<pe...@transferwise.com.invalid> wrote:

> Hi,
>
> I'd like to throttle the mirroring process when I start Mirror Maker 2 at
> the first time, so it starts to pull all the messages that exists on the
> source cluster. I'd like to do this only to avoid putting too much traffic
> on the source cluster that may slow down existing production client on it.
>
> I tried several quota setups on both the source and destination clusters,
> both none of them worked.
> - it either did not have any affect
> - or slowed down the mirroring but also cause issues like
> ERROR WorkerSourceTask{id=MirrorHeartbeatConnector-0} Failed to flush,
> timed out while waiting for producer to flush outstanding 115 messages
>
> Is there a good practice on how to initialize/bootstrap a MirrorMaker
> cluster on an existing Kafka cluster?
>
> Cheers,
> Peter
>