Mirror Maker 2: Incoming messages on source and target kafka cluster mismatch after mirroring

[fighter <ku...@gmail.com>]

We have did the kafka cluster migration from source kafka cluster to target
kafka cluster using MirrorMaker 2.5.1 in distributed mode using kafka
connect cluster. We see noticeable difference incoming messages rate per
sec on source and target. We also analyze that on kafka connect producer
has less output rate compared to consuming rate of consumer.Details are
given below. Do you suspect the message loss from kafka connect worker ?.
We have seen the offsetsync topic the snapshots and data rate is quite
well. Any help will be appreciated.

 Kafka connect Consumer enter image description here
<https://i.stack.imgur.com/YQGga.png>. kafka connect Producer enter image
description here <https://i.stack.imgur.com/asp4t.png>

source kafka:enter image description here
<https://i.stack.imgur.com/JrKmK.png> target kafka: enter image description
here <https://i.stack.imgur.com/Umtx7.png>

We notices Mirror maker tasks and workers has no issues. We have three
connect worker nodes  only one connect node has this producer & consumer
difference. As we don't have logs any insight will help to investigate the
issue. Kafka MM Source connector producer and consumer uses default config
only.

Re: Mirror Maker 2: Incoming messages on source and target kafka cluster mismatch after mirroring

[Ning Zhang <ni...@gmail.com>]

Hi fighter,

from the description, MM2 will not lose message if the MM2, source and target clusters are all in good heath. JMX metrics also rarely go significant wrong. My personal hint is: compare the aggregated network input and output of bandwidth of source and target cluster and see if match or not.

On 2021/04/29 09:34:49, fighter <ku...@gmail.com> wrote: 
> We migrated data from 7 node source cluster to target 9 node cluster. MM
> source connector producer does not have any compression as i mentioned
> earlier uses default producer. We use ByteArrayConverter for both
> producer key and value, idempotency =true. Is there a chance the JMX
> metrics can go wrong ?.
> 
> On Thu, Apr 29, 2021 at 12:09 AM fighter <ku...@gmail.com> wrote:
> 
> > We have did the kafka cluster migration from source kafka cluster to
> > target kafka cluster using MirrorMaker 2.5.1 in distributed mode using
> > kafka connect cluster. We see noticeable difference incoming messages rate
> > per sec on source and target. We also analyze that on kafka connect
> > producer has less output rate compared to consuming rate of
> > consumer.Details are given below. Do you suspect the message loss from
> > kafka connect worker ?. We have seen the offsetsync topic the snapshots and
> > data rate is quite well. Any help will be appreciated.
> >
> >  Kafka connect Consumer enter image description here
> > <https://i.stack.imgur.com/YQGga.png>. kafka connect Producer enter image
> > description here <https://i.stack.imgur.com/asp4t.png>
> >
> > source kafka:enter image description here
> > <https://i.stack.imgur.com/JrKmK.png> target kafka: enter image
> > description here <https://i.stack.imgur.com/Umtx7.png>
> >
> > We notices Mirror maker tasks and workers has no issues. We have three
> > connect worker nodes  only one connect node has this producer & consumer
> > difference. As we don't have logs any insight will help to investigate the
> > issue. Kafka MM Source connector producer and consumer uses default config
> > only.
> >
> >
> >
> 

Re: Mirror Maker 2: Incoming messages on source and target kafka cluster mismatch after mirroring

[fighter <ku...@gmail.com>]

We migrated data from 7 node source cluster to target 9 node cluster. MM
source connector producer does not have any compression as i mentioned
earlier uses default producer. We use ByteArrayConverter for both
producer key and value, idempotency =true. Is there a chance the JMX
metrics can go wrong ?.

On Thu, Apr 29, 2021 at 12:09 AM fighter <ku...@gmail.com> wrote:

> We have did the kafka cluster migration from source kafka cluster to
> target kafka cluster using MirrorMaker 2.5.1 in distributed mode using
> kafka connect cluster. We see noticeable difference incoming messages rate
> per sec on source and target. We also analyze that on kafka connect
> producer has less output rate compared to consuming rate of
> consumer.Details are given below. Do you suspect the message loss from
> kafka connect worker ?. We have seen the offsetsync topic the snapshots and
> data rate is quite well. Any help will be appreciated.
>
>  Kafka connect Consumer enter image description here
> <https://i.stack.imgur.com/YQGga.png>. kafka connect Producer enter image
> description here <https://i.stack.imgur.com/asp4t.png>
>
> source kafka:enter image description here
> <https://i.stack.imgur.com/JrKmK.png> target kafka: enter image
> description here <https://i.stack.imgur.com/Umtx7.png>
>
> We notices Mirror maker tasks and workers has no issues. We have three
> connect worker nodes  only one connect node has this producer & consumer
> difference. As we don't have logs any insight will help to investigate the
> issue. Kafka MM Source connector producer and consumer uses default config
> only.
>
>
>