You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Nils Hermansson <ni...@getdreams.com> on 2020/01/23 08:46:29 UTC
MirrorMaker 2 does not replicate all messages?
Hello,
I have setup MM2 and the config is similar to the example config.
https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties
Primary has 3 brokers and secondary has 1 broker.
I wrote a script to diff number of messages in topics between primary and
secondary cluster.
To get number of messages in topic I use.
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
localhost:9092 --topic mytopic --time -1 --offsets 1 | awk -F ":" '{sum +=
$3} END {print sum}'
Is there some reason not all messages gets replicated?
A first thought was that it may have been because I wiped the secondary
cluster and started from beginning. So i switched the name of the secondary
cluster and prefix in mm2 config. But I ended up with the same diff.
Result:
topic: topic1, primary: 236, secondary: 119, diff 117
topic: topic2, primary: 2910, secondary: 1223, diff 1687
topic: topic3, primary: 2248, secondary: 2248, diff 0
topic: topic4, primary: 627, secondary: 91, diff 536
topic: topic5, primary: 84, secondary: 77, diff 7
topic: topic6, primary: 333, secondary: 156, diff 177
topic: topic7, primary: 158, secondary: 45, diff 113
topic: topic8, primary: 247, secondary: 48, diff 199
...
...
BR
Nils
Re: MirrorMaker 2 does not replicate all messages?
Posted by Nils Hermansson <ni...@getdreams.com>.
Hello,
it has been figured out.
The method was not the best way to measure number of messages in a topic.
Can't measure topics which have compactaion with kafka.tools.GetOffsetShell
script.
BR
On Thu, Jan 23, 2020 at 3:03 PM Ryanne Dolan <ry...@gmail.com> wrote:
> Nils, are those record counts or offsets?
>
> Ryanne
>
> On Thu, Jan 23, 2020, 2:46 AM Nils Hermansson <
> nils.hermansson@getdreams.com>
> wrote:
>
> > Hello,
> > I have setup MM2 and the config is similar to the example config.
> >
> >
> https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties
> > Primary has 3 brokers and secondary has 1 broker.
> >
> > I wrote a script to diff number of messages in topics between primary and
> > secondary cluster.
> >
> > To get number of messages in topic I use.
> > bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
> > localhost:9092 --topic mytopic --time -1 --offsets 1 | awk -F ":" '{sum
> +=
> > $3} END {print sum}'
> >
> > Is there some reason not all messages gets replicated?
> >
> > A first thought was that it may have been because I wiped the secondary
> > cluster and started from beginning. So i switched the name of the
> secondary
> > cluster and prefix in mm2 config. But I ended up with the same diff.
> >
> > Result:
> > topic: topic1, primary: 236, secondary: 119, diff 117
> > topic: topic2, primary: 2910, secondary: 1223, diff 1687
> > topic: topic3, primary: 2248, secondary: 2248, diff 0
> > topic: topic4, primary: 627, secondary: 91, diff 536
> > topic: topic5, primary: 84, secondary: 77, diff 7
> > topic: topic6, primary: 333, secondary: 156, diff 177
> > topic: topic7, primary: 158, secondary: 45, diff 113
> > topic: topic8, primary: 247, secondary: 48, diff 199
> > ...
> > ...
> >
> >
> > BR
> > Nils
> >
>
Re: MirrorMaker 2 does not replicate all messages?
Posted by Ryanne Dolan <ry...@gmail.com>.
Nils, are those record counts or offsets?
Ryanne
On Thu, Jan 23, 2020, 2:46 AM Nils Hermansson <ni...@getdreams.com>
wrote:
> Hello,
> I have setup MM2 and the config is similar to the example config.
>
> https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties
> Primary has 3 brokers and secondary has 1 broker.
>
> I wrote a script to diff number of messages in topics between primary and
> secondary cluster.
>
> To get number of messages in topic I use.
> bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
> localhost:9092 --topic mytopic --time -1 --offsets 1 | awk -F ":" '{sum +=
> $3} END {print sum}'
>
> Is there some reason not all messages gets replicated?
>
> A first thought was that it may have been because I wiped the secondary
> cluster and started from beginning. So i switched the name of the secondary
> cluster and prefix in mm2 config. But I ended up with the same diff.
>
> Result:
> topic: topic1, primary: 236, secondary: 119, diff 117
> topic: topic2, primary: 2910, secondary: 1223, diff 1687
> topic: topic3, primary: 2248, secondary: 2248, diff 0
> topic: topic4, primary: 627, secondary: 91, diff 536
> topic: topic5, primary: 84, secondary: 77, diff 7
> topic: topic6, primary: 333, secondary: 156, diff 177
> topic: topic7, primary: 158, secondary: 45, diff 113
> topic: topic8, primary: 247, secondary: 48, diff 199
> ...
> ...
>
>
> BR
> Nils
>