You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Chris Neal <cw...@gmail.com> on 2017/10/27 15:59:50 UTC
Help with Mirror Maker config to produce to all partitions in target cluster
Hi all,
I've been struggling with this for awhile and need some help please. :)
I have a source cluster running Kafka 0.8.1.1 with about 900 topics.
I have a target cluster running Kafka 0.10.2 with those same topics
pre-created with the same number of partitions per topic as the source
cluster.
I'm running Mirror Maker from the 0.8.1.1 Kafka version, and cannot seem to
get the produced records to go anywhere besides partition 0 of all these
topics. :(
I found this page to be helpful WRT performance tuning, but it did not
solve my partition producer problem:
https://community.hortonworks.com/articles/79891/kafka-mirror-maker-best-practices.html
My consumer.config:
======================================
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
group.id=prod_perf_mirrormaker2_new1
num.consumer.fetchers=1100
client.id=MirrorMakerConsumer_perf
======================================
My producer config:
======================================
metadata.broker.list=kafka1:9092,kafka2:9092,kafka3:9092
compression.codec=gzip
producer.type=async
batch.num.messages=1000
message.send.max.retries=2
client.id=MirrorMakerProducer_perf
======================================
My start script:
======================================
#!/bin/sh
export KAFKA_HEAP_OPTS="-Xmx2G -Xms2G"
/home/chris.neal/kafka_2.9.2-0.8.1.1/bin/kafka-run-class.sh
kafka.tools.MirrorMaker \
--producer.config
/home/chris.neal/mirror_maker/prod_perf/producer.properties \
--consumer.config
/home/chris.neal/mirror_maker/prod_perf/consumer.properties \
--num.streams=10 \
--whitelist '<snip long list of about 60 topics>'
======================================
I do know that there is some combination of "num.streams" and
"num.consumer.fetchers" that should get it working, but I can't figure out
what that combination is.
All topics have the same setup of 10 partitions per topic.
Can anyone shed some light on how to make this go across all 10 partitions,
and not just partition 0?
Thank you SO much for your time and help.
Chris
Re: Help with Mirror Maker config to produce to all partitions in
target cluster
Posted by Chris Neal <cw...@gmail.com>.
Thanks Manikumar for the reply :)
For errors in the logs, I have a few WARN messages about "Reconnect due to
socked error: null", and an occasional "Connection timed out", but that's
it.
I definitely can turn on trace logs. Is there something specific you know
of that I should look for?
WRT keys, all 10 partitions are used on the source cluster, so doesn't that
indicate that the keys are different? I don't produce the data, so I'm not
100% sure, but I can try and get ahold of the producer's source code and
see how they are being created (if at all).
I did validate that the partition counts are the same in both clusters.
I was trying to determine if I had the right values set for num.streams
and num.consumer.fetchers based on consuming from/producing to 267 topics
with 10 partitions each...or if that even matters in this case.
Chris
On Thu, Nov 2, 2017 at 12:58 AM, Manikumar <ma...@gmail.com>
wrote:
> Any exceptions in the mirror maker logs? may be you can enable mirror maker
> trace logs.
> maybe all messages are having same key? Can you recheck partition count on
> target cluster?
>
>
> On Thu, Nov 2, 2017 at 2:45 AM, Chris Neal <cw...@gmail.com> wrote:
>
> > Apologies for bumping my own post, but really hoping someone has
> experience
> > with MirrorMaker who might be able to help with my question below. :)
> >
> > Thanks!
> > Chris
> >
> > On Fri, Oct 27, 2017 at 10:59 AM, Chris Neal <cw...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I've been struggling with this for awhile and need some help please. :)
> > >
> > > I have a source cluster running Kafka 0.8.1.1 with about 900 topics.
> > > I have a target cluster running Kafka 0.10.2 with those same topics
> > > pre-created with the same number of partitions per topic as the source
> > > cluster.
> > >
> > > I'm running Mirror Maker from the 0.8.1.1 Kafka version, and cannot
> seem
> > > to get the produced records to go anywhere besides partition 0 of all
> > these
> > > topics. :(
> > >
> > > I found this page to be helpful WRT performance tuning, but it did not
> > > solve my partition producer problem:
> > >
> > > https://community.hortonworks.com/articles/79891/kafka-
> > > mirror-maker-best-practices.html
> > >
> > > My consumer.config:
> > > ======================================
> > > zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
> > > group.id=prod_perf_mirrormaker2_new1
> > > num.consumer.fetchers=1100
> > > client.id=MirrorMakerConsumer_perf
> > > ======================================
> > >
> > > My producer config:
> > > ======================================
> > > metadata.broker.list=kafka1:9092,kafka2:9092,kafka3:9092
> > > compression.codec=gzip
> > > producer.type=async
> > > batch.num.messages=1000
> > > message.send.max.retries=2
> > > client.id=MirrorMakerProducer_perf
> > > ======================================
> > >
> > > My start script:
> > > ======================================
> > > #!/bin/sh
> > >
> > > export KAFKA_HEAP_OPTS="-Xmx2G -Xms2G"
> > >
> > > /home/chris.neal/kafka_2.9.2-0.8.1.1/bin/kafka-run-class.sh
> > > kafka.tools.MirrorMaker \
> > > --producer.config /home/chris.neal/mirror_maker/
> > prod_perf/producer.properties
> > > \
> > > --consumer.config /home/chris.neal/mirror_maker/
> > prod_perf/consumer.properties
> > > \
> > > --num.streams=10 \
> > > --whitelist '<snip long list of about 60 topics>'
> > > ======================================
> > >
> > > I do know that there is some combination of "num.streams" and
> > > "num.consumer.fetchers" that should get it working, but I can't figure
> > out
> > > what that combination is.
> > >
> > > All topics have the same setup of 10 partitions per topic.
> > >
> > > Can anyone shed some light on how to make this go across all 10
> > > partitions, and not just partition 0?
> > >
> > > Thank you SO much for your time and help.
> > > Chris
> > >
> >
>
Re: Help with Mirror Maker config to produce to all partitions in
target cluster
Posted by Manikumar <ma...@gmail.com>.
Any exceptions in the mirror maker logs? may be you can enable mirror maker
trace logs.
maybe all messages are having same key? Can you recheck partition count on
target cluster?
On Thu, Nov 2, 2017 at 2:45 AM, Chris Neal <cw...@gmail.com> wrote:
> Apologies for bumping my own post, but really hoping someone has experience
> with MirrorMaker who might be able to help with my question below. :)
>
> Thanks!
> Chris
>
> On Fri, Oct 27, 2017 at 10:59 AM, Chris Neal <cw...@gmail.com> wrote:
>
> > Hi all,
> >
> > I've been struggling with this for awhile and need some help please. :)
> >
> > I have a source cluster running Kafka 0.8.1.1 with about 900 topics.
> > I have a target cluster running Kafka 0.10.2 with those same topics
> > pre-created with the same number of partitions per topic as the source
> > cluster.
> >
> > I'm running Mirror Maker from the 0.8.1.1 Kafka version, and cannot seem
> > to get the produced records to go anywhere besides partition 0 of all
> these
> > topics. :(
> >
> > I found this page to be helpful WRT performance tuning, but it did not
> > solve my partition producer problem:
> >
> > https://community.hortonworks.com/articles/79891/kafka-
> > mirror-maker-best-practices.html
> >
> > My consumer.config:
> > ======================================
> > zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
> > group.id=prod_perf_mirrormaker2_new1
> > num.consumer.fetchers=1100
> > client.id=MirrorMakerConsumer_perf
> > ======================================
> >
> > My producer config:
> > ======================================
> > metadata.broker.list=kafka1:9092,kafka2:9092,kafka3:9092
> > compression.codec=gzip
> > producer.type=async
> > batch.num.messages=1000
> > message.send.max.retries=2
> > client.id=MirrorMakerProducer_perf
> > ======================================
> >
> > My start script:
> > ======================================
> > #!/bin/sh
> >
> > export KAFKA_HEAP_OPTS="-Xmx2G -Xms2G"
> >
> > /home/chris.neal/kafka_2.9.2-0.8.1.1/bin/kafka-run-class.sh
> > kafka.tools.MirrorMaker \
> > --producer.config /home/chris.neal/mirror_maker/
> prod_perf/producer.properties
> > \
> > --consumer.config /home/chris.neal/mirror_maker/
> prod_perf/consumer.properties
> > \
> > --num.streams=10 \
> > --whitelist '<snip long list of about 60 topics>'
> > ======================================
> >
> > I do know that there is some combination of "num.streams" and
> > "num.consumer.fetchers" that should get it working, but I can't figure
> out
> > what that combination is.
> >
> > All topics have the same setup of 10 partitions per topic.
> >
> > Can anyone shed some light on how to make this go across all 10
> > partitions, and not just partition 0?
> >
> > Thank you SO much for your time and help.
> > Chris
> >
>
Re: Help with Mirror Maker config to produce to all partitions in
target cluster
Posted by Chris Neal <cw...@gmail.com>.
Apologies for bumping my own post, but really hoping someone has experience
with MirrorMaker who might be able to help with my question below. :)
Thanks!
Chris
On Fri, Oct 27, 2017 at 10:59 AM, Chris Neal <cw...@gmail.com> wrote:
> Hi all,
>
> I've been struggling with this for awhile and need some help please. :)
>
> I have a source cluster running Kafka 0.8.1.1 with about 900 topics.
> I have a target cluster running Kafka 0.10.2 with those same topics
> pre-created with the same number of partitions per topic as the source
> cluster.
>
> I'm running Mirror Maker from the 0.8.1.1 Kafka version, and cannot seem
> to get the produced records to go anywhere besides partition 0 of all these
> topics. :(
>
> I found this page to be helpful WRT performance tuning, but it did not
> solve my partition producer problem:
>
> https://community.hortonworks.com/articles/79891/kafka-
> mirror-maker-best-practices.html
>
> My consumer.config:
> ======================================
> zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
> group.id=prod_perf_mirrormaker2_new1
> num.consumer.fetchers=1100
> client.id=MirrorMakerConsumer_perf
> ======================================
>
> My producer config:
> ======================================
> metadata.broker.list=kafka1:9092,kafka2:9092,kafka3:9092
> compression.codec=gzip
> producer.type=async
> batch.num.messages=1000
> message.send.max.retries=2
> client.id=MirrorMakerProducer_perf
> ======================================
>
> My start script:
> ======================================
> #!/bin/sh
>
> export KAFKA_HEAP_OPTS="-Xmx2G -Xms2G"
>
> /home/chris.neal/kafka_2.9.2-0.8.1.1/bin/kafka-run-class.sh
> kafka.tools.MirrorMaker \
> --producer.config /home/chris.neal/mirror_maker/prod_perf/producer.properties
> \
> --consumer.config /home/chris.neal/mirror_maker/prod_perf/consumer.properties
> \
> --num.streams=10 \
> --whitelist '<snip long list of about 60 topics>'
> ======================================
>
> I do know that there is some combination of "num.streams" and
> "num.consumer.fetchers" that should get it working, but I can't figure out
> what that combination is.
>
> All topics have the same setup of 10 partitions per topic.
>
> Can anyone shed some light on how to make this go across all 10
> partitions, and not just partition 0?
>
> Thank you SO much for your time and help.
> Chris
>