You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by prabhu v <pr...@gmail.com> on 2016/03/09 10:56:57 UTC
Mirror maker Configs 0.9.0
Hi Experts,
I am trying to mirror
--
Regards,
Prabhu.V
RE: Mirror maker Configs 0.9.0
Posted by Pr...@thomsonreuters.com.
Thanks a lot Stephen..
It worked:)
-----Original Message-----
From: Stephen Powis [mailto:spowis@salesforce.com]
Sent: Wednesday, March 09, 2016 6:58 PM
To: users@kafka.apache.org
Cc: Kishore N R
Subject: Re: Mirror maker Configs 0.9.0
I've attached my two configs here. Pay close attention to the --num-streams argument to mirror-maker. I have a lot of thru-put on my topics so I ended up matching the number of streams = number of partitions for each of my topics. A stream is essentially just a consumer and producer thread. If you have less streams/threads than number of partitions * topics replicated, it seems like mirror-maker roundrobins the threads across the partitions/topics, which may be fine based on your ingest rate and network latency between clusters.
Additionally, I had to raise the java heap size considerably to prevent mirror-maker from OOMing.
You can check the status of mirror maker and get the lag by running:
kafka-consumer-offset-checker.sh --topic "your-topic-here" --zookeeper
src-zk-host:2181 --group mirrormaker-consumer-id-here
I *think* I also edited config/tools-log4j.properties and made mirror-maker log info level messages by changing the first line to read as follows.
This allowed me to more easily debug issues and fine tune my settings.
log4j.rootLogger=INFO, fileAppender, stderr
consumer.properties
##################
# Zookeeper connection string
# comma separated host:port pairs, each corresponding to a zk # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002"
zookeeper.connect="source-zk-host1:2181,source-zk-host2:2181"
# timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
#consumer group id
group.id=mirrormaker-consumer-id-here
producer.properties
##################
# list of brokers used for bootstrapping knowledge about the rest of the cluster # format: host1:port1,host2:port2 ...
bootstrap.servers=dest-kafka-host1:9092,dest-kafka-host2:9092
# name of the partitioner class for partitioning events; default partition spreads data randomly (in most cases) ## I needed to use a custom partitioner to get my data to be distributed evenly, but you'll probably ## be fine leaving this commented out. see jira kafka-3333 if you're curious to what I used.
# partitioner.class=
# specify the compression codec for all data generated: none, gzip, snappy, lz4.
# the old config values work as well: 0, 1, 2, 3 for none, gzip, snappy, lz4, respectively compression.codec=none
## Increment size of batches
batch.size:65536
client.id: mirror-maker
On Wed, Mar 9, 2016 at 5:43 AM, prabhu v <pr...@gmail.com> wrote:
> Thanks for the reply..
>
> I will remove the bootstrap.servers property and add zookeeper.connect
> in consumer properties and let you know....
>
> Also, is there any way we can check how much data the target data
> center is lagging behind source DC?
>
>
> On Wed, Mar 9, 2016 at 3:41 PM, Gerard Klijs <ge...@dizzit.com>
> wrote:
>
> > What do you see in the logs?
> > It could be it goes wrong because you have the bootstrap.servers
> > property which is not supported for the old consumer.
> >
> > On Wed, Mar 9, 2016 at 11:05 AM Gerard Klijs
> > <ge...@dizzit.com>
> > wrote:
> >
> > > Don't know the actual question, it matters what you want to do.
> > > Just watch out trying to copy every topic using a new consumer,
> > > cause
> > then
> > > internal topics are copied, leading to errors.
> > > Here is a temple start script we used:
> > >
> > > #!/usr/bin/env bash
> > > export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dcom.sun.management.jmxremote.ssl=false
> > -Dcom.sun.management.jmxremote.local.only=false
> > -Djava.rmi.server.hostname=<NODE_HOSTNAME>
> > -Dcom.sun.management.jmxremote.rmi.port=<MM_JMX_PORT>"
> > > export JMX_PORT=<MM_JMX_PORT>
> > > /usr/bin/kafka-mirror-maker --consumer.config
> > $HOME_DIR/consumer.properties --producer.config
> > $HOME_DIR/producer.properties --whitelist='<REGEX_MM>' 1>>
> > $LOG_DIR/mirror-maker.log 2>> $LOG_DIR/mirror-maker.log
> > >
> > > Both the consumer and producer configs have sensible defaults,
> > > these
> are
> > out consumer.properties template:
> > >
> > > #Consumer template to be used with the mirror maker
> > > zookeeper.connect=<CONSUMER_ZOOKEEPERS>
> > > group.id=mirrormaker
> > > auto.offset.reset=smallest
> > > #next property is not available in new consumer
> > > exclude.internal.topics=true
> > >
> > > *And a producer.properties template:*
> > >
> > > #Producer template to be used with the mirror maker
> > > bootstrap.servers=<PRODUCER_BROKERS>
> > > client.id=mirrormaker
> > >
> > > Because the internal topics can't be excluded in the new consumer
> > > yet,
> > we use the old consumer.
> > >
> > > Hope this helps.
> > >
> > >
> > > On Wed, Mar 9, 2016 at 10:57 AM prabhu v
> > > <pr...@gmail.com>
> > wrote:
> > >
> > >> Hi Experts,
> > >>
> > >> I am trying to mirror
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >> Regards,
> > >>
> > >> Prabhu.V
> > >>
> > >
> >
>
>
>
> --
> Regards,
>
> Prabhu.V
>
Re: Mirror maker Configs 0.9.0
Posted by Stephen Powis <sp...@salesforce.com>.
I've attached my two configs here. Pay close attention to the
--num-streams argument to mirror-maker. I have a lot of thru-put on my
topics so I ended up
matching the number of streams = number of partitions for each of my
topics. A stream is essentially just a consumer and producer thread. If
you have less streams/threads than number of partitions * topics
replicated, it seems like mirror-maker roundrobins the threads across the
partitions/topics, which may be fine based on your ingest rate and network
latency between clusters.
Additionally, I had to raise the java heap size considerably to prevent
mirror-maker from OOMing.
You can check the status of mirror maker and get the lag by running:
kafka-consumer-offset-checker.sh --topic "your-topic-here" --zookeeper
src-zk-host:2181 --group mirrormaker-consumer-id-here
I *think* I also edited config/tools-log4j.properties and made mirror-maker
log info level messages by changing the first line to read as follows.
This allowed me to more easily debug issues and fine tune my settings.
log4j.rootLogger=INFO, fileAppender, stderr
consumer.properties
##################
# Zookeeper connection string
# comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002"
zookeeper.connect="source-zk-host1:2181,source-zk-host2:2181"
# timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
#consumer group id
group.id=mirrormaker-consumer-id-here
producer.properties
##################
# list of brokers used for bootstrapping knowledge about the rest of the cluster
# format: host1:port1,host2:port2 ...
bootstrap.servers=dest-kafka-host1:9092,dest-kafka-host2:9092
# name of the partitioner class for partitioning events; default
partition spreads data randomly (in most cases)
## I needed to use a custom partitioner to get my data to be
distributed evenly, but you'll probably
## be fine leaving this commented out. see jira kafka-3333 if you're
curious to what I used.
# partitioner.class=
# specify the compression codec for all data generated: none, gzip, snappy, lz4.
# the old config values work as well: 0, 1, 2, 3 for none, gzip,
snappy, lz4, respectively
compression.codec=none
## Increment size of batches
batch.size:65536
client.id: mirror-maker
On Wed, Mar 9, 2016 at 5:43 AM, prabhu v <pr...@gmail.com> wrote:
> Thanks for the reply..
>
> I will remove the bootstrap.servers property and add zookeeper.connect in
> consumer properties and let you know....
>
> Also, is there any way we can check how much data the target data center is
> lagging behind source DC?
>
>
> On Wed, Mar 9, 2016 at 3:41 PM, Gerard Klijs <ge...@dizzit.com>
> wrote:
>
> > What do you see in the logs?
> > It could be it goes wrong because you have the bootstrap.servers property
> > which is not supported for the old consumer.
> >
> > On Wed, Mar 9, 2016 at 11:05 AM Gerard Klijs <ge...@dizzit.com>
> > wrote:
> >
> > > Don't know the actual question, it matters what you want to do.
> > > Just watch out trying to copy every topic using a new consumer, cause
> > then
> > > internal topics are copied, leading to errors.
> > > Here is a temple start script we used:
> > >
> > > #!/usr/bin/env bash
> > > export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dcom.sun.management.jmxremote.ssl=false
> > -Dcom.sun.management.jmxremote.local.only=false
> > -Djava.rmi.server.hostname=<NODE_HOSTNAME>
> > -Dcom.sun.management.jmxremote.rmi.port=<MM_JMX_PORT>"
> > > export JMX_PORT=<MM_JMX_PORT>
> > > /usr/bin/kafka-mirror-maker --consumer.config
> > $HOME_DIR/consumer.properties --producer.config
> > $HOME_DIR/producer.properties --whitelist='<REGEX_MM>' 1>>
> > $LOG_DIR/mirror-maker.log 2>> $LOG_DIR/mirror-maker.log
> > >
> > > Both the consumer and producer configs have sensible defaults, these
> are
> > out consumer.properties template:
> > >
> > > #Consumer template to be used with the mirror maker
> > > zookeeper.connect=<CONSUMER_ZOOKEEPERS>
> > > group.id=mirrormaker
> > > auto.offset.reset=smallest
> > > #next property is not available in new consumer
> > > exclude.internal.topics=true
> > >
> > > *And a producer.properties template:*
> > >
> > > #Producer template to be used with the mirror maker
> > > bootstrap.servers=<PRODUCER_BROKERS>
> > > client.id=mirrormaker
> > >
> > > Because the internal topics can't be excluded in the new consumer yet,
> > we use the old consumer.
> > >
> > > Hope this helps.
> > >
> > >
> > > On Wed, Mar 9, 2016 at 10:57 AM prabhu v <pr...@gmail.com>
> > wrote:
> > >
> > >> Hi Experts,
> > >>
> > >> I am trying to mirror
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >> Regards,
> > >>
> > >> Prabhu.V
> > >>
> > >
> >
>
>
>
> --
> Regards,
>
> Prabhu.V
>
Re: Mirror maker Configs 0.9.0
Posted by prabhu v <pr...@gmail.com>.
Thanks for the reply..
I will remove the bootstrap.servers property and add zookeeper.connect in
consumer properties and let you know....
Also, is there any way we can check how much data the target data center is
lagging behind source DC?
On Wed, Mar 9, 2016 at 3:41 PM, Gerard Klijs <ge...@dizzit.com>
wrote:
> What do you see in the logs?
> It could be it goes wrong because you have the bootstrap.servers property
> which is not supported for the old consumer.
>
> On Wed, Mar 9, 2016 at 11:05 AM Gerard Klijs <ge...@dizzit.com>
> wrote:
>
> > Don't know the actual question, it matters what you want to do.
> > Just watch out trying to copy every topic using a new consumer, cause
> then
> > internal topics are copied, leading to errors.
> > Here is a temple start script we used:
> >
> > #!/usr/bin/env bash
> > export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.local.only=false
> -Djava.rmi.server.hostname=<NODE_HOSTNAME>
> -Dcom.sun.management.jmxremote.rmi.port=<MM_JMX_PORT>"
> > export JMX_PORT=<MM_JMX_PORT>
> > /usr/bin/kafka-mirror-maker --consumer.config
> $HOME_DIR/consumer.properties --producer.config
> $HOME_DIR/producer.properties --whitelist='<REGEX_MM>' 1>>
> $LOG_DIR/mirror-maker.log 2>> $LOG_DIR/mirror-maker.log
> >
> > Both the consumer and producer configs have sensible defaults, these are
> out consumer.properties template:
> >
> > #Consumer template to be used with the mirror maker
> > zookeeper.connect=<CONSUMER_ZOOKEEPERS>
> > group.id=mirrormaker
> > auto.offset.reset=smallest
> > #next property is not available in new consumer
> > exclude.internal.topics=true
> >
> > *And a producer.properties template:*
> >
> > #Producer template to be used with the mirror maker
> > bootstrap.servers=<PRODUCER_BROKERS>
> > client.id=mirrormaker
> >
> > Because the internal topics can't be excluded in the new consumer yet,
> we use the old consumer.
> >
> > Hope this helps.
> >
> >
> > On Wed, Mar 9, 2016 at 10:57 AM prabhu v <pr...@gmail.com>
> wrote:
> >
> >> Hi Experts,
> >>
> >> I am trying to mirror
> >>
> >>
> >>
> >>
> >> --
> >> Regards,
> >>
> >> Prabhu.V
> >>
> >
>
--
Regards,
Prabhu.V
Re: Mirror maker Configs 0.9.0
Posted by Gerard Klijs <ge...@dizzit.com>.
What do you see in the logs?
It could be it goes wrong because you have the bootstrap.servers property
which is not supported for the old consumer.
On Wed, Mar 9, 2016 at 11:05 AM Gerard Klijs <ge...@dizzit.com>
wrote:
> Don't know the actual question, it matters what you want to do.
> Just watch out trying to copy every topic using a new consumer, cause then
> internal topics are copied, leading to errors.
> Here is a temple start script we used:
>
> #!/usr/bin/env bash
> export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=<NODE_HOSTNAME> -Dcom.sun.management.jmxremote.rmi.port=<MM_JMX_PORT>"
> export JMX_PORT=<MM_JMX_PORT>
> /usr/bin/kafka-mirror-maker --consumer.config $HOME_DIR/consumer.properties --producer.config $HOME_DIR/producer.properties --whitelist='<REGEX_MM>' 1>> $LOG_DIR/mirror-maker.log 2>> $LOG_DIR/mirror-maker.log
>
> Both the consumer and producer configs have sensible defaults, these are out consumer.properties template:
>
> #Consumer template to be used with the mirror maker
> zookeeper.connect=<CONSUMER_ZOOKEEPERS>
> group.id=mirrormaker
> auto.offset.reset=smallest
> #next property is not available in new consumer
> exclude.internal.topics=true
>
> *And a producer.properties template:*
>
> #Producer template to be used with the mirror maker
> bootstrap.servers=<PRODUCER_BROKERS>
> client.id=mirrormaker
>
> Because the internal topics can't be excluded in the new consumer yet, we use the old consumer.
>
> Hope this helps.
>
>
> On Wed, Mar 9, 2016 at 10:57 AM prabhu v <pr...@gmail.com> wrote:
>
>> Hi Experts,
>>
>> I am trying to mirror
>>
>>
>>
>>
>> --
>> Regards,
>>
>> Prabhu.V
>>
>
Re: Mirror maker Configs 0.9.0
Posted by Gerard Klijs <ge...@dizzit.com>.
Don't know the actual question, it matters what you want to do.
Just watch out trying to copy every topic using a new consumer, cause then
internal topics are copied, leading to errors.
Here is a temple start script we used:
#!/usr/bin/env bash
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Djava.rmi.server.hostname=<NODE_HOSTNAME>
-Dcom.sun.management.jmxremote.rmi.port=<MM_JMX_PORT>"
export JMX_PORT=<MM_JMX_PORT>
/usr/bin/kafka-mirror-maker --consumer.config
$HOME_DIR/consumer.properties --producer.config
$HOME_DIR/producer.properties --whitelist='<REGEX_MM>' 1>>
$LOG_DIR/mirror-maker.log 2>> $LOG_DIR/mirror-maker.log
Both the consumer and producer configs have sensible defaults, these
are out consumer.properties template:
#Consumer template to be used with the mirror maker
zookeeper.connect=<CONSUMER_ZOOKEEPERS>
group.id=mirrormaker
auto.offset.reset=smallest
#next property is not available in new consumer
exclude.internal.topics=true
*And a producer.properties template:*
#Producer template to be used with the mirror maker
bootstrap.servers=<PRODUCER_BROKERS>
client.id=mirrormaker
Because the internal topics can't be excluded in the new consumer yet,
we use the old consumer.
Hope this helps.
On Wed, Mar 9, 2016 at 10:57 AM prabhu v <pr...@gmail.com> wrote:
> Hi Experts,
>
> I am trying to mirror
>
>
>
>
> --
> Regards,
>
> Prabhu.V
>
Re: Mirror maker Configs 0.9.0
Posted by prabhu v <pr...@gmail.com>.
Hi Experts,
I am trying to replicate data between different data centers using mirror
maker tool.
kafka-run-class.bat kafka.tools.MirrorMaker --consumer.config
consumer.properties --producer.config producer.properties --whitelist *
Can someone provide the sample consumer.properties and producer.properties
that we use for the above?
I have tried using the below configs, seems i am missing something..
Producer configs:
bootstrap.servers=us1s-cspapsv15:9097
producer.type = async
connect.timeout.ms = 10000
request.required.acks = 0
zookeeper.connect = us1s-cspapsv15:2181
serializer.class = kafka.serializer.DefaultEncoder
Consumer configs:
bootstrap.servers=c015cjqcmap01:9095
zookeeper.connect=c015cjqcmap01:2181
group.id=test-consumer-group
zookeeper.sync.time.ms = 2000
zookeeper.session.timeout.ms = 2000
zookeeper.connection.timeout.ms = 6000
Also to check the consumer position, I have tried using
kafka.tools.ConsumerOffsetChecker, seems it is no longer supported in 0.9
release.. Can some one tell me on how to check the consumer position in 0.9
release?
Thanks,
Prabhu
Thanks,
Prabhu
On Wed, Mar 9, 2016 at 3:26 PM, prabhu v <pr...@gmail.com> wrote:
> Hi Experts,
>
> I am trying to mirror
>
>
>
>
> --
> Regards,
>
> Prabhu.V
>
>
--
Regards,
Prabhu.V