You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Alex Melville <am...@g.hmc.edu> on 2015/02/15 01:39:02 UTC
Default MirrorMaker not copying over from source to target
Hi Kafka'ers,
I am trying to get the Mirrormaker working with two separate clusters, one
as the source and the other as the target. The topic I'm trying to copy
over exists on both the source and target clusters. Here are the relevant
entries in my consumer and producer properties files, which I'm specifying
the command I run to start the MM:
*mirrorconsumer.properties:*
zookeeper.connect=ad-0104:2181
zookeeper.connection.timeout.ms=6000
group.id=test-consumer-group
*mirrorproducer.properties:*
metadata.broker.list=ba-02:9092,ba-03:9092
producer.type=sync
compression.codec=none
serializer.class=kafka.serializer.DefaultEncoder
Then I run the following command:
bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
../config/mirrorconsumer.properties --producer.config
../config/mirrorproducer.properties --whitelist consolemm
so consolemm is the topic I'm trying to copy over. I've created consolemm
and have used to console-consumer to verify that there are messages in the
topic.
When I run this command... nothing happens. The process keeps running and
prints nothing to the Terminal. If I look in the output of the zookeeper on
the source cluster I get only the following:
[2015-02-15 00:34:06,102] INFO Accepted socket connection from /
10.7.162.75:42819 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-02-15 00:34:06,104] INFO Client attempting to establish new session
at /10.7.162.75:42819 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-02-15 00:34:06,106] INFO Established session 0x14b668b0fbe0033 with
negotiated timeout 6000 for client /10.7.162.75:42819
(org.apache.zookeeper.server.ZooKeeperServer)
and when I look at the output of one of the brokers on the source cluster I
get:
[2015-02-15 00:32:14,382] INFO Closing socket connection to /10.7.162.75.
(kafka.network.Processor)
and there is no output on the zookeeper on the target cluster.
Any advice on what is causing MM to not properly copy over data to the
target cluster would be extremely helpful.
-Alex
Re: Default MirrorMaker not copying over from source to target
Posted by tao xiao <xi...@gmail.com>.
Looks like you only have 4 messages in your topic and no more messages got
sent
2015-02-19 20:09:34,661] DEBUG initial fetch offset of consolemm:0: fetched
offset = 4: consumed offset = 4 is 4 (kafka.consumer.PartitionTopicInfo
You can try sending more messages to topic or give the MM a different
consumer group id and set auto.offset.reset=smallest
On Friday, February 20, 2015, Alex Melville <am...@g.hmc.edu> wrote:
> Tao,
>
>
> I updated the mirrorconsumer.properties config file as you suggested, and
> upped the MM's log level to DEBUG. I have the output of the DEBUG logger
> here in this pastebin, if you could take a minute to look for anything in
> its contents that would indicate a problem that would be extremely helpful.
> Note that my servers hostnames are of the form ad-010X or ba-0X where X is
> some integer between 1 and 4.
>
> http://pastebin.com/rBsxx15A
>
> When I run the mirrormaker and then spin up a console consumer to read from
> the source cluster, I get 0 messages consumed.
>
>
> Alex
>
> On Sun, Feb 15, 2015 at 3:00 AM, tao xiao <xiaotao183@gmail.com
> <javascript:;>> wrote:
>
> > Alex,
> >
> > Are you sure you have data continually being sent to the topic in source
> > cluster after you bring up MM? By default auto.offset.reset=largest in MM
> > consumer config which means MM only fetches the largest offset if the
> > consumer group has no initial offset in zookeeper.
> >
> > You can have MM print more log by changing the log level in
> > config/tools-log4j.properties
> >
> > On Sun, Feb 15, 2015 at 8:39 AM, Alex Melville <amelville@g.hmc.edu
> <javascript:;>>
> > wrote:
> >
> > > Hi Kafka'ers,
> > >
> > >
> > > I am trying to get the Mirrormaker working with two separate clusters,
> > one
> > > as the source and the other as the target. The topic I'm trying to copy
> > > over exists on both the source and target clusters. Here are the
> relevant
> > > entries in my consumer and producer properties files, which I'm
> > specifying
> > > the command I run to start the MM:
> > >
> > > *mirrorconsumer.properties:*
> > > zookeeper.connect=ad-0104:2181
> > > zookeeper.connection.timeout.ms=6000
> > > group.id=test-consumer-group
> > >
> > >
> > > *mirrorproducer.properties:*
> > > metadata.broker.list=ba-02:9092,ba-03:9092
> > > producer.type=sync
> > > compression.codec=none
> > > serializer.class=kafka.serializer.DefaultEncoder
> > >
> > >
> > > Then I run the following command:
> > > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
> > > ../config/mirrorconsumer.properties --producer.config
> > > ../config/mirrorproducer.properties --whitelist consolemm
> > >
> > > so consolemm is the topic I'm trying to copy over. I've created
> consolemm
> > > and have used to console-consumer to verify that there are messages in
> > the
> > > topic.
> > >
> > > When I run this command... nothing happens. The process keeps running
> and
> > > prints nothing to the Terminal. If I look in the output of the
> zookeeper
> > on
> > > the source cluster I get only the following:
> > >
> > > [2015-02-15 00:34:06,102] INFO Accepted socket connection from /
> > > 10.7.162.75:42819 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> > > [2015-02-15 00:34:06,104] INFO Client attempting to establish new
> session
> > > at /10.7.162.75:42819 (org.apache.zookeeper.server.ZooKeeperServer)
> > > [2015-02-15 00:34:06,106] INFO Established session 0x14b668b0fbe0033
> with
> > > negotiated timeout 6000 for client /10.7.162.75:42819
> > > (org.apache.zookeeper.server.ZooKeeperServer)
> > >
> > >
> > > and when I look at the output of one of the brokers on the source
> > cluster I
> > > get:
> > >
> > > [2015-02-15 00:32:14,382] INFO Closing socket connection to /
> 10.7.162.75
> > .
> > > (kafka.network.Processor)
> > >
> > > and there is no output on the zookeeper on the target cluster.
> > >
> > >
> > >
> > > Any advice on what is causing MM to not properly copy over data to the
> > > target cluster would be extremely helpful.
> > >
> > > -Alex
> > >
> >
> >
> >
> > --
> > Regards,
> > Tao
> >
>
--
Regards,
Tao
Re: Default MirrorMaker not copying over from source to target
Posted by Alex Melville <am...@g.hmc.edu>.
Tao,
I updated the mirrorconsumer.properties config file as you suggested, and
upped the MM's log level to DEBUG. I have the output of the DEBUG logger
here in this pastebin, if you could take a minute to look for anything in
its contents that would indicate a problem that would be extremely helpful.
Note that my servers hostnames are of the form ad-010X or ba-0X where X is
some integer between 1 and 4.
http://pastebin.com/rBsxx15A
When I run the mirrormaker and then spin up a console consumer to read from
the source cluster, I get 0 messages consumed.
Alex
On Sun, Feb 15, 2015 at 3:00 AM, tao xiao <xi...@gmail.com> wrote:
> Alex,
>
> Are you sure you have data continually being sent to the topic in source
> cluster after you bring up MM? By default auto.offset.reset=largest in MM
> consumer config which means MM only fetches the largest offset if the
> consumer group has no initial offset in zookeeper.
>
> You can have MM print more log by changing the log level in
> config/tools-log4j.properties
>
> On Sun, Feb 15, 2015 at 8:39 AM, Alex Melville <am...@g.hmc.edu>
> wrote:
>
> > Hi Kafka'ers,
> >
> >
> > I am trying to get the Mirrormaker working with two separate clusters,
> one
> > as the source and the other as the target. The topic I'm trying to copy
> > over exists on both the source and target clusters. Here are the relevant
> > entries in my consumer and producer properties files, which I'm
> specifying
> > the command I run to start the MM:
> >
> > *mirrorconsumer.properties:*
> > zookeeper.connect=ad-0104:2181
> > zookeeper.connection.timeout.ms=6000
> > group.id=test-consumer-group
> >
> >
> > *mirrorproducer.properties:*
> > metadata.broker.list=ba-02:9092,ba-03:9092
> > producer.type=sync
> > compression.codec=none
> > serializer.class=kafka.serializer.DefaultEncoder
> >
> >
> > Then I run the following command:
> > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
> > ../config/mirrorconsumer.properties --producer.config
> > ../config/mirrorproducer.properties --whitelist consolemm
> >
> > so consolemm is the topic I'm trying to copy over. I've created consolemm
> > and have used to console-consumer to verify that there are messages in
> the
> > topic.
> >
> > When I run this command... nothing happens. The process keeps running and
> > prints nothing to the Terminal. If I look in the output of the zookeeper
> on
> > the source cluster I get only the following:
> >
> > [2015-02-15 00:34:06,102] INFO Accepted socket connection from /
> > 10.7.162.75:42819 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> > [2015-02-15 00:34:06,104] INFO Client attempting to establish new session
> > at /10.7.162.75:42819 (org.apache.zookeeper.server.ZooKeeperServer)
> > [2015-02-15 00:34:06,106] INFO Established session 0x14b668b0fbe0033 with
> > negotiated timeout 6000 for client /10.7.162.75:42819
> > (org.apache.zookeeper.server.ZooKeeperServer)
> >
> >
> > and when I look at the output of one of the brokers on the source
> cluster I
> > get:
> >
> > [2015-02-15 00:32:14,382] INFO Closing socket connection to /10.7.162.75
> .
> > (kafka.network.Processor)
> >
> > and there is no output on the zookeeper on the target cluster.
> >
> >
> >
> > Any advice on what is causing MM to not properly copy over data to the
> > target cluster would be extremely helpful.
> >
> > -Alex
> >
>
>
>
> --
> Regards,
> Tao
>
Re: Default MirrorMaker not copying over from source to target
Posted by tao xiao <xi...@gmail.com>.
Alex,
Are you sure you have data continually being sent to the topic in source
cluster after you bring up MM? By default auto.offset.reset=largest in MM
consumer config which means MM only fetches the largest offset if the
consumer group has no initial offset in zookeeper.
You can have MM print more log by changing the log level in
config/tools-log4j.properties
On Sun, Feb 15, 2015 at 8:39 AM, Alex Melville <am...@g.hmc.edu> wrote:
> Hi Kafka'ers,
>
>
> I am trying to get the Mirrormaker working with two separate clusters, one
> as the source and the other as the target. The topic I'm trying to copy
> over exists on both the source and target clusters. Here are the relevant
> entries in my consumer and producer properties files, which I'm specifying
> the command I run to start the MM:
>
> *mirrorconsumer.properties:*
> zookeeper.connect=ad-0104:2181
> zookeeper.connection.timeout.ms=6000
> group.id=test-consumer-group
>
>
> *mirrorproducer.properties:*
> metadata.broker.list=ba-02:9092,ba-03:9092
> producer.type=sync
> compression.codec=none
> serializer.class=kafka.serializer.DefaultEncoder
>
>
> Then I run the following command:
> bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
> ../config/mirrorconsumer.properties --producer.config
> ../config/mirrorproducer.properties --whitelist consolemm
>
> so consolemm is the topic I'm trying to copy over. I've created consolemm
> and have used to console-consumer to verify that there are messages in the
> topic.
>
> When I run this command... nothing happens. The process keeps running and
> prints nothing to the Terminal. If I look in the output of the zookeeper on
> the source cluster I get only the following:
>
> [2015-02-15 00:34:06,102] INFO Accepted socket connection from /
> 10.7.162.75:42819 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> [2015-02-15 00:34:06,104] INFO Client attempting to establish new session
> at /10.7.162.75:42819 (org.apache.zookeeper.server.ZooKeeperServer)
> [2015-02-15 00:34:06,106] INFO Established session 0x14b668b0fbe0033 with
> negotiated timeout 6000 for client /10.7.162.75:42819
> (org.apache.zookeeper.server.ZooKeeperServer)
>
>
> and when I look at the output of one of the brokers on the source cluster I
> get:
>
> [2015-02-15 00:32:14,382] INFO Closing socket connection to /10.7.162.75.
> (kafka.network.Processor)
>
> and there is no output on the zookeeper on the target cluster.
>
>
>
> Any advice on what is causing MM to not properly copy over data to the
> target cluster would be extremely helpful.
>
> -Alex
>
--
Regards,
Tao