You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by svante karlsson <sa...@csi.se> on 2015/01/06 00:49:26 UTC
mirrormaker tool in 0.82beta
I'm using 0.82beta and I'm trying to push data with the mirrormaker tool
from several remote sites to two datacenters. I'm testing this from a node
containing zk, broker and mirrormaker and the data is pushed to a "normal"
cluster. 3 zk and 4 brokers with replication.
While the configuration seems straight forward things are a bit shaky.
First, if something goes wrong on either consumer or producer side it stops
transmitting data without dying which makes it harder to run under a
supervisor (upstart in my case)
I start pushing data to site 02 and mirrormaker pushes it to my datacenter
- so far all good...
But, when I run. (on my remote site)
kafka-consumer-offset-checker.sh
--group mirrormaker.from.site_02.to.site_ktv --zookeeper 192.168.0.106
Could not fetch offset for [saka.test.ext_datastream,7] due to
kafka.common.NotCoordinatorForConsumerException.
Could not fetch offset for [saka.test.ext_datastream,4] due to
kafka.common.NotCoordinatorForConsumerException.
Could not fetch offset for [saka.test.ext_datastream,2] due to
kafka.common.NotCoordinatorForConsumerException.
Could not fetch offset for [saka.test.ext_datastream,5] due to
kafka.common.NotCoordinatorForConsumerException.
Could not fetch offset for [saka.test.ext_datastream,6] due to
kafka.common.NotCoordinatorForConsumerException.
Could not fetch offset for [saka.test.ext_datastream,3] due to
kafka.common.NotCoordinatorForConsumerException.
Could not fetch offset for [saka.test.ext_datastream,1] due to
kafka.common.NotCoordinatorForConsumerException.
Could not fetch offset for [saka.test.ext_datastream,0] due to
kafka.common.NotCoordinatorForConsumerException.
Group Topic Pid Offset logSize
Lag Owner
mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 0
unknown 9310560 unknown
mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 1
unknown 9313497 unknown
mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 2
unknown 9323623 unknown
mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 3
unknown 9334005 unknown
mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 4
unknown 9324176 unknown
mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 5
unknown 9336504 unknown
mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 6
unknown 9316139 unknown
mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 7
unknown 9318770 unknown
mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
Since I can push data between my sites something is working but what can be
the case for NotCoordinatorForConsumerException on a single node cluster?
Finally the __consumer_offsets topic should probably be masked out from
mirroring without using whitelists or blacklists
thanks
/svante
Re: mirrormaker tool in 0.82beta
Posted by svante karlsson <sa...@csi.se>.
No, I missed that.
thanks,
svante
2015-01-07 6:44 GMT+01:00 Jun Rao <ju...@confluent.io>:
> Did you set offsets.storage to kafka in the consumer of mirror maker?
>
> Thanks,
>
> Jun
>
> On Mon, Jan 5, 2015 at 3:49 PM, svante karlsson <sa...@csi.se> wrote:
>
> > I'm using 0.82beta and I'm trying to push data with the mirrormaker tool
> > from several remote sites to two datacenters. I'm testing this from a
> node
> > containing zk, broker and mirrormaker and the data is pushed to a
> "normal"
> > cluster. 3 zk and 4 brokers with replication.
> >
> > While the configuration seems straight forward things are a bit shaky.
> >
> > First, if something goes wrong on either consumer or producer side it
> stops
> > transmitting data without dying which makes it harder to run under a
> > supervisor (upstart in my case)
> >
> > I start pushing data to site 02 and mirrormaker pushes it to my
> datacenter
> > - so far all good...
> >
> > But, when I run. (on my remote site)
> > kafka-consumer-offset-checker.sh
> > --group mirrormaker.from.site_02.to.site_ktv --zookeeper 192.168.0.106
> >
> > Could not fetch offset for [saka.test.ext_datastream,7] due to
> > kafka.common.NotCoordinatorForConsumerException.
> > Could not fetch offset for [saka.test.ext_datastream,4] due to
> > kafka.common.NotCoordinatorForConsumerException.
> > Could not fetch offset for [saka.test.ext_datastream,2] due to
> > kafka.common.NotCoordinatorForConsumerException.
> > Could not fetch offset for [saka.test.ext_datastream,5] due to
> > kafka.common.NotCoordinatorForConsumerException.
> > Could not fetch offset for [saka.test.ext_datastream,6] due to
> > kafka.common.NotCoordinatorForConsumerException.
> > Could not fetch offset for [saka.test.ext_datastream,3] due to
> > kafka.common.NotCoordinatorForConsumerException.
> > Could not fetch offset for [saka.test.ext_datastream,1] due to
> > kafka.common.NotCoordinatorForConsumerException.
> > Could not fetch offset for [saka.test.ext_datastream,0] due to
> > kafka.common.NotCoordinatorForConsumerException.
> >
> > Group Topic Pid Offset
> logSize
> > Lag Owner
> > mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 0
> > unknown 9310560 unknown
> > mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> > mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 1
> > unknown 9313497 unknown
> > mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> > mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 2
> > unknown 9323623 unknown
> > mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> > mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 3
> > unknown 9334005 unknown
> > mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> > mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 4
> > unknown 9324176 unknown
> > mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> > mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 5
> > unknown 9336504 unknown
> > mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> > mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 6
> > unknown 9316139 unknown
> > mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> > mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 7
> > unknown 9318770 unknown
> > mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> >
> >
> > Since I can push data between my sites something is working but what can
> be
> > the case for NotCoordinatorForConsumerException on a single node cluster?
> >
> >
> > Finally the __consumer_offsets topic should probably be masked out from
> > mirroring without using whitelists or blacklists
> >
> >
> > thanks
> > /svante
> >
>
Re: mirrormaker tool in 0.82beta
Posted by Jun Rao <ju...@confluent.io>.
Did you set offsets.storage to kafka in the consumer of mirror maker?
Thanks,
Jun
On Mon, Jan 5, 2015 at 3:49 PM, svante karlsson <sa...@csi.se> wrote:
> I'm using 0.82beta and I'm trying to push data with the mirrormaker tool
> from several remote sites to two datacenters. I'm testing this from a node
> containing zk, broker and mirrormaker and the data is pushed to a "normal"
> cluster. 3 zk and 4 brokers with replication.
>
> While the configuration seems straight forward things are a bit shaky.
>
> First, if something goes wrong on either consumer or producer side it stops
> transmitting data without dying which makes it harder to run under a
> supervisor (upstart in my case)
>
> I start pushing data to site 02 and mirrormaker pushes it to my datacenter
> - so far all good...
>
> But, when I run. (on my remote site)
> kafka-consumer-offset-checker.sh
> --group mirrormaker.from.site_02.to.site_ktv --zookeeper 192.168.0.106
>
> Could not fetch offset for [saka.test.ext_datastream,7] due to
> kafka.common.NotCoordinatorForConsumerException.
> Could not fetch offset for [saka.test.ext_datastream,4] due to
> kafka.common.NotCoordinatorForConsumerException.
> Could not fetch offset for [saka.test.ext_datastream,2] due to
> kafka.common.NotCoordinatorForConsumerException.
> Could not fetch offset for [saka.test.ext_datastream,5] due to
> kafka.common.NotCoordinatorForConsumerException.
> Could not fetch offset for [saka.test.ext_datastream,6] due to
> kafka.common.NotCoordinatorForConsumerException.
> Could not fetch offset for [saka.test.ext_datastream,3] due to
> kafka.common.NotCoordinatorForConsumerException.
> Could not fetch offset for [saka.test.ext_datastream,1] due to
> kafka.common.NotCoordinatorForConsumerException.
> Could not fetch offset for [saka.test.ext_datastream,0] due to
> kafka.common.NotCoordinatorForConsumerException.
>
> Group Topic Pid Offset logSize
> Lag Owner
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 0
> unknown 9310560 unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 1
> unknown 9313497 unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 2
> unknown 9323623 unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 3
> unknown 9334005 unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 4
> unknown 9324176 unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 5
> unknown 9336504 unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 6
> unknown 9316139 unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
> mirrormaker.from.site_02.to.site_ktv saka.test.ext_datastream 7
> unknown 9318770 unknown
> mirrormaker.from.site_02.to.site_ktv_kafka-4-1420497015616-725cd1b5-0
>
>
> Since I can push data between my sites something is working but what can be
> the case for NotCoordinatorForConsumerException on a single node cluster?
>
>
> Finally the __consumer_offsets topic should probably be masked out from
> mirroring without using whitelists or blacklists
>
>
> thanks
> /svante
>