You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Meghana Narasimhan <mn...@bandwidth.com> on 2015/12/10 23:32:03 UTC
Mirrormaker issue with Kafka 0.9 (confluent platform 2.0)
Hi,
We upgraded to the Confluent 2.0 platform today (we were earlier using the
Kafka 0.8 without the confluent platform). With the latest Kafka 0.9 that
is packaged with the platform we are facing issues starting the Mirror
maker with multiple consumer config files.
The original command that we were using with kafka 0.8 was,
bin/kafka-run-class kafka.tools.MirrorMaker --consumer.config
mirrorCluster1Consumer.properties --consumer.config
mirrorCluster2Consumer.properties --consumer.config
mirrorCluster3Consumer.properties --consumer.config
mirrorCluster4Consumer.properties --producer.config
mirrorProducer.properties --whitelist "topic1,topic2"
With the current Kafka 0.9 release, the tool throws the following error
ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$)
joptsimple.MultipleArgumentsForOptionException: Found multiple arguments
for option ['consumer.config'], but you asked for only one
at joptsimple.OptionSet.valueOf(OptionSet.java:177)
at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:225)
at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
Exception in thread "main" java.lang.NullPointerException
at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:276)
at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
So I tried setting the --num.streams=4 option in the command, and it still
throws the same error, next I tried to use the --new.consumer option but
the tool fails again with the same error.
Not sure how to get the Mirrormaker tool running with multiple consumer
config files.
Could not find much documentation regarding this except for the one in the
kafka documentation (
http://kafka.apache.org/documentation.html#basic_ops_mirror_maker).
ps : I was unable to post this in the confluent mailing list and hence
posting here.
Thanks,
Meghana
Re: Mirrormaker issue with Kafka 0.9 (confluent platform 2.0)
Posted by Meghana Narasimhan <mn...@bandwidth.com>.
Thanks Ewen and Guozhang for the details !
On Thu, Dec 10, 2015 at 10:31 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Meghana,
>
> As Ewen said in KAFKA-1650 we removed the support of multiple consumer
> configs in MM, so that you cannot get one MM instance to pull from multiple
> origin clusters, but instead you need to use one MM instance for each
> origin.
>
> The motivation for dropping this support is mainly operational: with the
> previous settings if one of the original cluster falls into a bad state, we
> cannot stop / debug that MM since it also pulls from other clusters. And
> other minor issues include metics reporting, etc. For example at LI people
> use one MM instance for each origin cluster, that are co-located with the
> destination cluster, so if you need to mirror from cluster A / B / C to D,
> you will have a A->D / B->D / C->D MM. Moving forward we are also trying to
> add exact mirroring features into MM, which would better be a 1-to-1
> mapping.
>
> Guozhang
>
>
> On Thu, Dec 10, 2015 at 3:21 PM, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
> > Meghana,
> >
> > It looks like this functionality was removed in
> > https://issues.apache.org/jira/browse/KAFKA-1650, although I don't see
> > explicit discussion of the issue in the JIRA so I'm not sure of the exact
> > motivation. Maybe Becket or Guozhang can offer some insight about if it
> is
> > necessary (that JIRA fixes a data loss issue) or an accidental
> > incompatibility.
> >
> > -Ewen
> >
> > On Thu, Dec 10, 2015 at 2:32 PM, Meghana Narasimhan <
> > mnarasimhan@bandwidth.com> wrote:
> >
> > > Hi,
> > >
> > > We upgraded to the Confluent 2.0 platform today (we were earlier using
> > the
> > > Kafka 0.8 without the confluent platform). With the latest Kafka 0.9
> that
> > > is packaged with the platform we are facing issues starting the Mirror
> > > maker with multiple consumer config files.
> > >
> > > The original command that we were using with kafka 0.8 was,
> > >
> > > bin/kafka-run-class kafka.tools.MirrorMaker --consumer.config
> > > mirrorCluster1Consumer.properties --consumer.config
> > > mirrorCluster2Consumer.properties --consumer.config
> > > mirrorCluster3Consumer.properties --consumer.config
> > > mirrorCluster4Consumer.properties --producer.config
> > > mirrorProducer.properties --whitelist "topic1,topic2"
> > >
> > > With the current Kafka 0.9 release, the tool throws the following error
> > >
> > > ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$)
> > > joptsimple.MultipleArgumentsForOptionException: Found multiple
> arguments
> > > for option ['consumer.config'], but you asked for only one
> > > at joptsimple.OptionSet.valueOf(OptionSet.java:177)
> > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:225)
> > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > > Exception in thread "main" java.lang.NullPointerException
> > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:276)
> > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > >
> > > So I tried setting the --num.streams=4 option in the command, and it
> > still
> > > throws the same error, next I tried to use the --new.consumer option
> but
> > > the tool fails again with the same error.
> > >
> > > Not sure how to get the Mirrormaker tool running with multiple consumer
> > > config files.
> > >
> > > Could not find much documentation regarding this except for the one in
> > the
> > > kafka documentation (
> > > http://kafka.apache.org/documentation.html#basic_ops_mirror_maker).
> > >
> > > ps : I was unable to post this in the confluent mailing list and hence
> > > posting here.
> > >
> > > Thanks,
> > > Meghana
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>
>
>
> --
> -- Guozhang
>
--
*Meghana Narasimhan*
*Software Development DevOps Engineer*
*o* 919.827.4318
*e *mnarasimhan@bandwidth.com
Re: Mirrormaker issue with Kafka 0.9 (confluent platform 2.0)
Posted by Ismael Juma <is...@juma.me.uk>.
On Fri, Dec 11, 2015 at 3:31 AM, Guozhang Wang <wa...@gmail.com> wrote:
> As Ewen said in KAFKA-1650 we removed the support of multiple consumer
> configs in MM, so that you cannot get one MM instance to pull from multiple
> origin clusters, but instead you need to use one MM instance for each
> origin.
>
Seems like we should mention this in the upgrade notes.
Best,
Ismael
Re: Mirrormaker issue with Kafka 0.9 (confluent platform 2.0)
Posted by Guozhang Wang <wa...@gmail.com>.
Meghana,
As Ewen said in KAFKA-1650 we removed the support of multiple consumer
configs in MM, so that you cannot get one MM instance to pull from multiple
origin clusters, but instead you need to use one MM instance for each
origin.
The motivation for dropping this support is mainly operational: with the
previous settings if one of the original cluster falls into a bad state, we
cannot stop / debug that MM since it also pulls from other clusters. And
other minor issues include metics reporting, etc. For example at LI people
use one MM instance for each origin cluster, that are co-located with the
destination cluster, so if you need to mirror from cluster A / B / C to D,
you will have a A->D / B->D / C->D MM. Moving forward we are also trying to
add exact mirroring features into MM, which would better be a 1-to-1
mapping.
Guozhang
On Thu, Dec 10, 2015 at 3:21 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> Meghana,
>
> It looks like this functionality was removed in
> https://issues.apache.org/jira/browse/KAFKA-1650, although I don't see
> explicit discussion of the issue in the JIRA so I'm not sure of the exact
> motivation. Maybe Becket or Guozhang can offer some insight about if it is
> necessary (that JIRA fixes a data loss issue) or an accidental
> incompatibility.
>
> -Ewen
>
> On Thu, Dec 10, 2015 at 2:32 PM, Meghana Narasimhan <
> mnarasimhan@bandwidth.com> wrote:
>
> > Hi,
> >
> > We upgraded to the Confluent 2.0 platform today (we were earlier using
> the
> > Kafka 0.8 without the confluent platform). With the latest Kafka 0.9 that
> > is packaged with the platform we are facing issues starting the Mirror
> > maker with multiple consumer config files.
> >
> > The original command that we were using with kafka 0.8 was,
> >
> > bin/kafka-run-class kafka.tools.MirrorMaker --consumer.config
> > mirrorCluster1Consumer.properties --consumer.config
> > mirrorCluster2Consumer.properties --consumer.config
> > mirrorCluster3Consumer.properties --consumer.config
> > mirrorCluster4Consumer.properties --producer.config
> > mirrorProducer.properties --whitelist "topic1,topic2"
> >
> > With the current Kafka 0.9 release, the tool throws the following error
> >
> > ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$)
> > joptsimple.MultipleArgumentsForOptionException: Found multiple arguments
> > for option ['consumer.config'], but you asked for only one
> > at joptsimple.OptionSet.valueOf(OptionSet.java:177)
> > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:225)
> > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > Exception in thread "main" java.lang.NullPointerException
> > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:276)
> > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> >
> > So I tried setting the --num.streams=4 option in the command, and it
> still
> > throws the same error, next I tried to use the --new.consumer option but
> > the tool fails again with the same error.
> >
> > Not sure how to get the Mirrormaker tool running with multiple consumer
> > config files.
> >
> > Could not find much documentation regarding this except for the one in
> the
> > kafka documentation (
> > http://kafka.apache.org/documentation.html#basic_ops_mirror_maker).
> >
> > ps : I was unable to post this in the confluent mailing list and hence
> > posting here.
> >
> > Thanks,
> > Meghana
> >
>
>
>
> --
> Thanks,
> Ewen
>
--
-- Guozhang
Re: Mirrormaker issue with Kafka 0.9 (confluent platform 2.0)
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Meghana,
It looks like this functionality was removed in
https://issues.apache.org/jira/browse/KAFKA-1650, although I don't see
explicit discussion of the issue in the JIRA so I'm not sure of the exact
motivation. Maybe Becket or Guozhang can offer some insight about if it is
necessary (that JIRA fixes a data loss issue) or an accidental
incompatibility.
-Ewen
On Thu, Dec 10, 2015 at 2:32 PM, Meghana Narasimhan <
mnarasimhan@bandwidth.com> wrote:
> Hi,
>
> We upgraded to the Confluent 2.0 platform today (we were earlier using the
> Kafka 0.8 without the confluent platform). With the latest Kafka 0.9 that
> is packaged with the platform we are facing issues starting the Mirror
> maker with multiple consumer config files.
>
> The original command that we were using with kafka 0.8 was,
>
> bin/kafka-run-class kafka.tools.MirrorMaker --consumer.config
> mirrorCluster1Consumer.properties --consumer.config
> mirrorCluster2Consumer.properties --consumer.config
> mirrorCluster3Consumer.properties --consumer.config
> mirrorCluster4Consumer.properties --producer.config
> mirrorProducer.properties --whitelist "topic1,topic2"
>
> With the current Kafka 0.9 release, the tool throws the following error
>
> ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$)
> joptsimple.MultipleArgumentsForOptionException: Found multiple arguments
> for option ['consumer.config'], but you asked for only one
> at joptsimple.OptionSet.valueOf(OptionSet.java:177)
> at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:225)
> at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> Exception in thread "main" java.lang.NullPointerException
> at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:276)
> at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
>
> So I tried setting the --num.streams=4 option in the command, and it still
> throws the same error, next I tried to use the --new.consumer option but
> the tool fails again with the same error.
>
> Not sure how to get the Mirrormaker tool running with multiple consumer
> config files.
>
> Could not find much documentation regarding this except for the one in the
> kafka documentation (
> http://kafka.apache.org/documentation.html#basic_ops_mirror_maker).
>
> ps : I was unable to post this in the confluent mailing list and hence
> posting here.
>
> Thanks,
> Meghana
>
--
Thanks,
Ewen