You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Greenhorn Techie <gr...@gmail.com> on 2017/01/03 23:07:12 UTC
MirrorMaker - Topics Identification and Replication
Hi,
I am new to Kafka and as well as MirrorMaker. So wondering whether MM would
pick-up new topics that are created on the source cluster automatically,
provided topic matches the while list pattern?
For example, if I start MM as below, would it replicate any new topics that
are created after the MM process is launched?
kafka-mirror-maker.sh --new.consumer --consumer.config /opt/kafka/config/
consumer.properties --producer.config /opt/kafka/config/producer.properties
--whitelist ".*"
Or is there a need to restart MM process when a new topic / pattern is
started on the source cluster?
What should be the recommended approach in regards to this?
Thanks
Re: MirrorMaker - Topics Identification and Replication
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
That all sounds right!
Usually the delay for picking up the metadata update and starting to
replicate the topic won't be an issue since it's a one-time issue (around
topic creation) and the time window is pretty small for that. In steady
state, none of the mentioned delays would apply.
-Ewen
On Thu, Jan 5, 2017 at 3:46 AM, Greenhorn Techie <gr...@gmail.com>
wrote:
> Thanks Ewen for your response.
>
> Just to summarise, here is my understanding. Apologies if something is
> mis-understood. I am new to Kafka and hence still short in knowledge.
>
>
> - MirrorMarker process automatically picks-up new topics added on the
> source cluster and hence no restart of the process is needed at regular
> intervals to update the list of topics
> - For existing topics, MirrorMaker will replicate messages to the target
> kafka cluster as and when it sees data in the source kafka cluster's
> topics
> - However for new topics, there might be a delay of up to (default) 5
> min i.e. metadata refresh interval to start replicating the data to the
> target kafka cluster
>
> Please let me know if something is wrong in my understanding.
>
> Thanks
>
> On Tue, 3 Jan 2017 at 23:24 Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
> > Yes, the consumer will pick up the new topics when it refreshes metadata
> > (defaults to every 5 min) and start subscribing to the new topics.
> >
> > -Ewen
> >
> > On Tue, Jan 3, 2017 at 3:07 PM, Greenhorn Techie <
> > greenhorntechie@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I am new to Kafka and as well as MirrorMaker. So wondering whether MM
> > would
> > > pick-up new topics that are created on the source cluster
> automatically,
> > > provided topic matches the while list pattern?
> > >
> > > For example, if I start MM as below, would it replicate any new topics
> > that
> > > are created after the MM process is launched?
> > >
> > > kafka-mirror-maker.sh --new.consumer --consumer.config
> /opt/kafka/config/
> > > consumer.properties --producer.config /opt/kafka/config/producer.
> > > properties
> > > --whitelist ".*"
> > >
> > > Or is there a need to restart MM process when a new topic / pattern is
> > > started on the source cluster?
> > >
> > > What should be the recommended approach in regards to this?
> > >
> > > Thanks
> > >
> >
>
Re: MirrorMaker - Topics Identification and Replication
Posted by Greenhorn Techie <gr...@gmail.com>.
Thanks Ewen for your response.
Just to summarise, here is my understanding. Apologies if something is
mis-understood. I am new to Kafka and hence still short in knowledge.
- MirrorMarker process automatically picks-up new topics added on the
source cluster and hence no restart of the process is needed at regular
intervals to update the list of topics
- For existing topics, MirrorMaker will replicate messages to the target
kafka cluster as and when it sees data in the source kafka cluster's topics
- However for new topics, there might be a delay of up to (default) 5
min i.e. metadata refresh interval to start replicating the data to the
target kafka cluster
Please let me know if something is wrong in my understanding.
Thanks
On Tue, 3 Jan 2017 at 23:24 Ewen Cheslack-Postava <ew...@confluent.io> wrote:
> Yes, the consumer will pick up the new topics when it refreshes metadata
> (defaults to every 5 min) and start subscribing to the new topics.
>
> -Ewen
>
> On Tue, Jan 3, 2017 at 3:07 PM, Greenhorn Techie <
> greenhorntechie@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am new to Kafka and as well as MirrorMaker. So wondering whether MM
> would
> > pick-up new topics that are created on the source cluster automatically,
> > provided topic matches the while list pattern?
> >
> > For example, if I start MM as below, would it replicate any new topics
> that
> > are created after the MM process is launched?
> >
> > kafka-mirror-maker.sh --new.consumer --consumer.config /opt/kafka/config/
> > consumer.properties --producer.config /opt/kafka/config/producer.
> > properties
> > --whitelist ".*"
> >
> > Or is there a need to restart MM process when a new topic / pattern is
> > started on the source cluster?
> >
> > What should be the recommended approach in regards to this?
> >
> > Thanks
> >
>
Re: MirrorMaker - Topics Identification and Replication
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Yes, the consumer will pick up the new topics when it refreshes metadata
(defaults to every 5 min) and start subscribing to the new topics.
-Ewen
On Tue, Jan 3, 2017 at 3:07 PM, Greenhorn Techie <gr...@gmail.com>
wrote:
> Hi,
>
> I am new to Kafka and as well as MirrorMaker. So wondering whether MM would
> pick-up new topics that are created on the source cluster automatically,
> provided topic matches the while list pattern?
>
> For example, if I start MM as below, would it replicate any new topics that
> are created after the MM process is launched?
>
> kafka-mirror-maker.sh --new.consumer --consumer.config /opt/kafka/config/
> consumer.properties --producer.config /opt/kafka/config/producer.
> properties
> --whitelist ".*"
>
> Or is there a need to restart MM process when a new topic / pattern is
> started on the source cluster?
>
> What should be the recommended approach in regards to this?
>
> Thanks
>