You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Chris Peart <ch...@peart.me.uk> on 2022/10/19 10:25:08 UTC

Help with MM2 active/passive configuration


Hi All,

I'm new to MirrorMaker and have a production cluster running version 
2.8.1 and have a development cluster running the same version.

Our prod cluster has 4 brokers and our dev cluster has 3 brokers, both 
have a replication factor of 3.

I would like to setup an active/passive replication using MM2, we used 
to do this with Cloudera but have we have decommissioned Cloudera and 
would like to know how to configure topic replication using MM2.

I believe i need a mm2.properties file to achieve this. i do see what 
looks like a configuration in 
/opt/kafka_2.13-2.8.1/config/connect-mirror-maker.properties but i'm not 
sure if this is an active/passive configuration.

Ideally an example file would be ideal if possible?

Many Thanks,

Chris

Re: Help with MM2 active/passive configuration

Posted by Luke Chen <sh...@gmail.com>.

This is what Ryanne mentioned doc:
https://github.com/apache/kafka/tree/trunk/connect/mirror

Thanks.
Luke

On Mon, Oct 24, 2022 at 6:38 PM Chris Peart <ch...@peart.me.uk> wrote:

>
>
> Hi Ryanne,
>
> I cannot find what you mentioned, all i'm looking for are some example
> configurations for and active/passive setup.
>
> Many Thanks,
>
> Chris
>
> On 2022-10-19 18:32, Chris Peart wrote:
>
> > Thanks Ryan, can you provide a link to the readme please.
> > Many Thanks
> > Chris
> >
> > On 19 Oct 2022, at 6:15 pm, Ryanne Dolan <ry...@gmail.com> wrote:
> >
> > Hey Chris, check out the readme in connect/mirror for examples on how
> > to
> > run mirror maker as a standalone process. It's similar to how
> > Cloudera's
> > mirror maker is configured. If you've got a Connect cluster already, I
> > recommend using that and manually configuring the
> > MirrorSourceConnector.
> >
> > Ryanne
> >
> > On Wed, Oct 19, 2022, 5:26 AM Chris Peart <ch...@peart.me.uk> wrote:
> >
> > Hi All,
> >
> > I'm new to MirrorMaker and have a production cluster running version
> > 2.8.1 and have a development cluster running the same version.
> >
> > Our prod cluster has 4 brokers and our dev cluster has 3 brokers, both
> > have a replication factor of 3.
> >
> > I would like to setup an active/passive replication using MM2, we used
> > to do this with Cloudera but have we have decommissioned Cloudera and
> > would like to know how to configure topic replication using MM2.
> >
> > I believe i need a mm2.properties file to achieve this. i do see what
> > looks like a configuration in
> > /opt/kafka_2.13-2.8.1/config/connect-mirror-maker.properties but i'm
> > not
> > sure if this is an active/passive configuration.
> >
> > Ideally an example file would be ideal if possible?
> >
> > Many Thanks,
> >
> > Chris

Re: Help with MM2 active/passive configuration

Posted by Chris Peart <ch...@peart.me.uk>.


Hi Ryanne,

I cannot find what you mentioned, all i'm looking for are some example 
configurations for and active/passive setup.

Many Thanks,

Chris

On 2022-10-19 18:32, Chris Peart wrote:

> Thanks Ryan, can you provide a link to the readme please.
> Many Thanks
> Chris
> 
> On 19 Oct 2022, at 6:15 pm, Ryanne Dolan <ry...@gmail.com> wrote:
> 
> Hey Chris, check out the readme in connect/mirror for examples on how 
> to
> run mirror maker as a standalone process. It's similar to how 
> Cloudera's
> mirror maker is configured. If you've got a Connect cluster already, I
> recommend using that and manually configuring the 
> MirrorSourceConnector.
> 
> Ryanne
> 
> On Wed, Oct 19, 2022, 5:26 AM Chris Peart <ch...@peart.me.uk> wrote:
> 
> Hi All,
> 
> I'm new to MirrorMaker and have a production cluster running version
> 2.8.1 and have a development cluster running the same version.
> 
> Our prod cluster has 4 brokers and our dev cluster has 3 brokers, both
> have a replication factor of 3.
> 
> I would like to setup an active/passive replication using MM2, we used
> to do this with Cloudera but have we have decommissioned Cloudera and
> would like to know how to configure topic replication using MM2.
> 
> I believe i need a mm2.properties file to achieve this. i do see what
> looks like a configuration in
> /opt/kafka_2.13-2.8.1/config/connect-mirror-maker.properties but i'm 
> not
> sure if this is an active/passive configuration.
> 
> Ideally an example file would be ideal if possible?
> 
> Many Thanks,
> 
> Chris

Re: Help with MM2 active/passive configuration

Posted by Chris Peart <ch...@peart.me.uk>.

Thanks Ryan, can you provide a link to the readme please. 
Many Thanks 
Chris

> On 19 Oct 2022, at 6:15 pm, Ryanne Dolan <ry...@gmail.com> wrote:
> 
> Hey Chris, check out the readme in connect/mirror for examples on how to
> run mirror maker as a standalone process. It's similar to how Cloudera's
> mirror maker is configured. If you've got a Connect cluster already, I
> recommend using that and manually configuring the MirrorSourceConnector.
> 
> Ryanne
> 
>> On Wed, Oct 19, 2022, 5:26 AM Chris Peart <ch...@peart.me.uk> wrote:
>> 
>> 
>> 
>> Hi All,
>> 
>> I'm new to MirrorMaker and have a production cluster running version
>> 2.8.1 and have a development cluster running the same version.
>> 
>> Our prod cluster has 4 brokers and our dev cluster has 3 brokers, both
>> have a replication factor of 3.
>> 
>> I would like to setup an active/passive replication using MM2, we used
>> to do this with Cloudera but have we have decommissioned Cloudera and
>> would like to know how to configure topic replication using MM2.
>> 
>> I believe i need a mm2.properties file to achieve this. i do see what
>> looks like a configuration in
>> /opt/kafka_2.13-2.8.1/config/connect-mirror-maker.properties but i'm not
>> sure if this is an active/passive configuration.
>> 
>> Ideally an example file would be ideal if possible?
>> 
>> Many Thanks,
>> 
>> Chris

Re: Help with MM2 active/passive configuration

Posted by Ryanne Dolan <ry...@gmail.com>.

Hey Chris, check out the readme in connect/mirror for examples on how to
run mirror maker as a standalone process. It's similar to how Cloudera's
mirror maker is configured. If you've got a Connect cluster already, I
recommend using that and manually configuring the MirrorSourceConnector.

Ryanne

On Wed, Oct 19, 2022, 5:26 AM Chris Peart <ch...@peart.me.uk> wrote:

>
>
> Hi All,
>
> I'm new to MirrorMaker and have a production cluster running version
> 2.8.1 and have a development cluster running the same version.
>
> Our prod cluster has 4 brokers and our dev cluster has 3 brokers, both
> have a replication factor of 3.
>
> I would like to setup an active/passive replication using MM2, we used
> to do this with Cloudera but have we have decommissioned Cloudera and
> would like to know how to configure topic replication using MM2.
>
> I believe i need a mm2.properties file to achieve this. i do see what
> looks like a configuration in
> /opt/kafka_2.13-2.8.1/config/connect-mirror-maker.properties but i'm not
> sure if this is an active/passive configuration.
>
> Ideally an example file would be ideal if possible?
>
> Many Thanks,
>
> Chris