Clustered MirrorMaker 2 configuration update

[Péter Sinóros-Szabó <pe...@transferwise.com.INVALID>]

Hi,

I run two instances of MM2 with the command connect-mirror-maker.sh

Q1., Is there any requirement to cluster MM2? Like a network connection
between the nodes? How do MM2 coordinate the work between nodes?

Q2., Assuming I run two instances and want to update the configuration,
should it work if I rolling restart the nodes after each other? It seems
that after the restart both nodes still run with the old configuration. So
what should be the correct way of reconfiguring a MM2 cluster?

Cheers,
Peter

Re: Clustered MirrorMaker 2 configuration update

[Ryanne Dolan <ry...@gmail.com>]

That's right. The leader(s) will apply the config by writing it to the
mm2-config topic(s), which the followers will pick up. If the entire
cluster is bounced, the new config will have been applied.

Ryanne

On Fri, Jan 17, 2020, 4:28 AM Péter Sinóros-Szabó
<pe...@transferwise.com.invalid> wrote:

> Am I right that the configuration that is actually being used will come
> from the leader? Well, except the exception like the bootstrap list...
>
> Asking because I'd like to have a clear process to change the configuration
> in a way that at the end of the process I can be sure that all nodes use
> the new configuration.
>
> So assuming that I have a 2+ node MM2 cluster, all nodes has the same
> configuration and I'd like to change the configuration for the cluster, I
> plan to restart each node after each other with the new configuration. When
> I change configuration, I change it to the same on all nodes. Am I right
> that at the end I can be sure that the cluster uses the new configuration,
> because on the process I will bounce the leader for sure with the new
> config.
>
> I actually plan to run it on Kubernetes, so I am using a rolling restart on
> Kubernetes.
> I only have one mirroring path, but I guess this would work if I have more
> paths as well.
>
> Peter
>
> On Thu, 16 Jan 2020 at 16:36, Ryanne Dolan <ry...@gmail.com> wrote:
>
> > MM2 nodes only communicate via Kafka -- no connection is required between
> > them.
> >
> > To reconfigure, a rolling restart probably won't do what you expect,
> since
> > the configuration is always dictated by a single leader. Once the leader
> is
> > bounced, it will broadcast the new configuration via Kafka. If you
> bounce a
> > follower, it will still use the old configuration.
> >
> > It's slightly more complicated than that actually, since there are
> multiple
> > connectors in each cluster->cluster "herder". When there are multiple
> > clusters being replicated, there may be dozens of leaders. So a rolling
> > restart might be a good idea for larger deployments.
> >
> > And technically there are some properties that do affect followers, e.g.
> a
> > follower will read connection info like bootstrap.servers directly from
> > mm2.properties, not from the leader via Kafka. Obviously that would be a
> > chicken-egg problem otherwise! So a rolling restart would be prudent when
> > making such changes.
> >
> > So I guess, generally speaking, rolling restarts are a good idea -- just
> be
> > advised that it won't necessarily behave as you expect.
> >
> > Ryanne
> >
> > On Thu, Jan 16, 2020, 7:55 AM Péter Sinóros-Szabó
> > <pe...@transferwise.com.invalid> wrote:
> >
> > > Hi,
> > >
> > > I run two instances of MM2 with the command connect-mirror-maker.sh
> > >
> > > Q1., Is there any requirement to cluster MM2? Like a network connection
> > > between the nodes? How do MM2 coordinate the work between nodes?
> > >
> > > Q2., Assuming I run two instances and want to update the configuration,
> > > should it work if I rolling restart the nodes after each other? It
> seems
> > > that after the restart both nodes still run with the old configuration.
> > So
> > > what should be the correct way of reconfiguring a MM2 cluster?
> > >
> > > Cheers,
> > > Peter
> > >
> >
>

Re: Clustered MirrorMaker 2 configuration update

[Péter Sinóros-Szabó <pe...@transferwise.com.INVALID>]

Am I right that the configuration that is actually being used will come
from the leader? Well, except the exception like the bootstrap list...

Asking because I'd like to have a clear process to change the configuration
in a way that at the end of the process I can be sure that all nodes use
the new configuration.

So assuming that I have a 2+ node MM2 cluster, all nodes has the same
configuration and I'd like to change the configuration for the cluster, I
plan to restart each node after each other with the new configuration. When
I change configuration, I change it to the same on all nodes. Am I right
that at the end I can be sure that the cluster uses the new configuration,
because on the process I will bounce the leader for sure with the new
config.

I actually plan to run it on Kubernetes, so I am using a rolling restart on
Kubernetes.
I only have one mirroring path, but I guess this would work if I have more
paths as well.

Peter

On Thu, 16 Jan 2020 at 16:36, Ryanne Dolan <ry...@gmail.com> wrote:

> MM2 nodes only communicate via Kafka -- no connection is required between
> them.
>
> To reconfigure, a rolling restart probably won't do what you expect, since
> the configuration is always dictated by a single leader. Once the leader is
> bounced, it will broadcast the new configuration via Kafka. If you bounce a
> follower, it will still use the old configuration.
>
> It's slightly more complicated than that actually, since there are multiple
> connectors in each cluster->cluster "herder". When there are multiple
> clusters being replicated, there may be dozens of leaders. So a rolling
> restart might be a good idea for larger deployments.
>
> And technically there are some properties that do affect followers, e.g. a
> follower will read connection info like bootstrap.servers directly from
> mm2.properties, not from the leader via Kafka. Obviously that would be a
> chicken-egg problem otherwise! So a rolling restart would be prudent when
> making such changes.
>
> So I guess, generally speaking, rolling restarts are a good idea -- just be
> advised that it won't necessarily behave as you expect.
>
> Ryanne
>
> On Thu, Jan 16, 2020, 7:55 AM Péter Sinóros-Szabó
> <pe...@transferwise.com.invalid> wrote:
>
> > Hi,
> >
> > I run two instances of MM2 with the command connect-mirror-maker.sh
> >
> > Q1., Is there any requirement to cluster MM2? Like a network connection
> > between the nodes? How do MM2 coordinate the work between nodes?
> >
> > Q2., Assuming I run two instances and want to update the configuration,
> > should it work if I rolling restart the nodes after each other? It seems
> > that after the restart both nodes still run with the old configuration.
> So
> > what should be the correct way of reconfiguring a MM2 cluster?
> >
> > Cheers,
> > Peter
> >
>

Re: Clustered MirrorMaker 2 configuration update

[Ryanne Dolan <ry...@gmail.com>]

MM2 nodes only communicate via Kafka -- no connection is required between
them.

To reconfigure, a rolling restart probably won't do what you expect, since
the configuration is always dictated by a single leader. Once the leader is
bounced, it will broadcast the new configuration via Kafka. If you bounce a
follower, it will still use the old configuration.

It's slightly more complicated than that actually, since there are multiple
connectors in each cluster->cluster "herder". When there are multiple
clusters being replicated, there may be dozens of leaders. So a rolling
restart might be a good idea for larger deployments.

And technically there are some properties that do affect followers, e.g. a
follower will read connection info like bootstrap.servers directly from
mm2.properties, not from the leader via Kafka. Obviously that would be a
chicken-egg problem otherwise! So a rolling restart would be prudent when
making such changes.

So I guess, generally speaking, rolling restarts are a good idea -- just be
advised that it won't necessarily behave as you expect.

Ryanne

On Thu, Jan 16, 2020, 7:55 AM Péter Sinóros-Szabó
<pe...@transferwise.com.invalid> wrote:

> Hi,
>
> I run two instances of MM2 with the command connect-mirror-maker.sh
>
> Q1., Is there any requirement to cluster MM2? Like a network connection
> between the nodes? How do MM2 coordinate the work between nodes?
>
> Q2., Assuming I run two instances and want to update the configuration,
> should it work if I rolling restart the nodes after each other? It seems
> that after the restart both nodes still run with the old configuration. So
> what should be the correct way of reconfiguring a MM2 cluster?
>
> Cheers,
> Peter
>