Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

[Dániel Urbán <ur...@gmail.com>]

Hello everyone,

* Sorry, I meant KIP-710.

Right now the MirrorMaker cluster is somewhat unreliable, and not
supporting running in a cluster properly. I'd say that fixing this would be
a nice addition.
Does anyone have some input on this?

Thanks in advance
Daniel

Dániel Urbán <ur...@gmail.com> ezt írta (időpont: 2021. jan. 26., K,
15:56):

> Hello everyone,
>
> I would like to start a discussion on KIP-709, which addresses some
> missing features in MM2 dedicated mode.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters
>
> Currently, the dedicated mode of MM2 does not fully support running in a
> cluster. The core issue is that the Connect REST Server is not included in
> the dedicated mode, which makes follower->leader communication impossible.
> In some cases, this results in the cluster not being able to react to
> dynamic configuration changes (e.g. dynamic topic filter changes).
> Another smaller detail is that MM2 dedicated mode eagerly resolves config
> provider references in the Connector configurations, which is undesirable
> and a breaking change compared to vanilla Connect. This can cause an issue
> for example when there is an environment variable reference, which contains
> some host-specific information, like a file path. The leader resolves the
> reference eagerly, and the resolved value is propagated to other MM2 nodes
> instead of the reference being resolved locally, separately on each node.
>
> The KIP addresses these by adding the Connect REST Server to the MM2
> dedicated mode for each replication flow, and postponing the config
> provider reference resolution.
>
> Please discuss, I know this is a major change, but also an important
> feature for MM2 users.
>
> Daniel
>