Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

[Mickael Maison <mi...@gmail.com>]

Hi Daniel,

Thanks for the KIP.

It would be nice to expose more of the REST API as some endpoints are
really useful, for example /admin or
/connectors/<connector>/tasks-config. However as dedicated mode is
currently unusable, I think the approach of "just fixing it" by
exposing the internal endpoints is fine. It also does not seem to
corner us too much if we decide to make further changes in the future.

One suggestion I have is to avoid using "mm" in the configuration
name. Could we rename mm.enable.internal.rest to
dedicated.mode.enable.internal.rest or something like that?

Thanks,
Mickael

On Tue, Sep 27, 2022 at 3:56 PM Chris Egerton <ch...@aiven.io.invalid> wrote:
>
> Thanks Daniel! No further comments from me, looks good.
>
> On Tue, Sep 27, 2022 at 4:39 AM Dániel Urbán <ur...@gmail.com> wrote:
>
> > Hi Chris,
> >
> > I understand your points, and I agree that this path is safer in terms of
> > future plans, I accept it.
> > Updated the KIP accordingly.
> >
> > Thanks,
> > Daniel
> >
> > Chris Egerton <ch...@aiven.io.invalid> ezt írta (időpont: 2022. szept.
> > 21., Sze, 18:54):
> >
> > > Hi Daniel,
> > >
> > > I'm a little hesitant to add support for REST extensions and the admin
> > API
> > > to dedicated MM2 nodes because they may restrict our options in the
> > future
> > > if/when we add a public-facing REST API.
> > >
> > > Regarding REST extensions specifically, I understand their security value
> > > for public-facing APIs, but for the internal API--which should only ever
> > be
> > > targeted by MM2 nodes, and never by humans, UIs, CLIs, etc.--I'm not sure
> > > there's enough need there to add that API this time around. The internal
> > > endpoints used by Kafka Connect should be secured by the session key
> > > mechanism introduced in KIP-507 [1], and IIUC, with this KIP, users will
> > > also be able to configure their cluster to use mTLS.
> > >
> > > Regarding the admin API, I consider it part of the public-facing REST API
> > > for Kafka Connect, so I was assuming we wouldn't want to add it to this
> > KIP
> > > since we're intentionally slimming down the REST API to just the bare
> > > essentials (i.e., just the internal endpoints). I can see value for it,
> > but
> > > it's similar to the status endpoints in the Kafka Connect REST API; we
> > > might choose to expose this sometime down the line, but IMO it'd be
> > better
> > > to do that in a separate KIP so that we can iron out the details of
> > exactly
> > > what kind of REST API would best suit dedicated MM2 clusters, and how
> > that
> > > would differ from the API provided by vanilla Kafka Connect.
> > >
> > > Let me know what you think!
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > [1] -
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> > >
> > > On Wed, Sep 21, 2022 at 4:59 AM Dániel Urbán <ur...@gmail.com>
> > > wrote:
> > >
> > > > Hi Chris,
> > > >
> > > > About the worker id: makes sense. I changed the wording, but kept it
> > > listed
> > > > as it is a change compared to existing MM2 code. Updated the KIP
> > > > accordingly.
> > > >
> > > > About the REST server configurations: again, I agree, it should be
> > > > generalized. But I'm not sure about those exceptions you listed, as all
> > > of
> > > > them make sense in MM2 as well. For example, security related rest
> > > > extensions could work with MM2 as well. Admin listeners are also
> > useful,
> > > as
> > > > they would allow the same admin operations for MM2 nodes. Am I mistaken
> > > > here? Updated the KIP without mentioning exceptions for now.
> > > >
> > > > I think yes, the lazy config resolution should be unconditional. Even
> > if
> > > we
> > > > don't consider the distributed mode of MM2, the eager resolution does
> > not
> > > > allow using sensitive configs in the mm2 properties, as they will be
> > > > written by value into the config topic. I didn't really consider this
> > as
> > > > breaking change, but I might be wrong.
> > > >
> > > > Enable flag property name: also makes sense, updated the KIP.
> > > >
> > > > Thanks a lot!
> > > > Daniel
> > > >
> > > > Chris Egerton <ch...@aiven.io.invalid> ezt írta (időpont: 2022.
> > szept.
> > > > 20., K, 20:29):
> > > >
> > > > > Hi Daniel,
> > > > >
> > > > > Looking pretty good! Regarding the worker ID generation--apologies, I
> > > was
> > > > > unclear with my question. I was wondering if we had to adopt any
> > > special
> > > > > logic at all for MM2, or if we could use the same logic that vanilla
> > > > Kafka
> > > > > Connect does in distributed mode, where the worker ID is its
> > advertised
> > > > URL
> > > > > (e.g., "connect:8083" or "localhost:25565"). Unless I'm mistaken,
> > this
> > > > > should allow MM2 nodes to be identified unambiguously. Do you think
> > it
> > > > > makes sense to follow this strategy?
> > > > >
> > > > > Now that the details on the new REST interface have been fleshed out,
> > > I'm
> > > > > also wondering if we want to add support for the "
> > > > > rest.advertised.host.name",
> > > > > "rest.advertised.port", and "rest.advertised.listener" properties,
> > > which
> > > > > are vital for being able to run Kafka Connect in distributed mode
> > from
> > > > > within containers. In fact, I'm wondering if we can generalize some
> > of
> > > > the
> > > > > specification in the KIP around which REST properties are accepted by
> > > > > stating that all REST-related properties that are available with
> > > vanilla
> > > > > Kafka Connect will be supported for dedicated MM2 nodes when
> > > > > "mm.enable.rest" is set to "true", except for ones related to the
> > > > > public-facing REST API like "rest.extension.classes",
> > > "admin.listeners",
> > > > > and "admin.listeners.https.*".
> > > > >
> > > > > One other thought--is the lazy evaluation of config provider
> > references
> > > > > going to take place unconditionally, or only when the internal REST
> > API
> > > > is
> > > > > enabled on a worker?
> > > > >
> > > > > Finally, do you think we could change "mm.enable.rest" to
> > > > > "mm.enable.internal.rest"? That way, if we want to introduce a
> > > > > public-facing REST API later on, we can use "mm.enable.rest" as the
> > > name
> > > > > for that property.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Fri, Sep 16, 2022 at 9:28 AM Dániel Urbán <ur...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Chris,
> > > > > >
> > > > > > I went through the KIP and updated it based on our discussion. I
> > > think
> > > > > your
> > > > > > suggestions simplified (and shortened) the KIP significantly.
> > > > > >
> > > > > > Thanks,
> > > > > > Daniel
> > > > > >
> > > > > > Dániel Urbán <du...@cloudera.com.invalid> ezt írta (időpont:
> > 2022.
> > > > > szept.
> > > > > > 16., P, 15:15):
> > > > > >
> > > > > > > Hi Chris,
> > > > > > >
> > > > > > > 1. For the REST-server-per-flow setup, it made sense to introduce
> > > > some
> > > > > > > simplified configuration. With a single REST server, it doesn't
> > > make
> > > > > > sense
> > > > > > > anymore, I'm removing it from the KIP.
> > > > > > > 2. I think that changing the worker ID generation still makes
> > > sense,
> > > > > > > otherwise there is no way to differentiate between the MM2
> > > instances.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Daniel
> > > > > > >
> > > > > > > On Wed, Aug 31, 2022 at 8:39 PM Chris Egerton
> > > > <chrise@aiven.io.invalid
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Daniel,
> > > > > > > >
> > > > > > > > I've taken a look at the KIP in detail. Here are my complete
> > > > thoughts
> > > > > > > > (minus the aforementioned sections that may be affected by
> > > changes
> > > > to
> > > > > > an
> > > > > > > > internal-only REST API):
> > > > > > > >
> > > > > > > > 1. Why introduce new mm.host.name and mm.rest.protocol
> > > properties
> > > > > > > instead
> > > > > > > > of using the properties that are already used by Kafka Connect:
> > > > > > > listeners,
> > > > > > > > rest.advertised.host.name, rest.advertised.host.port, and
> > > > > > > > rest.advertised.listener? We used to have the rest.host.name
> > and
> > > > > > > rest.port
> > > > > > > > properties in Connect but deprecated and eventually removed
> > them
> > > in
> > > > > > favor
> > > > > > > > of the listeners property in KIP-208 [1]; I'm hoping we can
> > keep
> > > > > things
> > > > > > > as
> > > > > > > > similar as possible between MM2 and Connect in order to make it
> > > > > easier
> > > > > > > for
> > > > > > > > users to work with both. I'm also hoping that we can allow
> > users
> > > to
> > > > > > > > configure the port that their MM2 nodes listen on instead of
> > > > > hardcoding
> > > > > > > MM2
> > > > > > > > to bind to port 0.
> > > > > > > >
> > > > > > > > 2. Do we still need to change the worker IDs that get used in
> > the
> > > > > > status
> > > > > > > > topic?
> > > > > > > >
> > > > > > > > Everything else looks good, or should change once the KIP is
> > > > updated
> > > > > > with
> > > > > > > > the internal-only REST API alternative.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > [1] -
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface
> > > > > > > >
> > > > > > > > On Mon, Aug 29, 2022 at 1:55 PM Chris Egerton <chrise@aiven.io
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Daniel,
> > > > > > > > >
> > > > > > > > > Yeah, I think that's the way to go. Adding multiple servers
> > for
> > > > > each
> > > > > > > > > herder seems like it'd be too much of a pain for users to
> > > > > configure,
> > > > > > > and
> > > > > > > > if
> > > > > > > > > we keep the API strictly internal for now, we shouldn't be
> > > > painting
> > > > > > > > > ourselves into too much of a corner if/when we decide to
> > > expose a
> > > > > > > > > public-facing REST API for dedicated MM2 clusters.
> > > > > > > > >
> > > > > > > > > I plan to take a look at the rest of the KIP and provide a
> > > > complete
> > > > > > > > review
> > > > > > > > > sometime this week; I'll hold off on commenting on anything
> > > that
> > > > > > seems
> > > > > > > > like
> > > > > > > > > it'll be affected by switching to an internal-only REST API
> > > until
> > > > > > those
> > > > > > > > > changes are published, but should be able to review
> > everything
> > > > > else.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Mon, Aug 29, 2022 at 6:57 AM Dániel Urbán <
> > > > > urb.daniel7@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi Chris,
> > > > > > > > >>
> > > > > > > > >> I understand your point, sounds good to me.
> > > > > > > > >> So in short, we should opt for an internal-only API, and
> > > > > preferably
> > > > > > a
> > > > > > > > >> single server solution. Is that right?
> > > > > > > > >>
> > > > > > > > >> Thanks
> > > > > > > > >> Daniel
> > > > > > > > >>
> > > > > > > > >> Chris Egerton <ch...@aiven.io.invalid> ezt írta (időpont:
> > > > 2022.
> > > > > > aug.
> > > > > > > > >> 26.,
> > > > > > > > >> P, 17:36):
> > > > > > > > >>
> > > > > > > > >> > Hi Daniel,
> > > > > > > > >> >
> > > > > > > > >> > Glad to hear from you!
> > > > > > > > >> >
> > > > > > > > >> > With regards to the stripped-down REST API alternative, I
> > > > don't
> > > > > > see
> > > > > > > > how
> > > > > > > > >> > this would prevent us from introducing the fully-fledged
> > > > Connect
> > > > > > > REST
> > > > > > > > >> API,
> > > > > > > > >> > or even an augmented variant of it, at some point down the
> > > > road.
> > > > > > If
> > > > > > > we
> > > > > > > > >> go
> > > > > > > > >> > with the internal-only API now, and want to expand later,
> > > > can't
> > > > > we
> > > > > > > > gate
> > > > > > > > >> the
> > > > > > > > >> > expansion behind a feature flag configuration property
> > that
> > > by
> > > > > > > default
> > > > > > > > >> > disables the new feature?
> > > > > > > > >> >
> > > > > > > > >> > I'm also not sure that we'd ever want to expose the raw
> > > > Connect
> > > > > > REST
> > > > > > > > API
> > > > > > > > >> > for dedicated MM2 clusters. If people want that
> > capability,
> > > > they
> > > > > > can
> > > > > > > > >> > already spin up a vanilla Connect cluster and run as many
> > > MM2
> > > > > > > > >> connectors as
> > > > > > > > >> > they'd like on it, and as of KIP-458 [1], it's even
> > possible
> > > > to
> > > > > > use
> > > > > > > a
> > > > > > > > >> > single Connect cluster to replicate between any two Kafka
> > > > > clusters
> > > > > > > > >> instead
> > > > > > > > >> > of only targeting the Kafka cluster that the vanilla
> > Connect
> > > > > > cluster
> > > > > > > > >> > operates on top of. I do agree that it'd be great to be
> > able
> > > > to
> > > > > > > > >> dynamically
> > > > > > > > >> > adjust things like topic filters without having to
> > restart a
> > > > > > > dedicated
> > > > > > > > >> MM2
> > > > > > > > >> > node; I'm just not sure that the vanilla Connect REST API
> > is
> > > > the
> > > > > > > > >> > appropriate way to do that, especially since the exact
> > > > > mechanisms
> > > > > > > that
> > > > > > > > >> make
> > > > > > > > >> > a single Connect cluster viable for replicating across any
> > > two
> > > > > > Kafka
> > > > > > > > >> > clusters could be abused and cause a dedicated MM2 cluster
> > > to
> > > > > > start
> > > > > > > > >> writing
> > > > > > > > >> > to a completely different Kafka cluster that's not even
> > > > defined
> > > > > in
> > > > > > > its
> > > > > > > > >> > config file.
> > > > > > > > >> >
> > > > > > > > >> > Finally, as far as security goes--since this is
> > essentially
> > > a
> > > > > bug
> > > > > > > fix,
> > > > > > > > >> I'm
> > > > > > > > >> > inclined to make it as easy as possible for users to adopt
> > > it.
> > > > > > MTLS
> > > > > > > > is a
> > > > > > > > >> > great start for securing a REST API, but it's not
> > sufficient
> > > > on
> > > > > > its
> > > > > > > > own
> > > > > > > > >> > since anyone who could issue an authenticated REST request
> > > > > against
> > > > > > > the
> > > > > > > > >> MM2
> > > > > > > > >> > cluster would still be able to make any changes they want
> > > > (with
> > > > > > the
> > > > > > > > >> > exception of accessing internal endpoints, which were
> > > secured
> > > > > with
> > > > > > > > >> > KIP-507). If we were to bring up the fully-fledged Connect
> > > > REST
> > > > > > API,
> > > > > > > > >> > cluster administrators would also likely have to add some
> > > kind
> > > > > of
> > > > > > > > >> > authorization layer to prevent people from using the REST
> > > API
> > > > to
> > > > > > > mess
> > > > > > > > >> with
> > > > > > > > >> > the configurations of the connectors that MM2 brought up.
> > > One
> > > > > way
> > > > > > of
> > > > > > > > >> doing
> > > > > > > > >> > that is to add a REST extension to your Connect cluster,
> > but
> > > > > > > > >> implementing
> > > > > > > > >> > and configuring one in order to be able to run a
> > multi-node
> > > > MM2
> > > > > > > > cluster
> > > > > > > > >> > without hitting this bug seems like too much work to be
> > > worth
> > > > > it.
> > > > > > > > >> >
> > > > > > > > >> > I think if we had a better picture of what a REST API for
> > > > > > dedicated
> > > > > > > > MM2
> > > > > > > > >> > clusters would/should look like, then it would be easier
> > to
> > > go
> > > > > > along
> > > > > > > > >> with
> > > > > > > > >> > this, and we could even just add the feature flag in this
> > > KIP
> > > > > > right
> > > > > > > > now
> > > > > > > > >> to
> > > > > > > > >> > address any security concerns. My instinct would be to
> > > address
> > > > > > this
> > > > > > > > in a
> > > > > > > > >> > follow-up KIP in order to reduce scope creep, though, and
> > > keep
> > > > > > this
> > > > > > > > KIP
> > > > > > > > >> > focused on addressing the bug with multi-node dedicated
> > MM2
> > > > > > > clusters.
> > > > > > > > >> What
> > > > > > > > >> > do you think?
> > > > > > > > >> >
> > > > > > > > >> > Cheers,
> > > > > > > > >> >
> > > > > > > > >> > Chris
> > > > > > > > >> >
> > > > > > > > >> > [1] -
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy
> > > > > > > > >> >
> > > > > > > > >> > On Thu, Aug 25, 2022 at 3:55 AM Dániel Urbán <
> > > > > > urb.daniel7@gmail.com
> > > > > > > >
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Hi Chris,
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks for bringing this up again :)
> > > > > > > > >> > >
> > > > > > > > >> > > 1. I think that is reasonable, though I find the current
> > > > state
> > > > > > of
> > > > > > > > MM2
> > > > > > > > >> to
> > > > > > > > >> > be
> > > > > > > > >> > > confusing. The current issue with the distributed mode
> > is
> > > > not
> > > > > > > > >> documented
> > > > > > > > >> > > properly, but maybe the logging will help a bit.
> > > > > > > > >> > >
> > > > > > > > >> > > 2. Going for an internal-only Connect REST version would
> > > > lock
> > > > > > MM2
> > > > > > > > out
> > > > > > > > >> of
> > > > > > > > >> > a
> > > > > > > > >> > > path where the REST API can be used to dynamically
> > > > reconfigure
> > > > > > > > >> > > replications. For now, I agree, it would be easy to
> > > corrupt
> > > > > the
> > > > > > > > state
> > > > > > > > >> of
> > > > > > > > >> > > MM2 if someone wanted to use the properties and the REST
> > > at
> > > > > the
> > > > > > > same
> > > > > > > > >> > time,
> > > > > > > > >> > > but in the future, we might have a chance to introduce a
> > > > > > different
> > > > > > > > >> config
> > > > > > > > >> > > mechanism, where only the cluster connections have to be
> > > > > > specified
> > > > > > > > in
> > > > > > > > >> the
> > > > > > > > >> > > properties file, and everything else can be configured
> > > > through
> > > > > > > REST
> > > > > > > > >> > > (enabling replications, changing topic filters, etc.).
> > > > Because
> > > > > > of
> > > > > > > > >> this,
> > > > > > > > >> > I'm
> > > > > > > > >> > > leaning towards a full Connect REST API. To avoid issues
> > > > with
> > > > > > > > >> conflicts
> > > > > > > > >> > > between the props file and the REST, we could document
> > > > > security
> > > > > > > best
> > > > > > > > >> > > practices (e.g. turn on basic auth or mTLS on the
> > Connect
> > > > REST
> > > > > > to
> > > > > > > > >> avoid
> > > > > > > > >> > > unwanted interactions).
> > > > > > > > >> > >
> > > > > > > > >> > > 3. That is a good point, and I agree, a big plus for
> > > > > motivation.
> > > > > > > > >> > >
> > > > > > > > >> > > I have a working version of this in which all flows spin
> > > up
> > > > a
> > > > > > > > >> dedicated
> > > > > > > > >> > > Connect REST, but I can give other solutions a try, too.
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks,
> > > > > > > > >> > > Daniel
> > > > > > > > >> > >
> > > > > > > > >> > > Chris Egerton <ch...@aiven.io.invalid> ezt írta
> > > (időpont:
> > > > > > 2022.
> > > > > > > > aug.
> > > > > > > > >> > 24.,
> > > > > > > > >> > > Sze, 17:46):
> > > > > > > > >> > >
> > > > > > > > >> > > > Hi Daniel,
> > > > > > > > >> > > >
> > > > > > > > >> > > > I'd like to resurface this KIP in case you're still
> > > > > interested
> > > > > > > in
> > > > > > > > >> > > pursuing
> > > > > > > > >> > > > it. I know it's been a while since you published it,
> > and
> > > > it
> > > > > > > hasn't
> > > > > > > > >> > > received
> > > > > > > > >> > > > much attention, but I'm hoping we can give it a try
> > now
> > > > and
> > > > > > > > finally
> > > > > > > > >> put
> > > > > > > > >> > > > this long-standing bug to rest. To that end, I have
> > some
> > > > > > > thoughts
> > > > > > > > >> about
> > > > > > > > >> > > the
> > > > > > > > >> > > > proposal. This isn't a complete review, but I wanted
> > to
> > > > give
> > > > > > > > enough
> > > > > > > > >> to
> > > > > > > > >> > > get
> > > > > > > > >> > > > the ball rolling:
> > > > > > > > >> > > >
> > > > > > > > >> > > > 1. Some environments with firewalls or strict security
> > > > > > policies
> > > > > > > > may
> > > > > > > > >> not
> > > > > > > > >> > > be
> > > > > > > > >> > > > able to bring up a REST server for each MM2 node. If
> > we
> > > > > decide
> > > > > > > > that
> > > > > > > > >> > we'd
> > > > > > > > >> > > > like to use the Connect REST API (or even just parts
> > of
> > > > it)
> > > > > to
> > > > > > > > >> address
> > > > > > > > >> > > this
> > > > > > > > >> > > > bug with MM2, it does make sense to eventually make
> > the
> > > > > > > > >> availability of
> > > > > > > > >> > > the
> > > > > > > > >> > > > REST API a hard requirement for running MM2, but it
> > > might
> > > > > be a
> > > > > > > bit
> > > > > > > > >> too
> > > > > > > > >> > > > abrupt to do that all in a single release. What do you
> > > > think
> > > > > > > about
> > > > > > > > >> > making
> > > > > > > > >> > > > the REST API optional for now, but noting that it will
> > > > > become
> > > > > > > > >> required
> > > > > > > > >> > > in a
> > > > > > > > >> > > > later release (probably 4.0.0 or, if that's not enough
> > > > time,
> > > > > > > > >> 5.0.0)? We
> > > > > > > > >> > > > could choose not to bring the REST server for any node
> > > > whose
> > > > > > > > >> > > configuration
> > > > > > > > >> > > > doesn't explicitly opt into one, and maybe log a
> > warning
> > > > > > message
> > > > > > > > on
> > > > > > > > >> > > startup
> > > > > > > > >> > > > if none is configured. In effect, we'd be marking the
> > > > > current
> > > > > > > mode
> > > > > > > > >> (no
> > > > > > > > >> > > REST
> > > > > > > > >> > > > server) as deprecated.
> > > > > > > > >> > > >
> > > > > > > > >> > > > 2. I'm not sure that we should count out the "Creating
> > > an
> > > > > > > > >> internal-only
> > > > > > > > >> > > > derivation of the Connect REST API" rejected
> > > alternative.
> > > > > > Right
> > > > > > > > now,
> > > > > > > > >> > the
> > > > > > > > >> > > > single source of truth for the configuration of a MM2
> > > > > cluster
> > > > > > > > >> (assuming
> > > > > > > > >> > > > it's being run in dedicated mode, and not as a
> > connector
> > > > in
> > > > > a
> > > > > > > > >> vanilla
> > > > > > > > >> > > > Connect cluster) is the configuration file used for
> > the
> > > > > > process.
> > > > > > > > By
> > > > > > > > >> > > > bringing up the REST API, we'd expose endpoints to
> > > modify
> > > > > > > > connector
> > > > > > > > >> > > > configurations, which would not only add complexity to
> > > the
> > > > > > > > operation
> > > > > > > > >> > of a
> > > > > > > > >> > > > MM2 cluster, but even qualify as an attack vector for
> > > > > > malicious
> > > > > > > > >> > entities.
> > > > > > > > >> > > > Thanks to KIP-507 we have some amount of security
> > around
> > > > the
> > > > > > > > >> > > internal-only
> > > > > > > > >> > > > endpoints used by the Connect framework, but for any
> > > > public
> > > > > > > > >> endpoints,
> > > > > > > > >> > > the
> > > > > > > > >> > > > Connect REST API doesn't come with any security out of
> > > the
> > > > > > box.
> > > > > > > > >> > > >
> > > > > > > > >> > > > 3. Small point, but with support for exactly-once
> > source
> > > > > > > > connectors
> > > > > > > > >> > > coming
> > > > > > > > >> > > > out in 3.3.0, it's also worth noting that that's
> > another
> > > > > > feature
> > > > > > > > >> that
> > > > > > > > >> > > won't
> > > > > > > > >> > > > work properly with multi-node MM2 clusters without
> > > adding
> > > > a
> > > > > > REST
> > > > > > > > >> server
> > > > > > > > >> > > for
> > > > > > > > >> > > > each node (or some substitute that accomplishes the
> > same
> > > > > > goal).
> > > > > > > I
> > > > > > > > >> don't
> > > > > > > > >> > > > think this will affect the direction of the design
> > > > > discussion
> > > > > > > too
> > > > > > > > >> much,
> > > > > > > > >> > > but
> > > > > > > > >> > > > it does help strengthen the motivation.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Cheers,
> > > > > > > > >> > > >
> > > > > > > > >> > > > Chris
> > > > > > > > >> > > >
> > > > > > > > >> > > > On 2021/02/18 15:57:36 Dániel Urbán wrote:
> > > > > > > > >> > > > > Hello everyone,
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > * Sorry, I meant KIP-710.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Right now the MirrorMaker cluster is somewhat
> > > > unreliable,
> > > > > > and
> > > > > > > > not
> > > > > > > > >> > > > > supporting running in a cluster properly. I'd say
> > that
> > > > > > fixing
> > > > > > > > this
> > > > > > > > >> > > would
> > > > > > > > >> > > > be
> > > > > > > > >> > > > > a nice addition.
> > > > > > > > >> > > > > Does anyone have some input on this?
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Thanks in advance
> > > > > > > > >> > > > > Daniel
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Dániel Urbán <ur...@gmail.com> ezt írta (időpont:
> > > 2021.
> > > > > > jan.
> > > > > > > > >> 26., K,
> > > > > > > > >> > > > > 15:56):
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > Hello everyone,
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > I would like to start a discussion on KIP-709,
> > which
> > > > > > > addresses
> > > > > > > > >> some
> > > > > > > > >> > > > > > missing features in MM2 dedicated mode.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Currently, the dedicated mode of MM2 does not
> > fully
> > > > > > support
> > > > > > > > >> running
> > > > > > > > >> > > in
> > > > > > > > >> > > > a
> > > > > > > > >> > > > > > cluster. The core issue is that the Connect REST
> > > > Server
> > > > > is
> > > > > > > not
> > > > > > > > >> > > included
> > > > > > > > >> > > > in
> > > > > > > > >> > > > > > the dedicated mode, which makes follower->leader
> > > > > > > communication
> > > > > > > > >> > > > impossible.
> > > > > > > > >> > > > > > In some cases, this results in the cluster not
> > being
> > > > > able
> > > > > > to
> > > > > > > > >> react
> > > > > > > > >> > to
> > > > > > > > >> > > > > > dynamic configuration changes (e.g. dynamic topic
> > > > filter
> > > > > > > > >> changes).
> > > > > > > > >> > > > > > Another smaller detail is that MM2 dedicated mode
> > > > > eagerly
> > > > > > > > >> resolves
> > > > > > > > >> > > > config
> > > > > > > > >> > > > > > provider references in the Connector
> > configurations,
> > > > > which
> > > > > > > is
> > > > > > > > >> > > > undesirable
> > > > > > > > >> > > > > > and a breaking change compared to vanilla Connect.
> > > > This
> > > > > > can
> > > > > > > > >> cause
> > > > > > > > >> > an
> > > > > > > > >> > > > issue
> > > > > > > > >> > > > > > for example when there is an environment variable
> > > > > > reference,
> > > > > > > > >> which
> > > > > > > > >> > > > contains
> > > > > > > > >> > > > > > some host-specific information, like a file path.
> > > The
> > > > > > leader
> > > > > > > > >> > resolves
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > reference eagerly, and the resolved value is
> > > > propagated
> > > > > to
> > > > > > > > other
> > > > > > > > >> > MM2
> > > > > > > > >> > > > nodes
> > > > > > > > >> > > > > > instead of the reference being resolved locally,
> > > > > > separately
> > > > > > > on
> > > > > > > > >> each
> > > > > > > > >> > > > node.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > The KIP addresses these by adding the Connect REST
> > > > > Server
> > > > > > to
> > > > > > > > the
> > > > > > > > >> > MM2
> > > > > > > > >> > > > > > dedicated mode for each replication flow, and
> > > > postponing
> > > > > > the
> > > > > > > > >> config
> > > > > > > > >> > > > > > provider reference resolution.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Please discuss, I know this is a major change, but
> > > > also
> > > > > an
> > > > > > > > >> > important
> > > > > > > > >> > > > > > feature for MM2 users.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Daniel
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Re: Re: [DISCUSS] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

[Dániel Urbán <ur...@gmail.com>]

Hi Mickael,
Thanks for the input, I updated the KIP with the config name you suggested.
Daniel

Mickael Maison <mi...@gmail.com> ezt írta (időpont: 2022. nov. 7.,
H, 10:48):

> Hi Daniel,
>
> Thanks for the KIP.
>
> It would be nice to expose more of the REST API as some endpoints are
> really useful, for example /admin or
> /connectors/<connector>/tasks-config. However as dedicated mode is
> currently unusable, I think the approach of "just fixing it" by
> exposing the internal endpoints is fine. It also does not seem to
> corner us too much if we decide to make further changes in the future.
>
> One suggestion I have is to avoid using "mm" in the configuration
> name. Could we rename mm.enable.internal.rest to
> dedicated.mode.enable.internal.rest or something like that?
>
> Thanks,
> Mickael
>
> On Tue, Sep 27, 2022 at 3:56 PM Chris Egerton <ch...@aiven.io.invalid>
> wrote:
> >
> > Thanks Daniel! No further comments from me, looks good.
> >
> > On Tue, Sep 27, 2022 at 4:39 AM Dániel Urbán <ur...@gmail.com>
> wrote:
> >
> > > Hi Chris,
> > >
> > > I understand your points, and I agree that this path is safer in terms
> of
> > > future plans, I accept it.
> > > Updated the KIP accordingly.
> > >
> > > Thanks,
> > > Daniel
> > >
> > > Chris Egerton <ch...@aiven.io.invalid> ezt írta (időpont: 2022.
> szept.
> > > 21., Sze, 18:54):
> > >
> > > > Hi Daniel,
> > > >
> > > > I'm a little hesitant to add support for REST extensions and the
> admin
> > > API
> > > > to dedicated MM2 nodes because they may restrict our options in the
> > > future
> > > > if/when we add a public-facing REST API.
> > > >
> > > > Regarding REST extensions specifically, I understand their security
> value
> > > > for public-facing APIs, but for the internal API--which should only
> ever
> > > be
> > > > targeted by MM2 nodes, and never by humans, UIs, CLIs, etc.--I'm not
> sure
> > > > there's enough need there to add that API this time around. The
> internal
> > > > endpoints used by Kafka Connect should be secured by the session key
> > > > mechanism introduced in KIP-507 [1], and IIUC, with this KIP, users
> will
> > > > also be able to configure their cluster to use mTLS.
> > > >
> > > > Regarding the admin API, I consider it part of the public-facing
> REST API
> > > > for Kafka Connect, so I was assuming we wouldn't want to add it to
> this
> > > KIP
> > > > since we're intentionally slimming down the REST API to just the bare
> > > > essentials (i.e., just the internal endpoints). I can see value for
> it,
> > > but
> > > > it's similar to the status endpoints in the Kafka Connect REST API;
> we
> > > > might choose to expose this sometime down the line, but IMO it'd be
> > > better
> > > > to do that in a separate KIP so that we can iron out the details of
> > > exactly
> > > > what kind of REST API would best suit dedicated MM2 clusters, and how
> > > that
> > > > would differ from the API provided by vanilla Kafka Connect.
> > > >
> > > > Let me know what you think!
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > [1] -
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> > > >
> > > > On Wed, Sep 21, 2022 at 4:59 AM Dániel Urbán <ur...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Chris,
> > > > >
> > > > > About the worker id: makes sense. I changed the wording, but kept
> it
> > > > listed
> > > > > as it is a change compared to existing MM2 code. Updated the KIP
> > > > > accordingly.
> > > > >
> > > > > About the REST server configurations: again, I agree, it should be
> > > > > generalized. But I'm not sure about those exceptions you listed,
> as all
> > > > of
> > > > > them make sense in MM2 as well. For example, security related rest
> > > > > extensions could work with MM2 as well. Admin listeners are also
> > > useful,
> > > > as
> > > > > they would allow the same admin operations for MM2 nodes. Am I
> mistaken
> > > > > here? Updated the KIP without mentioning exceptions for now.
> > > > >
> > > > > I think yes, the lazy config resolution should be unconditional.
> Even
> > > if
> > > > we
> > > > > don't consider the distributed mode of MM2, the eager resolution
> does
> > > not
> > > > > allow using sensitive configs in the mm2 properties, as they will
> be
> > > > > written by value into the config topic. I didn't really consider
> this
> > > as
> > > > > breaking change, but I might be wrong.
> > > > >
> > > > > Enable flag property name: also makes sense, updated the KIP.
> > > > >
> > > > > Thanks a lot!
> > > > > Daniel
> > > > >
> > > > > Chris Egerton <ch...@aiven.io.invalid> ezt írta (időpont: 2022.
> > > szept.
> > > > > 20., K, 20:29):
> > > > >
> > > > > > Hi Daniel,
> > > > > >
> > > > > > Looking pretty good! Regarding the worker ID
> generation--apologies, I
> > > > was
> > > > > > unclear with my question. I was wondering if we had to adopt any
> > > > special
> > > > > > logic at all for MM2, or if we could use the same logic that
> vanilla
> > > > > Kafka
> > > > > > Connect does in distributed mode, where the worker ID is its
> > > advertised
> > > > > URL
> > > > > > (e.g., "connect:8083" or "localhost:25565"). Unless I'm mistaken,
> > > this
> > > > > > should allow MM2 nodes to be identified unambiguously. Do you
> think
> > > it
> > > > > > makes sense to follow this strategy?
> > > > > >
> > > > > > Now that the details on the new REST interface have been fleshed
> out,
> > > > I'm
> > > > > > also wondering if we want to add support for the "
> > > > > > rest.advertised.host.name",
> > > > > > "rest.advertised.port", and "rest.advertised.listener"
> properties,
> > > > which
> > > > > > are vital for being able to run Kafka Connect in distributed mode
> > > from
> > > > > > within containers. In fact, I'm wondering if we can generalize
> some
> > > of
> > > > > the
> > > > > > specification in the KIP around which REST properties are
> accepted by
> > > > > > stating that all REST-related properties that are available with
> > > > vanilla
> > > > > > Kafka Connect will be supported for dedicated MM2 nodes when
> > > > > > "mm.enable.rest" is set to "true", except for ones related to the
> > > > > > public-facing REST API like "rest.extension.classes",
> > > > "admin.listeners",
> > > > > > and "admin.listeners.https.*".
> > > > > >
> > > > > > One other thought--is the lazy evaluation of config provider
> > > references
> > > > > > going to take place unconditionally, or only when the internal
> REST
> > > API
> > > > > is
> > > > > > enabled on a worker?
> > > > > >
> > > > > > Finally, do you think we could change "mm.enable.rest" to
> > > > > > "mm.enable.internal.rest"? That way, if we want to introduce a
> > > > > > public-facing REST API later on, we can use "mm.enable.rest" as
> the
> > > > name
> > > > > > for that property.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Fri, Sep 16, 2022 at 9:28 AM Dániel Urbán <
> urb.daniel7@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Chris,
> > > > > > >
> > > > > > > I went through the KIP and updated it based on our discussion.
> I
> > > > think
> > > > > > your
> > > > > > > suggestions simplified (and shortened) the KIP significantly.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Daniel
> > > > > > >
> > > > > > > Dániel Urbán <du...@cloudera.com.invalid> ezt írta (időpont:
> > > 2022.
> > > > > > szept.
> > > > > > > 16., P, 15:15):
> > > > > > >
> > > > > > > > Hi Chris,
> > > > > > > >
> > > > > > > > 1. For the REST-server-per-flow setup, it made sense to
> introduce
> > > > > some
> > > > > > > > simplified configuration. With a single REST server, it
> doesn't
> > > > make
> > > > > > > sense
> > > > > > > > anymore, I'm removing it from the KIP.
> > > > > > > > 2. I think that changing the worker ID generation still makes
> > > > sense,
> > > > > > > > otherwise there is no way to differentiate between the MM2
> > > > instances.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Daniel
> > > > > > > >
> > > > > > > > On Wed, Aug 31, 2022 at 8:39 PM Chris Egerton
> > > > > <chrise@aiven.io.invalid
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Daniel,
> > > > > > > > >
> > > > > > > > > I've taken a look at the KIP in detail. Here are my
> complete
> > > > > thoughts
> > > > > > > > > (minus the aforementioned sections that may be affected by
> > > > changes
> > > > > to
> > > > > > > an
> > > > > > > > > internal-only REST API):
> > > > > > > > >
> > > > > > > > > 1. Why introduce new mm.host.name and mm.rest.protocol
> > > > properties
> > > > > > > > instead
> > > > > > > > > of using the properties that are already used by Kafka
> Connect:
> > > > > > > > listeners,
> > > > > > > > > rest.advertised.host.name, rest.advertised.host.port, and
> > > > > > > > > rest.advertised.listener? We used to have the
> rest.host.name
> > > and
> > > > > > > > rest.port
> > > > > > > > > properties in Connect but deprecated and eventually removed
> > > them
> > > > in
> > > > > > > favor
> > > > > > > > > of the listeners property in KIP-208 [1]; I'm hoping we can
> > > keep
> > > > > > things
> > > > > > > > as
> > > > > > > > > similar as possible between MM2 and Connect in order to
> make it
> > > > > > easier
> > > > > > > > for
> > > > > > > > > users to work with both. I'm also hoping that we can allow
> > > users
> > > > to
> > > > > > > > > configure the port that their MM2 nodes listen on instead
> of
> > > > > > hardcoding
> > > > > > > > MM2
> > > > > > > > > to bind to port 0.
> > > > > > > > >
> > > > > > > > > 2. Do we still need to change the worker IDs that get used
> in
> > > the
> > > > > > > status
> > > > > > > > > topic?
> > > > > > > > >
> > > > > > > > > Everything else looks good, or should change once the KIP
> is
> > > > > updated
> > > > > > > with
> > > > > > > > > the internal-only REST API alternative.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > [1] -
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface
> > > > > > > > >
> > > > > > > > > On Mon, Aug 29, 2022 at 1:55 PM Chris Egerton <
> chrise@aiven.io
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Daniel,
> > > > > > > > > >
> > > > > > > > > > Yeah, I think that's the way to go. Adding multiple
> servers
> > > for
> > > > > > each
> > > > > > > > > > herder seems like it'd be too much of a pain for users to
> > > > > > configure,
> > > > > > > > and
> > > > > > > > > if
> > > > > > > > > > we keep the API strictly internal for now, we shouldn't
> be
> > > > > painting
> > > > > > > > > > ourselves into too much of a corner if/when we decide to
> > > > expose a
> > > > > > > > > > public-facing REST API for dedicated MM2 clusters.
> > > > > > > > > >
> > > > > > > > > > I plan to take a look at the rest of the KIP and provide
> a
> > > > > complete
> > > > > > > > > review
> > > > > > > > > > sometime this week; I'll hold off on commenting on
> anything
> > > > that
> > > > > > > seems
> > > > > > > > > like
> > > > > > > > > > it'll be affected by switching to an internal-only REST
> API
> > > > until
> > > > > > > those
> > > > > > > > > > changes are published, but should be able to review
> > > everything
> > > > > > else.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
> > > > > > > > > > Chris
> > > > > > > > > >
> > > > > > > > > > On Mon, Aug 29, 2022 at 6:57 AM Dániel Urbán <
> > > > > > urb.daniel7@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi Chris,
> > > > > > > > > >>
> > > > > > > > > >> I understand your point, sounds good to me.
> > > > > > > > > >> So in short, we should opt for an internal-only API, and
> > > > > > preferably
> > > > > > > a
> > > > > > > > > >> single server solution. Is that right?
> > > > > > > > > >>
> > > > > > > > > >> Thanks
> > > > > > > > > >> Daniel
> > > > > > > > > >>
> > > > > > > > > >> Chris Egerton <ch...@aiven.io.invalid> ezt írta
> (időpont:
> > > > > 2022.
> > > > > > > aug.
> > > > > > > > > >> 26.,
> > > > > > > > > >> P, 17:36):
> > > > > > > > > >>
> > > > > > > > > >> > Hi Daniel,
> > > > > > > > > >> >
> > > > > > > > > >> > Glad to hear from you!
> > > > > > > > > >> >
> > > > > > > > > >> > With regards to the stripped-down REST API
> alternative, I
> > > > > don't
> > > > > > > see
> > > > > > > > > how
> > > > > > > > > >> > this would prevent us from introducing the
> fully-fledged
> > > > > Connect
> > > > > > > > REST
> > > > > > > > > >> API,
> > > > > > > > > >> > or even an augmented variant of it, at some point
> down the
> > > > > road.
> > > > > > > If
> > > > > > > > we
> > > > > > > > > >> go
> > > > > > > > > >> > with the internal-only API now, and want to expand
> later,
> > > > > can't
> > > > > > we
> > > > > > > > > gate
> > > > > > > > > >> the
> > > > > > > > > >> > expansion behind a feature flag configuration property
> > > that
> > > > by
> > > > > > > > default
> > > > > > > > > >> > disables the new feature?
> > > > > > > > > >> >
> > > > > > > > > >> > I'm also not sure that we'd ever want to expose the
> raw
> > > > > Connect
> > > > > > > REST
> > > > > > > > > API
> > > > > > > > > >> > for dedicated MM2 clusters. If people want that
> > > capability,
> > > > > they
> > > > > > > can
> > > > > > > > > >> > already spin up a vanilla Connect cluster and run as
> many
> > > > MM2
> > > > > > > > > >> connectors as
> > > > > > > > > >> > they'd like on it, and as of KIP-458 [1], it's even
> > > possible
> > > > > to
> > > > > > > use
> > > > > > > > a
> > > > > > > > > >> > single Connect cluster to replicate between any two
> Kafka
> > > > > > clusters
> > > > > > > > > >> instead
> > > > > > > > > >> > of only targeting the Kafka cluster that the vanilla
> > > Connect
> > > > > > > cluster
> > > > > > > > > >> > operates on top of. I do agree that it'd be great to
> be
> > > able
> > > > > to
> > > > > > > > > >> dynamically
> > > > > > > > > >> > adjust things like topic filters without having to
> > > restart a
> > > > > > > > dedicated
> > > > > > > > > >> MM2
> > > > > > > > > >> > node; I'm just not sure that the vanilla Connect REST
> API
> > > is
> > > > > the
> > > > > > > > > >> > appropriate way to do that, especially since the exact
> > > > > > mechanisms
> > > > > > > > that
> > > > > > > > > >> make
> > > > > > > > > >> > a single Connect cluster viable for replicating
> across any
> > > > two
> > > > > > > Kafka
> > > > > > > > > >> > clusters could be abused and cause a dedicated MM2
> cluster
> > > > to
> > > > > > > start
> > > > > > > > > >> writing
> > > > > > > > > >> > to a completely different Kafka cluster that's not
> even
> > > > > defined
> > > > > > in
> > > > > > > > its
> > > > > > > > > >> > config file.
> > > > > > > > > >> >
> > > > > > > > > >> > Finally, as far as security goes--since this is
> > > essentially
> > > > a
> > > > > > bug
> > > > > > > > fix,
> > > > > > > > > >> I'm
> > > > > > > > > >> > inclined to make it as easy as possible for users to
> adopt
> > > > it.
> > > > > > > MTLS
> > > > > > > > > is a
> > > > > > > > > >> > great start for securing a REST API, but it's not
> > > sufficient
> > > > > on
> > > > > > > its
> > > > > > > > > own
> > > > > > > > > >> > since anyone who could issue an authenticated REST
> request
> > > > > > against
> > > > > > > > the
> > > > > > > > > >> MM2
> > > > > > > > > >> > cluster would still be able to make any changes they
> want
> > > > > (with
> > > > > > > the
> > > > > > > > > >> > exception of accessing internal endpoints, which were
> > > > secured
> > > > > > with
> > > > > > > > > >> > KIP-507). If we were to bring up the fully-fledged
> Connect
> > > > > REST
> > > > > > > API,
> > > > > > > > > >> > cluster administrators would also likely have to add
> some
> > > > kind
> > > > > > of
> > > > > > > > > >> > authorization layer to prevent people from using the
> REST
> > > > API
> > > > > to
> > > > > > > > mess
> > > > > > > > > >> with
> > > > > > > > > >> > the configurations of the connectors that MM2 brought
> up.
> > > > One
> > > > > > way
> > > > > > > of
> > > > > > > > > >> doing
> > > > > > > > > >> > that is to add a REST extension to your Connect
> cluster,
> > > but
> > > > > > > > > >> implementing
> > > > > > > > > >> > and configuring one in order to be able to run a
> > > multi-node
> > > > > MM2
> > > > > > > > > cluster
> > > > > > > > > >> > without hitting this bug seems like too much work to
> be
> > > > worth
> > > > > > it.
> > > > > > > > > >> >
> > > > > > > > > >> > I think if we had a better picture of what a REST API
> for
> > > > > > > dedicated
> > > > > > > > > MM2
> > > > > > > > > >> > clusters would/should look like, then it would be
> easier
> > > to
> > > > go
> > > > > > > along
> > > > > > > > > >> with
> > > > > > > > > >> > this, and we could even just add the feature flag in
> this
> > > > KIP
> > > > > > > right
> > > > > > > > > now
> > > > > > > > > >> to
> > > > > > > > > >> > address any security concerns. My instinct would be to
> > > > address
> > > > > > > this
> > > > > > > > > in a
> > > > > > > > > >> > follow-up KIP in order to reduce scope creep, though,
> and
> > > > keep
> > > > > > > this
> > > > > > > > > KIP
> > > > > > > > > >> > focused on addressing the bug with multi-node
> dedicated
> > > MM2
> > > > > > > > clusters.
> > > > > > > > > >> What
> > > > > > > > > >> > do you think?
> > > > > > > > > >> >
> > > > > > > > > >> > Cheers,
> > > > > > > > > >> >
> > > > > > > > > >> > Chris
> > > > > > > > > >> >
> > > > > > > > > >> > [1] -
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy
> > > > > > > > > >> >
> > > > > > > > > >> > On Thu, Aug 25, 2022 at 3:55 AM Dániel Urbán <
> > > > > > > urb.daniel7@gmail.com
> > > > > > > > >
> > > > > > > > > >> > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > Hi Chris,
> > > > > > > > > >> > >
> > > > > > > > > >> > > Thanks for bringing this up again :)
> > > > > > > > > >> > >
> > > > > > > > > >> > > 1. I think that is reasonable, though I find the
> current
> > > > > state
> > > > > > > of
> > > > > > > > > MM2
> > > > > > > > > >> to
> > > > > > > > > >> > be
> > > > > > > > > >> > > confusing. The current issue with the distributed
> mode
> > > is
> > > > > not
> > > > > > > > > >> documented
> > > > > > > > > >> > > properly, but maybe the logging will help a bit.
> > > > > > > > > >> > >
> > > > > > > > > >> > > 2. Going for an internal-only Connect REST version
> would
> > > > > lock
> > > > > > > MM2
> > > > > > > > > out
> > > > > > > > > >> of
> > > > > > > > > >> > a
> > > > > > > > > >> > > path where the REST API can be used to dynamically
> > > > > reconfigure
> > > > > > > > > >> > > replications. For now, I agree, it would be easy to
> > > > corrupt
> > > > > > the
> > > > > > > > > state
> > > > > > > > > >> of
> > > > > > > > > >> > > MM2 if someone wanted to use the properties and the
> REST
> > > > at
> > > > > > the
> > > > > > > > same
> > > > > > > > > >> > time,
> > > > > > > > > >> > > but in the future, we might have a chance to
> introduce a
> > > > > > > different
> > > > > > > > > >> config
> > > > > > > > > >> > > mechanism, where only the cluster connections have
> to be
> > > > > > > specified
> > > > > > > > > in
> > > > > > > > > >> the
> > > > > > > > > >> > > properties file, and everything else can be
> configured
> > > > > through
> > > > > > > > REST
> > > > > > > > > >> > > (enabling replications, changing topic filters,
> etc.).
> > > > > Because
> > > > > > > of
> > > > > > > > > >> this,
> > > > > > > > > >> > I'm
> > > > > > > > > >> > > leaning towards a full Connect REST API. To avoid
> issues
> > > > > with
> > > > > > > > > >> conflicts
> > > > > > > > > >> > > between the props file and the REST, we could
> document
> > > > > > security
> > > > > > > > best
> > > > > > > > > >> > > practices (e.g. turn on basic auth or mTLS on the
> > > Connect
> > > > > REST
> > > > > > > to
> > > > > > > > > >> avoid
> > > > > > > > > >> > > unwanted interactions).
> > > > > > > > > >> > >
> > > > > > > > > >> > > 3. That is a good point, and I agree, a big plus for
> > > > > > motivation.
> > > > > > > > > >> > >
> > > > > > > > > >> > > I have a working version of this in which all flows
> spin
> > > > up
> > > > > a
> > > > > > > > > >> dedicated
> > > > > > > > > >> > > Connect REST, but I can give other solutions a try,
> too.
> > > > > > > > > >> > >
> > > > > > > > > >> > > Thanks,
> > > > > > > > > >> > > Daniel
> > > > > > > > > >> > >
> > > > > > > > > >> > > Chris Egerton <ch...@aiven.io.invalid> ezt írta
> > > > (időpont:
> > > > > > > 2022.
> > > > > > > > > aug.
> > > > > > > > > >> > 24.,
> > > > > > > > > >> > > Sze, 17:46):
> > > > > > > > > >> > >
> > > > > > > > > >> > > > Hi Daniel,
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > I'd like to resurface this KIP in case you're
> still
> > > > > > interested
> > > > > > > > in
> > > > > > > > > >> > > pursuing
> > > > > > > > > >> > > > it. I know it's been a while since you published
> it,
> > > and
> > > > > it
> > > > > > > > hasn't
> > > > > > > > > >> > > received
> > > > > > > > > >> > > > much attention, but I'm hoping we can give it a
> try
> > > now
> > > > > and
> > > > > > > > > finally
> > > > > > > > > >> put
> > > > > > > > > >> > > > this long-standing bug to rest. To that end, I
> have
> > > some
> > > > > > > > thoughts
> > > > > > > > > >> about
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > proposal. This isn't a complete review, but I
> wanted
> > > to
> > > > > give
> > > > > > > > > enough
> > > > > > > > > >> to
> > > > > > > > > >> > > get
> > > > > > > > > >> > > > the ball rolling:
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > 1. Some environments with firewalls or strict
> security
> > > > > > > policies
> > > > > > > > > may
> > > > > > > > > >> not
> > > > > > > > > >> > > be
> > > > > > > > > >> > > > able to bring up a REST server for each MM2 node.
> If
> > > we
> > > > > > decide
> > > > > > > > > that
> > > > > > > > > >> > we'd
> > > > > > > > > >> > > > like to use the Connect REST API (or even just
> parts
> > > of
> > > > > it)
> > > > > > to
> > > > > > > > > >> address
> > > > > > > > > >> > > this
> > > > > > > > > >> > > > bug with MM2, it does make sense to eventually
> make
> > > the
> > > > > > > > > >> availability of
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > REST API a hard requirement for running MM2, but
> it
> > > > might
> > > > > > be a
> > > > > > > > bit
> > > > > > > > > >> too
> > > > > > > > > >> > > > abrupt to do that all in a single release. What
> do you
> > > > > think
> > > > > > > > about
> > > > > > > > > >> > making
> > > > > > > > > >> > > > the REST API optional for now, but noting that it
> will
> > > > > > become
> > > > > > > > > >> required
> > > > > > > > > >> > > in a
> > > > > > > > > >> > > > later release (probably 4.0.0 or, if that's not
> enough
> > > > > time,
> > > > > > > > > >> 5.0.0)? We
> > > > > > > > > >> > > > could choose not to bring the REST server for any
> node
> > > > > whose
> > > > > > > > > >> > > configuration
> > > > > > > > > >> > > > doesn't explicitly opt into one, and maybe log a
> > > warning
> > > > > > > message
> > > > > > > > > on
> > > > > > > > > >> > > startup
> > > > > > > > > >> > > > if none is configured. In effect, we'd be marking
> the
> > > > > > current
> > > > > > > > mode
> > > > > > > > > >> (no
> > > > > > > > > >> > > REST
> > > > > > > > > >> > > > server) as deprecated.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > 2. I'm not sure that we should count out the
> "Creating
> > > > an
> > > > > > > > > >> internal-only
> > > > > > > > > >> > > > derivation of the Connect REST API" rejected
> > > > alternative.
> > > > > > > Right
> > > > > > > > > now,
> > > > > > > > > >> > the
> > > > > > > > > >> > > > single source of truth for the configuration of a
> MM2
> > > > > > cluster
> > > > > > > > > >> (assuming
> > > > > > > > > >> > > > it's being run in dedicated mode, and not as a
> > > connector
> > > > > in
> > > > > > a
> > > > > > > > > >> vanilla
> > > > > > > > > >> > > > Connect cluster) is the configuration file used
> for
> > > the
> > > > > > > process.
> > > > > > > > > By
> > > > > > > > > >> > > > bringing up the REST API, we'd expose endpoints to
> > > > modify
> > > > > > > > > connector
> > > > > > > > > >> > > > configurations, which would not only add
> complexity to
> > > > the
> > > > > > > > > operation
> > > > > > > > > >> > of a
> > > > > > > > > >> > > > MM2 cluster, but even qualify as an attack vector
> for
> > > > > > > malicious
> > > > > > > > > >> > entities.
> > > > > > > > > >> > > > Thanks to KIP-507 we have some amount of security
> > > around
> > > > > the
> > > > > > > > > >> > > internal-only
> > > > > > > > > >> > > > endpoints used by the Connect framework, but for
> any
> > > > > public
> > > > > > > > > >> endpoints,
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > Connect REST API doesn't come with any security
> out of
> > > > the
> > > > > > > box.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > 3. Small point, but with support for exactly-once
> > > source
> > > > > > > > > connectors
> > > > > > > > > >> > > coming
> > > > > > > > > >> > > > out in 3.3.0, it's also worth noting that that's
> > > another
> > > > > > > feature
> > > > > > > > > >> that
> > > > > > > > > >> > > won't
> > > > > > > > > >> > > > work properly with multi-node MM2 clusters without
> > > > adding
> > > > > a
> > > > > > > REST
> > > > > > > > > >> server
> > > > > > > > > >> > > for
> > > > > > > > > >> > > > each node (or some substitute that accomplishes
> the
> > > same
> > > > > > > goal).
> > > > > > > > I
> > > > > > > > > >> don't
> > > > > > > > > >> > > > think this will affect the direction of the design
> > > > > > discussion
> > > > > > > > too
> > > > > > > > > >> much,
> > > > > > > > > >> > > but
> > > > > > > > > >> > > > it does help strengthen the motivation.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Cheers,
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Chris
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > On 2021/02/18 15:57:36 Dániel Urbán wrote:
> > > > > > > > > >> > > > > Hello everyone,
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > * Sorry, I meant KIP-710.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Right now the MirrorMaker cluster is somewhat
> > > > > unreliable,
> > > > > > > and
> > > > > > > > > not
> > > > > > > > > >> > > > > supporting running in a cluster properly. I'd
> say
> > > that
> > > > > > > fixing
> > > > > > > > > this
> > > > > > > > > >> > > would
> > > > > > > > > >> > > > be
> > > > > > > > > >> > > > > a nice addition.
> > > > > > > > > >> > > > > Does anyone have some input on this?
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Thanks in advance
> > > > > > > > > >> > > > > Daniel
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Dániel Urbán <ur...@gmail.com> ezt írta
> (időpont:
> > > > 2021.
> > > > > > > jan.
> > > > > > > > > >> 26., K,
> > > > > > > > > >> > > > > 15:56):
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > Hello everyone,
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > I would like to start a discussion on KIP-709,
> > > which
> > > > > > > > addresses
> > > > > > > > > >> some
> > > > > > > > > >> > > > > > missing features in MM2 dedicated mode.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > Currently, the dedicated mode of MM2 does not
> > > fully
> > > > > > > support
> > > > > > > > > >> running
> > > > > > > > > >> > > in
> > > > > > > > > >> > > > a
> > > > > > > > > >> > > > > > cluster. The core issue is that the Connect
> REST
> > > > > Server
> > > > > > is
> > > > > > > > not
> > > > > > > > > >> > > included
> > > > > > > > > >> > > > in
> > > > > > > > > >> > > > > > the dedicated mode, which makes
> follower->leader
> > > > > > > > communication
> > > > > > > > > >> > > > impossible.
> > > > > > > > > >> > > > > > In some cases, this results in the cluster not
> > > being
> > > > > > able
> > > > > > > to
> > > > > > > > > >> react
> > > > > > > > > >> > to
> > > > > > > > > >> > > > > > dynamic configuration changes (e.g. dynamic
> topic
> > > > > filter
> > > > > > > > > >> changes).
> > > > > > > > > >> > > > > > Another smaller detail is that MM2 dedicated
> mode
> > > > > > eagerly
> > > > > > > > > >> resolves
> > > > > > > > > >> > > > config
> > > > > > > > > >> > > > > > provider references in the Connector
> > > configurations,
> > > > > > which
> > > > > > > > is
> > > > > > > > > >> > > > undesirable
> > > > > > > > > >> > > > > > and a breaking change compared to vanilla
> Connect.
> > > > > This
> > > > > > > can
> > > > > > > > > >> cause
> > > > > > > > > >> > an
> > > > > > > > > >> > > > issue
> > > > > > > > > >> > > > > > for example when there is an environment
> variable
> > > > > > > reference,
> > > > > > > > > >> which
> > > > > > > > > >> > > > contains
> > > > > > > > > >> > > > > > some host-specific information, like a file
> path.
> > > > The
> > > > > > > leader
> > > > > > > > > >> > resolves
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > reference eagerly, and the resolved value is
> > > > > propagated
> > > > > > to
> > > > > > > > > other
> > > > > > > > > >> > MM2
> > > > > > > > > >> > > > nodes
> > > > > > > > > >> > > > > > instead of the reference being resolved
> locally,
> > > > > > > separately
> > > > > > > > on
> > > > > > > > > >> each
> > > > > > > > > >> > > > node.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > The KIP addresses these by adding the Connect
> REST
> > > > > > Server
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > >> > MM2
> > > > > > > > > >> > > > > > dedicated mode for each replication flow, and
> > > > > postponing
> > > > > > > the
> > > > > > > > > >> config
> > > > > > > > > >> > > > > > provider reference resolution.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > Please discuss, I know this is a major
> change, but
> > > > > also
> > > > > > an
> > > > > > > > > >> > important
> > > > > > > > > >> > > > > > feature for MM2 users.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > Daniel
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>