You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Greg Harris <gr...@aiven.io.INVALID> on 2024/01/02 19:49:07 UTC
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Hi Snehashis,
Thank you for the KIP! This is something I've wanted for a long time.
I know the discussion has gone cold, are you still interested in
pursuing this feature? I'll make time to review the KIP if you are
still accepting comments.
Thanks,
Greg
On Tue, Nov 22, 2022 at 12:29 PM Snehashis <sn...@gmail.com> wrote:
>
> Thanks for the points Sagar.
>
> > 1) Should we update the GET /connectors endpoint to include the version of
> > the plugin that is running? It could be useful to figure out the version
> of
> > the plugin or I am assuming it gets returned by the expand=info call?
>
> I think this is good to have and possible future enhancement. The version
> info will be present in the config of the connector if the user has
> specified the version. Otherwise it is the latest version which the user
> can find out from the connector-plugin endpoint. The information can be
> introduced to the response of the GET /connectors endpoint itself, however
> the most ideal way of doing this would be to get the currently running
> instance of the connector and get the version directly from there. This is
> slightly tricky as the connector could be running in a different node.
> One way to do this would be to persist the version information in the
> status backing store during instantiation of the connector. It requires
> some more thought and since the version is part of the configs if provided
> and evident otherwise, I have not included it in this KIP.
>
> > 2) I am not aware of this and hence asking, can 2 connectors with
> different
> > versions have the same name? Does the plugin isolation allow this? This
> > could have a bearing when using the lifecycle endpoints for connectors
> like
> > DELETE etc.
>
> All connectors in a cluster need to have uniquire connector names
> regardless of what version of the plugin the connector is running
> underneath. This is something enforced by the connect runtime itself. All
> connect CRUD operations are keyed on the connector name so there will not
> be an issue.
>
> Regards
> Snehashis
>
> On Tue, Nov 22, 2022 at 3:16 PM Sagar <sa...@gmail.com> wrote:
>
> > Hey Snehashsih,
> >
> > Thanks for the KIP. It looks like a very useful feature. Couple of
> > small-ish points, let me know what you think:
> >
> > 1) Should we update the GET /connectors endpoint to include the version of
> > the plugin that is running? It could be useful to figure out the version of
> > the plugin or I am assuming it gets returned by the expand=info call?
> > 2) I am not aware of this and hence asking, can 2 connectors with different
> > versions have the same name? Does the plugin isolation allow this? This
> > could have a bearing when using the lifecycle endpoints for connectors like
> > DELETE etc.
> >
> > Thanks!
> > Sagar.
> >
> >
> > On Tue, Nov 22, 2022 at 2:10 PM Ashwin <ap...@confluent.io.invalid>
> > wrote:
> >
> > > Hi Snehasis,
> > >
> > > > IIUC (please correct me if I am wrong here), what you highlighted
> > above,
> > > is
> > > a versioning scheme for a connector config for the same connector (and
> > not
> > > different versions of a connector plugin).
> > >
> > > Sorry for not being more precise in my wording - I meant registering
> > > versions of schema for connector config.
> > >
> > > Let's take the example of a fictional connector which uses a fictional
> > AWS
> > > service.
> > >
> > > Fictional Connector Config schema version:2.0
> > > ---
> > > {
> > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > "type": "object",
> > > "properties": {
> > > "name": {
> > > "type": "string"
> > > },
> > > "schema_version": {
> > > "type": "string"
> > > },
> > > "aws_access_key": {
> > > "type": "string"
> > > },
> > > "aws_secret_key": {
> > > "type": "string"
> > > }
> > > },
> > > "required": [
> > > "name",
> > > "schema_version",
> > > "aws_access_key",
> > > "aws_secret_key"
> > > ]
> > > }
> > >
> > > Fictional Connector config schema version:3.0
> > > ---
> > > {
> > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > "type": "object",
> > > "properties": {
> > > "name": {
> > > "type": "string"
> > > },
> > > "schema_version": {
> > > "type": "string"
> > > },
> > > "iam_role": {
> > > "type": "string"
> > > }
> > > },
> > > "required": [
> > > "name",
> > > "schema_version",
> > > "iam_role"
> > > ]
> > > }
> > >
> > > The connector which supports Fictional config schema 2.0 will validate
> > the
> > > access key and secret key.
> > > Whereas a connector which supports config with schema version 3.0 will
> > only
> > > validate the IAM role.
> > >
> > > This is the alternative which I wanted to suggest. Each plugin will
> > > register the schema versions of connector config which it supports.
> > >
> > > The plugin paths may be optionally different i.e we don't have to
> > > mandatorily add a new plugin path to support a new schema version.
> > >
> > > Thanks,
> > > Ashwin
> > >
> > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <sn...@gmail.com>
> > > wrote:
> > >
> > > > Thanks for the input Ashwin.
> > > >
> > > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > > > config is versioned and has a schema. Then a single plugin (whose
> > > > > dependencies have not changed) can handle multiple config versions
> > for
> > > > the
> > > > > same connector class.
> > > >
> > > > IIUC (please correct me if I am wrong here), what you highlighted
> > above,
> > > is
> > > > a versioning scheme for a connector config for the same connector (and
> > > not
> > > > different versions of a connector plugin). That is a somewhat
> > tangential
> > > > problem. While it is definitely a useful feature to have, like a log to
> > > > check what changes were made over time to the config which might make
> > it
> > > > easier to do rollbacks, it is not the focus here. Here by version we
> > mean
> > > > to say what underlying version of the plugin should the given
> > > configuration
> > > > of the connector use. Perhaps it is better to change the name of the
> > > > parameter from connector.version to connector.plugin.version or
> > > > plugin.version if it was confusing. wdyt?
> > > >
> > > > > 2. Any plans to support assisted migration e.g if a user invokes
> > "POST
> > > > > connector/config?migrate=latest", the latest version __attempts__ to
> > > > > transform the existing config to the newer version. This would
> > require
> > > > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > > > connector interface.
> > > >
> > > > This is an enhancement we can think of doing in future. Users can
> > simply
> > > do
> > > > a PUT call with the updated config which has the updated version
> > number.
> > > > The assisted mode could be handy as the user does not need to know the
> > > > config but beyond this it does not seem to justify its existence.
> > > >
> > > > Regards
> > > > Snehashis
> > > >
> > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin <ap...@confluent.io.invalid>
> > > > wrote:
> > > >
> > > > > Hi Snehasis,
> > > > >
> > > > > This is a really useful feature and thanks for initiating this
> > > > discussion.
> > > > >
> > > > > I had the following questions -
> > > > >
> > > > >
> > > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > > > config is versioned and has a schema. Then a single plugin (whose
> > > > > dependencies have not changed) can handle multiple config versions
> > for
> > > > the
> > > > > same connector class.
> > > > >
> > > > > 2. Any plans to support assisted migration e.g if a user invokes
> > "POST
> > > > > connector/config?migrate=latest", the latest version __attempts__ to
> > > > > transform the existing config to the newer version. This would
> > require
> > > > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > > > connector interface.
> > > > >
> > > > > Thanks,
> > > > > Ashwin
> > > > >
> > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <sn...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to start a discussion thread on KIP-891: Running multiple
> > > > > versions
> > > > > > of a connector.
> > > > > >
> > > > > > The KIP aims to add the ability for the connect runtime to run
> > > multiple
> > > > > > versions of a connector.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > >
> > > > > > Please take a look and let me know what you think.
> > > > > >
> > > > > > Thank you
> > > > > > Snehashis Pal
> > > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Snehashis <sn...@gmail.com>.
Hi Greg,
I have started a vote thread and added it to the doc. Thanks for all your
help on this. Looking forward to this implementation.
Regards
Snehashis
On Wed, Sep 25, 2024 at 10:38 PM Greg Harris <gr...@aiven.io.invalid>
wrote:
> Hey Snehashis,
>
> I updated the KIP to remove some stale mentions of the soft version
> requirements, and the crashing workers on startup. I also added more detail
> to the REST API.
>
> IMHO we're ready to move to voting, so please open the thread if you also
> believe it is ready.
>
> Thanks!
>
> Greg
>
> On Fri, Aug 23, 2024 at 7:21 AM Snehashis <sn...@gmail.com>
> wrote:
>
> > Hi Greg,
> >
> > Thanks for the clarification!
> >
> > I agree with not breaking compatibility and opting to not fail worker
> > startup by validating converter versions.
> >
> > Let's also go with the unclosed variation of the requirements as a hard
> > requirement i.e. 3.8 instead of [3.8]. I have updated the KIP and added a
> > line there highlighting this.
> >
> > Regards
> > Snehashis
> >
> > On Wed, Aug 21, 2024 at 9:24 PM Greg Harris <greg.harris@aiven.io.invalid
> >
> > wrote:
> >
> > > Hey Snehashis,
> > >
> > > Thanks for your reply!
> > >
> > > > Deviating
> > > > it from the spec seems unnecessary if we document it accordingly,
> > however
> > > > It's probably less intuitive and can lead to confusion. I would just
> > keep
> > > > it as is but making it simpler is also fine.
> > >
> > > It sounds like you don't have a strong opinion on this, similar to me.
> > > Chris had a more firm stance on this. I think that you're right that we
> > > could document this, but it would still be the single biggest foot-gun
> of
> > > the feature, and I think we would regret it later.
> > >
> > > > Also for converter versions
> > > > specified as part of the worker configs I believe we concluded that
> > this
> > > > step need not be fatal during worker startup if the required version
> is
> > > not
> > > > found but LMK if otherwise.
> > >
> > > Re-reading our earlier discussion, I think Chris had a strong opinion
> > that
> > > we shouldn't fail on startup because it would be inconsistent. I made
> an
> > > offhand comment that if this was released in 4.0, we could change it so
> > > both invalid classes and invalid versions cause the worker to fail,
> which
> > > would be inconsistent but backwards incompatible.
> > > I think in the interest of not breaking compatibility needlessly and
> > > keeping consistent behavior, we should ignore invalid versions in the
> > > worker config.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Wed, Aug 21, 2024 at 1:58 AM Snehashis <sn...@gmail.com>
> > > wrote:
> > >
> > > > Hi Greg,
> > > >
> > > > No issues, I have been caught up in a few things myself.
> > > >
> > > > I have added the points we discussed. In addition, I have added
> config
> > > > providers as part of the set of plugins which will not support
> > > versioning,
> > > > for the same reason as to why it is not supported in the other
> plugins
> > > that
> > > > are initiated on startup.
> > > >
> > > > On whether to deviate from maven versioning for hard requirements as
> > > > discussed between you and Chris.Whether to simply simply specify
> > > > connector.version=1.1.1 as a hard requirement instead of [1.1.1].
> > > Deviating
> > > > it from the spec seems unnecessary if we document it accordingly,
> > however
> > > > It's probably less intuitive and can lead to confusion. I would just
> > keep
> > > > it as is but making it simpler is also fine. Also for converter
> > versions
> > > > specified as part of the worker configs I believe we concluded that
> > this
> > > > step need not be fatal during worker startup if the required version
> is
> > > not
> > > > found but LMK if otherwise.
> > > >
> > > > Regards
> > > > Snehashis
> > > >
> > > >
> > > >
> > > > On Mon, Aug 19, 2024 at 9:24 PM Greg Harris
> > <greg.harris@aiven.io.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hi Snehashis,
> > > > >
> > > > > Sorry for the late reply.
> > > > >
> > > > > > Heterogeneous dependencies in a multi cluster deployment is
> highly
> > > > > discouraged
> > > > >
> > > > > Right, this remains unchanged in this KIP.
> > > > >
> > > > > > Let's add the version information for both connector and tasks in
> > the
> > > > > connector status itself
> > > > > > once we add these two additions to the KIP (LMK if you want me to
> > > take
> > > > > that up).
> > > > >
> > > > > Could you make these additions?
> > > > >
> > > > > I'm interested to see if we can include this in 4.0.
> > > > >
> > > > > Thanks,
> > > > > Greg
> > > > >
> > > > > On Tue, Jul 2, 2024 at 2:52 AM Snehashis <snehashisp1995@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi Greg,
> > > > > >
> > > > > > Thanks for taking a look at this, to conclude on the two points
> > > above.
> > > > > >
> > > > > > 1. I'm okay with the status quo of leaving the dependency
> > management
> > > of
> > > > > > plugins to systems outside of the connect runtime as it is now.
> > Given
> > > > > that
> > > > > > the dependencies are homogenous across a connect cluster, it
> should
> > > > > ensure
> > > > > > that task and connector versions are uniform across a deployment,
> > > after
> > > > > it
> > > > > > has stabilised post an upgrade operation. Heterogeneous
> > dependencies
> > > > in a
> > > > > > multi cluster deployment is highly discouraged and we should
> point
> > > out
> > > > > this
> > > > > > can lead to inconsistent behaviour with connectors and even
> > > > validations.
> > > > > >
> > > > > > 2. Let's add the version information for both connector and tasks
> > in
> > > > the
> > > > > > connector status itself, I don't see another endpoint being
> > required
> > > > for
> > > > > > this.
> > > > > >
> > > > > > I don't have any further points, so if you are okay we can put
> this
> > > to
> > > > a
> > > > > > vote, once we add these two additions to the KIP (LMK if you want
> > me
> > > to
> > > > > > take that up).
> > > > > >
> > > > > > Regards
> > > > > > Snehashis
> > > > > >
> > > > > > On Tue, Jul 2, 2024 at 12:08 AM Greg Harris
> > > > <greg.harris@aiven.io.invalid
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hey Snehashis,
> > > > > > >
> > > > > > > Sorry for the late reply, and thanks for helping close out the
> > > > > > discussion.
> > > > > > >
> > > > > > > > Note that if my assumptions are correct then
> > > > > > > > this can happen with the existing framework as well, or is
> > there
> > > > some
> > > > > > > > safeguard from this happening?
> > > > > > >
> > > > > > > Currently, if a cluster has a heterogeneous plugin
> installation,
> > > each
> > > > > > > worker may have a different definition of "latest". If you call
> > to
> > > > > > > validate a configuration, it will validate against whatever
> > latest
> > > > > > version
> > > > > > > is present locally.
> > > > > > > With this KIP, that behavior will continue when the version
> isn't
> > > > > > > specified, when the version is a soft requirement, or when the
> > > > version
> > > > > > is a
> > > > > > > range. For users that do not want to tolerate their tasks
> having
> > > > > > > heterogeneous versions, the single-version hard requirements
> are
> > > > there,
> > > > > > but
> > > > > > > come at the cost of micromanagement of upgrades.
> > > > > > > Perhaps there is some room in the future for a connector
> > configured
> > > > > with
> > > > > > a
> > > > > > > range to "pin" the version that all of its tasks should use. Or
> > > > > > > version-aware-scheduling can schedule connectors and tasks to
> > > workers
> > > > > > that
> > > > > > > will resolve the same versions. The important primitive is
> being
> > > able
> > > > > to
> > > > > > > enforce which version is being used, everything else looks more
> > > like
> > > > a
> > > > > > > convenience feature to me.
> > > > > > >
> > > > > > > > So far, we could have pointed to the
> > > > > > > > misconfigured cluster configuration and somewhat differ this
> > > > problem
> > > > > to
> > > > > > > > something outside of connect runtime.
> > > > > > >
> > > > > > > I think this will still be the case. Even with this feature, we
> > > will
> > > > > > still
> > > > > > > recommend homogeneous clusters that have the same set of
> plugins
> > > > > > installed
> > > > > > > on every worker. Version-aware scheduling is a rejected
> > > alternative,
> > > > > and
> > > > > > > your example for performing validation requests on an arbitrary
> > > > worker
> > > > > > is a
> > > > > > > great example of a complication for heterogeneous clusters that
> > > we're
> > > > > not
> > > > > > > ready to solve.
> > > > > > >
> > > > > > > > We can introduce
> > > > > > > > a new path under this for version
> > > > > (/connector/connector-name/version),
> > > > > > > but
> > > > > > > > perhaps adding this as part of the status is a valid
> > alternative.
> > > > > > >
> > > > > > > I think either is fine. Adding a new field to the existing
> > > endpoints
> > > > is
> > > > > > > fine, and shouldn't require special compatibility such as a
> query
> > > > > > > parameter.
> > > > > > >
> > > > > > > > Also, to go further I think
> > > > > > > > version information for tasks could also be available,
> > > > > > >
> > > > > > > I completely agree. The version of each connector and each task
> > > > should
> > > > > be
> > > > > > > visible individually, since they may be different during a
> > rolling
> > > > > > upgrade.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Greg
> > > > > > >
> > > > > > > On Tue, Jun 18, 2024 at 5:18 AM Snehashis <
> > > snehashisp1995@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Greg, Chris
> > > > > > > >
> > > > > > > > Thanks for the in-depth discussion, I have a couple of
> > discussion
> > > > > > points
> > > > > > > > and would like your thoughts on this.
> > > > > > > >
> > > > > > > > 1) One concern I have with the new addition of 'soft' and
> > 'hard'
> > > > > > version
> > > > > > > > requirements is that there could be a mismatch in the plugin
> > > > version
> > > > > > that
> > > > > > > > two different tasks are running, if a soft requirement is
> > > provided
> > > > > and
> > > > > > > the
> > > > > > > > nodes a multi cluster deployment are not in sync w.r.t the
> > plugin
> > > > > > > versions
> > > > > > > > that they are configured with. Note that if my assumptions
> are
> > > > > correct
> > > > > > > then
> > > > > > > > this can happen with the existing framework as well, or is
> > there
> > > > some
> > > > > > > > safeguard from this happening? So far, we could have pointed
> to
> > > the
> > > > > > > > misconfigured cluster configuration and somewhat differ this
> > > > problem
> > > > > to
> > > > > > > > something outside of connect runtime. With this feature in
> > place
> > > > > > perhaps
> > > > > > > > the expectation is more on connect to not be running with
> such
> > > > > > > > inconsistency, especially if a connector version is
> specified.
> > > This
> > > > > is
> > > > > > > also
> > > > > > > > a problem with validation if different cluster have different
> > > > > > > > configurations, as IIRC validations are local to the worker
> > which
> > > > > > > receives
> > > > > > > > the rest call for validate. So, we might be validating with a
> > > > certain
> > > > > > > > version which is different from the one that will be used to
> > > create
> > > > > > > > connector and tasks. Again, this is likely how the current
> > state
> > > > is,
> > > > > > but
> > > > > > > > perhaps such inconsistencies warrant a deeper look with the
> > > > addition
> > > > > of
> > > > > > > > this feature. The problems associated with them can be
> somewhat
> > > > > > insidious
> > > > > > > > and hard to diagnose.
> > > > > > > >
> > > > > > > > 2) There was some discussion on the need for a new REST
> > endpoint
> > > to
> > > > > > > provide
> > > > > > > > information on the versions of running connectors, and I
> think
> > > > adding
> > > > > > > this
> > > > > > > > information via REST is a valuable addition. The way I see it
> > the
> > > > > > version
> > > > > > > > is an intrinsic property of an instance of a running
> connector
> > > and
> > > > > > hence
> > > > > > > > this should be part of the set of APIs under
> > > > > > /connector/<connector-name>
> > > > > > > > (also the /connectors API should also have this information
> as
> > it
> > > > is
> > > > > an
> > > > > > > > amalgamation of all the individual connector information). We
> > can
> > > > > > > introduce
> > > > > > > > a new path under this for version
> > > > > (/connector/connector-name/version),
> > > > > > > but
> > > > > > > > perhaps adding this as part of the status is a valid
> > alternative.
> > > > > This
> > > > > > is
> > > > > > > > mentioned as a rejected alternative right now. Also, to go
> > > further
> > > > I
> > > > > > > think
> > > > > > > > version information for tasks could also be available,
> > especially
> > > > if
> > > > > we
> > > > > > > > choose to not address the pitfalls discussed in my point 1),
> > this
> > > > > will
> > > > > > > > at-least provide admins a quick and easy way to determine if
> > such
> > > > and
> > > > > > > > inconsistent state exist in any of the connectors.
> > > > > > > >
> > > > > > > > Thanks again for reviving my original KIP and working to
> > improve
> > > > it.
> > > > > > > > Looking forward to your thoughts on the points mentioned
> above.
> > > > > > > > Regards
> > > > > > > > Snehashis
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, May 29, 2024 at 9:59 PM Chris Egerton
> > > > > <chrise@aiven.io.invalid
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Greg,
> > > > > > > > >
> > > > > > > > > First, an apology! I mistakenly assumed that each plugin
> > > appeared
> > > > > > only
> > > > > > > > once
> > > > > > > > > in the responses from GET
> > > > /connector-plugins?connectorsOnly=false.
> > > > > > > Thank
> > > > > > > > > you for correcting me and pointing out that all versions of
> > > each
> > > > > > plugin
> > > > > > > > > appear in that response, which does indeed satisfy my
> desire
> > > for
> > > > > > users
> > > > > > > to
> > > > > > > > > discover this information in at most two REST requests (and
> > in
> > > > > fact,
> > > > > > > does
> > > > > > > > > it in only one)!
> > > > > > > > >
> > > > > > > > > And secondly, with the revelation about recommenders, I
> agree
> > > > that
> > > > > > it's
> > > > > > > > > best to leave the "version" property out of the lists of
> > > > properties
> > > > > > > > > returned from the GET /connector-plugins/<plugin>/config
> > > > endpoint.
> > > > > > > > >
> > > > > > > > > With those two points settled, I think the only unresolved
> > item
> > > > is
> > > > > > the
> > > > > > > > > small change to version parsing added to the KIP (where raw
> > > > version
> > > > > > > > numbers
> > > > > > > > > are treated as an exact match, instead of a best-effort
> match
> > > > with
> > > > > a
> > > > > > > > > fallback on the default version). If the KIP is updated
> with
> > > that
> > > > > > then
> > > > > > > > I'd
> > > > > > > > > be ready to vote on it.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Wed, May 29, 2024 at 12:00 PM Greg Harris
> > > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hey Chris,
> > > > > > > > > >
> > > > > > > > > > Thanks for your thoughts.
> > > > > > > > > >
> > > > > > > > > > > Won't it still only expose the
> > > > > > > > > > > latest version for each plugin, instead of the range of
> > > > > versions
> > > > > > > > > > available?
> > > > > > > > > >
> > > > > > > > > > Here is a snippet of the current output of the GET
> > > > > > > > > > /connector-plugins?connectorsOnly=false endpoint, after I
> > > > > installed
> > > > > > > two
> > > > > > > > > > versions of the debezium PostgresConnector:
> > > > > > > > > >
> > > > > > > > > > {
> > > > > > > > > > "class":
> > > > > "io.debezium.connector.postgresql.PostgresConnector",
> > > > > > > > > > "type": "source",
> > > > > > > > > > "version": "2.0.1.Final"
> > > > > > > > > > },
> > > > > > > > > > {
> > > > > > > > > > "class":
> > > > > "io.debezium.connector.postgresql.PostgresConnector",
> > > > > > > > > > "type": "source",
> > > > > > > > > > "version": "2.6.1.Final"
> > > > > > > > > > },
> > > > > > > > > >
> > > > > > > > > > I think this satisfies your requirement to learn about
> all
> > > > > plugins
> > > > > > > and
> > > > > > > > > all
> > > > > > > > > > versions in two or fewer REST calls.
> > > > > > > > > >
> > > > > > > > > > I tried to get an example of the output of `/config` by
> > > > > hardcoding
> > > > > > > the
> > > > > > > > > > Recommender, and realized that Recommenders aren't
> executed
> > > on
> > > > > the
> > > > > > > > > > `/config` endpoint at all: only during validation, when a
> > > > > > > configuration
> > > > > > > > > is
> > > > > > > > > > actually present.
> > > > > > > > > > And this led me to discover that the `/config` endpoint
> > > > returns a
> > > > > > > > > > List<ConfigKeyInfo>, and ConfigKeyInfo does not contain a
> > > > > > > > > recommendedValues
> > > > > > > > > > field. The ConfigValue field is the object which contains
> > > > > > > > > > recommendedValues, and it is only generated during
> > > validation.
> > > > > > > > > > I think it's out of scope to start calling recommenders
> on
> > > > empty
> > > > > > > > configs
> > > > > > > > > > that might throw exceptions, changing the existing REST
> > > > entities,
> > > > > > or
> > > > > > > > > > changing the core ConfigDef implementation.
> > > > > > > > > > Someone could add this functionality later, I don't think
> > > it's
> > > > > > > > necessary
> > > > > > > > > > here.
> > > > > > > > > >
> > > > > > > > > > Then the question is: should "version" without
> recommenders
> > > > > appear
> > > > > > in
> > > > > > > > > > non-connector plugins? I think I'd rather be consistent
> > with
> > > > > > > > "predicate"
> > > > > > > > > > and "negate" on release, and let a later improvement add
> > > them.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Greg
> > > > > > > > > >
> > > > > > > > > > On Wed, May 29, 2024 at 8:06 AM Chris Egerton
> > > > > > > <chrise@aiven.io.invalid
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Greg,
> > > > > > > > > > >
> > > > > > > > > > > I'm confused about the behavior for GET
> > > > > > > > > > > /connector-plugins?connectorsOnly=false. Won't it still
> > > only
> > > > > > expose
> > > > > > > > the
> > > > > > > > > > > latest version for each plugin, instead of the range of
> > > > > versions
> > > > > > > > > > available?
> > > > > > > > > > >
> > > > > > > > > > > I'm hoping we can provide a flow where people need at
> > most
> > > > two
> > > > > > REST
> > > > > > > > > calls
> > > > > > > > > > > to discover 1) the complete set of plugins available on
> > the
> > > > > > worker
> > > > > > > > > (which
> > > > > > > > > > > is already possible with the endpoint under discussion)
> > and
> > > > 2)
> > > > > > the
> > > > > > > > set
> > > > > > > > > of
> > > > > > > > > > > versions available for a specific plugin on the worker
> > > (which
> > > > > > > doesn't
> > > > > > > > > > > appear to be possible, at least for some plugin types).
> > > This
> > > > > > > wouldn't
> > > > > > > > > > > require any out-of-band knowledge and would be valuable
> > for
> > > > > > > connector
> > > > > > > > > > users
> > > > > > > > > > > (who may want to, for example, know what their options
> > are
> > > > when
> > > > > > > > > > considering
> > > > > > > > > > > a plugin upgrade) and cluster administrators (who could
> > use
> > > > it
> > > > > > as a
> > > > > > > > > > sanity
> > > > > > > > > > > check for the setup of their Kafka Connect clusters
> > without
> > > > > > having
> > > > > > > to
> > > > > > > > > > pore
> > > > > > > > > > > over log files).
> > > > > > > > > > >
> > > > > > > > > > > As far as modifying the content of the GET
> > > > > > > > > > > /connector-plugins/<plugin>/config endpoint to include
> a
> > > > > > "version"
> > > > > > > > > > property
> > > > > > > > > > > goes, I think your point about returning that property
> > from
> > > > > > > requests
> > > > > > > > to
> > > > > > > > > > > that endpoint that include a version query parameter is
> > > > > salient,
> > > > > > > but
> > > > > > > > it
> > > > > > > > > > > also unfortunately applies to all types of plugin. I
> > don't
> > > > > think
> > > > > > it
> > > > > > > > > > should
> > > > > > > > > > > completely disqualify that option. I also wasn't
> > imagining
> > > > > adding
> > > > > > > > > > anything
> > > > > > > > > > > besides that single property to that endpoint, so no
> new
> > > > > > > "predicate"
> > > > > > > > > > > property for SMTs, no new "negate" property for
> > predicates,
> > > > and
> > > > > > no
> > > > > > > > new
> > > > > > > > > > > "type" property for either. I'm not necessarily opposed
> > to
> > > > > adding
> > > > > > > > > those,
> > > > > > > > > > > but it can be done without being pulled into the scope
> of
> > > > this
> > > > > > KIP.
> > > > > > > > > > >
> > > > > > > > > > > So to summarize how I imagine people might use the REST
> > API
> > > > > after
> > > > > > > > this
> > > > > > > > > > KIP:
> > > > > > > > > > >
> > > > > > > > > > > 1. Discover the set of plugins on the worker via GET
> > > > > > > > > > > /connector-plugins?connectorsOnly=false
> > > > > > > > > > > 2. For any of those plugins, discover the set of
> > available
> > > > > > versions
> > > > > > > > by
> > > > > > > > > > > examining the recommended values for the "version"
> > property
> > > > > after
> > > > > > > > > hitting
> > > > > > > > > > > the GET GET /connector-plugins/<plugin>/config endpoint
> > > > > > > > > > >
> > > > > > > > > > > I don't think this is necessarily optimal since
> plucking
> > > the
> > > > > > > > "version"
> > > > > > > > > > > property out of the response from that endpoint might
> be
> > a
> > > > PITA
> > > > > > for
> > > > > > > > ad
> > > > > > > > > > hoc
> > > > > > > > > > > queries via, e.g., curl, but I think it'd be enough for
> > > this
> > > > > KIP.
> > > > > > > > Still
> > > > > > > > > > > open to other options though, and if I've misunderstood
> > > > > anything
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > proposal, please let me know!
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 22, 2024 at 4:05 PM Greg Harris
> > > > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hey Chris,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your comments, and I'm glad that it seems
> > like
> > > > > we're
> > > > > > > > > > > > aligning on the vision here.
> > > > > > > > > > > >
> > > > > > > > > > > > > An
> > > > > > > > > > > > > alternative could be to change existing behavior to
> > > fail
> > > > > fast
> > > > > > > on
> > > > > > > > > any
> > > > > > > > > > > > > invalid default converter configuration instead of
> > just
> > > > for
> > > > > > > > invalid
> > > > > > > > > > > > > versions
> > > > > > > > > > > >
> > > > > > > > > > > > I suppose if this is landing in 4.0, we have the
> > > > opportunity
> > > > > to
> > > > > > > > break
> > > > > > > > > > > > compatibility and strictly validate the worker class
> > > > configs,
> > > > > > and
> > > > > > > > > > > > could have a new consistent behavior instead of
> > > > inconsistent
> > > > > > but
> > > > > > > > > > > > backwards-compatible behavior.
> > > > > > > > > > > >
> > > > > > > > > > > > > no other part of the KIP requires this change.
> > > > > > > > > > > >
> > > > > > > > > > > > This is correct, and a compelling argument. I'm fine
> > > > leaving
> > > > > > the
> > > > > > > > > > > > strict worker validation off of this configuration to
> > > > > > potentially
> > > > > > > > be
> > > > > > > > > > > > added later. If a KIP was raised to perform strict
> > > > validation
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > worker config, it would include the version config,
> and
> > > can
> > > > > > > address
> > > > > > > > > > > > backwards compatibility for both configs together.
> > > > > > > > > > > >
> > > > > > > > > > > > > RE exposing the version property in the
> > > > > > > > > > > > /connector-plugins/<plugin>/config
> > > > > > > > > > > > > endpoint, the behavior is inconsistent across
> plugin
> > > > types.
> > > > > > > > > > > >
> > > > > > > > > > > > Yes it is, and that started in a bugfix, once people
> > > > started
> > > > > > > using
> > > > > > > > > > > > this endpoint:
> > > > > > https://issues.apache.org/jira/browse/KAFKA-14843
> > > > > > > > . I
> > > > > > > > > > > > wasn't planning on converging the behavior in this
> KIP,
> > > as
> > > > I
> > > > > > > > > > > > considered it out-of-scope, and was going to follow
> the
> > > > > current
> > > > > > > > > > > > behavior. To summarize:
> > > > > > > > > > > >
> > > > > > > > > > > > GET /connector-plugins/<connector>/config will emit:
> > > > > > > > > > > > * connector.class (already implemented)
> > > > > > > > > > > > * connector.version (new)
> > > > > > > > > > > > * key.converter (already implemented)
> > > > > > > > > > > > * key.converter.version (new)
> > > > > > > > > > > > * value.converter (already implemented)
> > > > > > > > > > > > * value.converter.version (new)
> > > > > > > > > > > > * header.converter (already implemented)
> > > > > > > > > > > > * header.converter.version (new)
> > > > > > > > > > > > But will NOT emit:
> > > > > > > > > > > > * transforms.<alias>.type
> > > > > > > > > > > > * transforms.<alias>.version
> > > > > > > > > > > > * transforms.<alias>.predicate
> > > > > > > > > > > > * predicates.<alias>.type
> > > > > > > > > > > > * predicates.<alias>.version
> > > > > > > > > > > > * predicates.<alias>.negate
> > > > > > > > > > > >
> > > > > > > > > > > > GET /connector-plugins/<converter>/config will NOT
> > emit:
> > > > > > > > > > > > * "" (there's not even a ".class" prefix!)
> > > > > > > > > > > > * version
> > > > > > > > > > > >
> > > > > > > > > > > > GET /connector-plugins/<transform>/config will NOT
> > emit:
> > > > > > > > > > > > * type
> > > > > > > > > > > > * version
> > > > > > > > > > > > * predicate
> > > > > > > > > > > >
> > > > > > > > > > > > GET /connector-plugins/<predicate>/config will NOT
> > emit:
> > > > > > > > > > > > * type
> > > > > > > > > > > > * version
> > > > > > > > > > > > * negate
> > > > > > > > > > > >
> > > > > > > > > > > > Do you want the converter, transform, and predicate
> > > > endpoints
> > > > > > > > > changed?
> > > > > > > > > > > > Do you want just "version", or do you want all of the
> > > > > prefixed
> > > > > > > > > configs
> > > > > > > > > > > > including "type", "predicate" and "negate"? How would
> > you
> > > > > want
> > > > > > to
> > > > > > > > > > > > handle the converters?
> > > > > > > > > > > > And when I say the configs are "not part of the
> plugin
> > > > config
> > > > > > > > itself"
> > > > > > > > > > > > I mean that saying that GET
> > > > > > > > > > > > /connector-plugins/Flatten$Key/config?version=3.8.0
> > has a
> > > > > > > "version"
> > > > > > > > > > > > config that must be "3.8.0" is a little bit nonsense,
> > as
> > > > the
> > > > > > > > version
> > > > > > > > > > > > is already specified.
> > > > > > > > > > > >
> > > > > > > > > > > > > IMO
> > > > > > > > > > > > > it's worth including this information somewhere
> > > directly
> > > > > > > > accessible
> > > > > > > > > > > > without
> > > > > > > > > > > > > having to provide a full connector config. FWIW I'd
> > be
> > > > fine
> > > > > > > with
> > > > > > > > > GET
> > > > > > > > > > > > > /connector-plugins/<plugin>/versions as a
> first-class
> > > > > > endpoint
> > > > > > > > > > > >
> > > > > > > > > > > > You don't have to provide a configuration to call GET
> > > > > > > > > > > > /connector-plugins?connectorsOnly=false , is that
> > > endpoint
> > > > > not
> > > > > > > > close
> > > > > > > > > > > > enough to what you have in mind? See also the
> Rejected
> > > > > > > Alternative
> > > > > > > > > > > > "Adding new REST API endpoints"
> > > > > > > > > > > >
> > > > > > > > > > > > If you're calling
> > /connector-plugins/<transform>/config,
> > > > you
> > > > > > know
> > > > > > > > the
> > > > > > > > > > > > name of a plugin right? That either comes from
> > > out-of-band
> > > > > > > > knowledge,
> > > > > > > > > > > > validating a connector config, or calling GET
> > > > > > > > > > > > /connector-plugins?connectorsOnly=false.
> > > > > > > > > > > > * If you have out-of-band knowledge of plugin
> classes,
> > > > > perhaps
> > > > > > > you
> > > > > > > > > > > > have out-of-band knowledge of versions too.
> > > > > > > > > > > > * If you've just validated a connector config, there
> > > should
> > > > > be
> > > > > > an
> > > > > > > > > > > > accompanying "version" field there with an accurate
> > > default
> > > > > > value
> > > > > > > > and
> > > > > > > > > > > > recommenders.
> > > > > > > > > > > > * If you've called GET
> > > > > /connector-plugins?connectorsOnly=false,
> > > > > > > > that
> > > > > > > > > > > > endpoint includes version information.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Greg
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, May 22, 2024 at 11:05 AM Chris Egerton
> > > > > > > > > <chrise@aiven.io.invalid
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hope you had a nice weekend! Gonna try to keep
> things
> > > > > > concise.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Concluded points:
> > > > > > > > > > > > >
> > > > > > > > > > > > > RE version recommenders, I agree it's likely that
> > > > > > programmatic
> > > > > > > > UIs
> > > > > > > > > > will
> > > > > > > > > > > > > already be able to handle dynamic configuration
> > > > > definitions,
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > > > detail
> > > > > > > > > > > > > about SMTs is a great point. I still anticipate
> some
> > > > > > > awkwardness
> > > > > > > > > with
> > > > > > > > > > > > > connector versions, though: if the latest version
> > > > supports
> > > > > > some
> > > > > > > > new
> > > > > > > > > > > > > properties, then a user switches to an earlier
> > > version, a
> > > > > UI
> > > > > > > may
> > > > > > > > > > > respond
> > > > > > > > > > > > by
> > > > > > > > > > > > > wiping values for these properties. I guess we can
> > bite
> > > > the
> > > > > > > > bullet,
> > > > > > > > > > > > though.
> > > > > > > > > > > > >
> > > > > > > > > > > > > RE double-dinging during preflight validation for
> > > invalid
> > > > > > > > > versions, I
> > > > > > > > > > > > like
> > > > > > > > > > > > > the analogy with login credentials. I'm convinced
> > that
> > > > the
> > > > > > > > proposal
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > KIP is best 👍
> > > > > > > > > > > > >
> > > > > > > > > > > > > Continued points:
> > > > > > > > > > > > >
> > > > > > > > > > > > > RE failing on worker startup, sorry, I should be
> > > clearer:
> > > > > > there
> > > > > > > > is
> > > > > > > > > no
> > > > > > > > > > > > _new_
> > > > > > > > > > > > > justification for it that doesn't also apply to
> > > existing
> > > > > > > > behavior.
> > > > > > > > > We
> > > > > > > > > > > > > shouldn't diverge from existing behavior solely for
> > > this
> > > > > new
> > > > > > > > case.
> > > > > > > > > An
> > > > > > > > > > > > > alternative could be to change existing behavior to
> > > fail
> > > > > fast
> > > > > > > on
> > > > > > > > > any
> > > > > > > > > > > > > invalid default converter configuration instead of
> > just
> > > > for
> > > > > > > > invalid
> > > > > > > > > > > > > versions, but I'd vote to just stick to existing
> > > behavior
> > > > > and
> > > > > > > not
> > > > > > > > > > > > > complicate things, especially since no other part
> of
> > > the
> > > > > KIP
> > > > > > > > > requires
> > > > > > > > > > > > this
> > > > > > > > > > > > > change.
> > > > > > > > > > > > >
> > > > > > > > > > > > > RE exposing the version property in the
> > > > > > > > > > > > /connector-plugins/<plugin>/config
> > > > > > > > > > > > > endpoint, the behavior is inconsistent across
> plugin
> > > > types.
> > > > > > > > Hitting
> > > > > > > > > > the
> > > > > > > > > > > > > endpoint for the FileStreamSinkConnector on version
> > > 3.7.0
> > > > > > > yields
> > > > > > > > a
> > > > > > > > > > > > response
> > > > > > > > > > > > > that includes, among other things, the "topics",
> > > > > > > "topics.regex",
> > > > > > > > > and
> > > > > > > > > > > > > "errors.tolerance" properties. I see that we don't
> do
> > > > this
> > > > > > > > > everywhere
> > > > > > > > > > > > (the
> > > > > > > > > > > > > examples you cite for SMT and converter properties
> > are
> > > > > > > accurate),
> > > > > > > > > but
> > > > > > > > > > > IMO
> > > > > > > > > > > > > it's worth including this information somewhere
> > > directly
> > > > > > > > accessible
> > > > > > > > > > > > without
> > > > > > > > > > > > > having to provide a full connector config. FWIW I'd
> > be
> > > > fine
> > > > > > > with
> > > > > > > > > GET
> > > > > > > > > > > > > /connector-plugins/<plugin>/versions as a
> first-class
> > > > > > endpoint
> > > > > > > > > either
> > > > > > > > > > > > > instead of or in addition to adding recommended
> > values
> > > > for
> > > > > > all
> > > > > > > > > plugin
> > > > > > > > > > > > > versions.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your continued work on this KIP, and
> with
> > > the
> > > > > > > progress
> > > > > > > > > > we're
> > > > > > > > > > > > > making I'm optimistic about its chances of
> appearing
> > in
> > > > > > 4.0.0.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Chris
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, May 15, 2024 at 1:22 PM Greg Harris
> > > > > > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hey Chris,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for your quick follow up.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > But this risk is already present with
> > > > > > > > > > > > > > > existing error cases, and I don't see anything
> > that
> > > > > > > justifies
> > > > > > > > > > > > changing
> > > > > > > > > > > > > > > existing behavior with an invalid converter
> > class,
> > > or
> > > > > > > > diverging
> > > > > > > > > > > from
> > > > > > > > > > > > it
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the case of invalid converter versions.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The justification is to fail-fast, and prevent
> REST
> > > API
> > > > > > users
> > > > > > > > > from
> > > > > > > > > > > > > > receiving errors from bad configs that they
> didn't
> > > > write,
> > > > > > or
> > > > > > > > > maybe
> > > > > > > > > > > > > > don't even know apply to them.
> > > > > > > > > > > > > > Up until recently errors in these configurations
> > > > surfaced
> > > > > > as
> > > > > > > > > > failures
> > > > > > > > > > > > > > to create the connector, or failures to start,
> and
> > > you
> > > > > made
> > > > > > > > them
> > > > > > > > > > > > > > fail-fast during validation. I think this change
> is
> > > in
> > > > > the
> > > > > > > same
> > > > > > > > > > > > > > spirit, pulling the validation further forward
> and
> > > not
> > > > > > > letting
> > > > > > > > > > errors
> > > > > > > > > > > > > > lie dormant.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > And to call back to your original concern about
> > > > > > interrupting
> > > > > > > > > > > > > > connectors that explicitly provide these
> > > configurations
> > > > > and
> > > > > > > > don't
> > > > > > > > > > use
> > > > > > > > > > > > > > the worker configs: I expect that operators with
> a
> > > > > majority
> > > > > > > of
> > > > > > > > > > these
> > > > > > > > > > > > > > sorts of clients aren't going to be setting the
> > > worker
> > > > > > > .version
> > > > > > > > > > > > > > properties, because it would have no effect on
> the
> > > > > majority
> > > > > > > of
> > > > > > > > > > their
> > > > > > > > > > > > > > connectors. They would be able to rely on
> > > > > > > > backwards-compatibility
> > > > > > > > > > and
> > > > > > > > > > > > > > continue to ignore the class properties.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > CLI
> > > > > > > > > > > > > > > and programmatic UI developers will want to
> > develop
> > > > > their
> > > > > > > own
> > > > > > > > > > > tooling
> > > > > > > > > > > > > > > layers.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is a very compelling argument. Snehashis do
> > you
> > > > want
> > > > > > to
> > > > > > > > > figure
> > > > > > > > > > > > > > out a REST API design for this use-case?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regarding the GET
> > > /connector-plugins/<plugin>/config
> > > > > > > > endpoint,
> > > > > > > > > I
> > > > > > > > > > > was
> > > > > > > > > > > > > > > thinking about the response for non-connector
> > > > plugins,
> > > > > > > e.g.,
> > > > > > > > > > > > > > > GET /connector-plugins/RegexRouter/config.
> Would
> > a
> > > > > > > "version"
> > > > > > > > > > > property
> > > > > > > > > > > > > > > appear with recommended values?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I intended for the ".version" property to be like
> > > other
> > > > > > > > framework
> > > > > > > > > > > > > > configs (".class", ".type", ".predicate",
> > ".negate")
> > > > > where
> > > > > > > they
> > > > > > > > > are
> > > > > > > > > > > > > > inside the plugin namespace, but not part of the
> > > plugin
> > > > > > > config
> > > > > > > > > > > itself.
> > > > > > > > > > > > > > Perhaps we can deviate from those configs because
> > it
> > > > > would
> > > > > > > aid
> > > > > > > > in
> > > > > > > > > > > > > > discovering other valid `GET
> > > > > > > > > > > > > > /connector-plugins/<plugin>/config?version=`
> calls,
> > > > > without
> > > > > > > > > calling
> > > > > > > > > > > > > > `GET /connector-plugins?connectorsOnly=false`.
> > > > > > > > > > > > > > I don't really feel strongly either way.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > if a user changes the
> > > > > > > > > > > > > > > connector version in, e.g., a dropdown menu,
> then
> > > the
> > > > > UI
> > > > > > > > either
> > > > > > > > > > has
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > re-fetch the ConfigDef for the new version, or
> > risk
> > > > > > > operating
> > > > > > > > > on
> > > > > > > > > > > > stale
> > > > > > > > > > > > > > > information
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is a really interesting situation, thanks
> for
> > > > > finding
> > > > > > > > that!
> > > > > > > > > > This
> > > > > > > > > > > > > > is already a footgun with transformations and
> > > > predicates;
> > > > > > > Once
> > > > > > > > > you
> > > > > > > > > > > > > > fill out a transformation class (which includes a
> > > > > > > recommender)
> > > > > > > > > you
> > > > > > > > > > > > > > will then see all of the transformations
> > > > configurations.
> > > > > > > > > > > > > > I think this means that UI developers should have
> > > > already
> > > > > > > > > developed
> > > > > > > > > > > > > > the infrastructure for handling dynamic
> > recommenders,
> > > > or
> > > > > > else
> > > > > > > > > > they've
> > > > > > > > > > > > > > had this bug since KIP-66 in 2017. It may require
> > > some
> > > > > > manual
> > > > > > > > > > > > > > attention to roll-out support for the ".version"
> > > > > properties
> > > > > > > > > > > > > > specifically, but I don't think that should
> prevent
> > > us
> > > > > from
> > > > > > > > > > providing
> > > > > > > > > > > > > > this information in a natural place.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > but will it be possible to configure
> > > > > > > > > > > > > > > a connector to use two instances of the same
> > > > transform
> > > > > or
> > > > > > > > > > > predicate,
> > > > > > > > > > > > but
> > > > > > > > > > > > > > > with different versions for each?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yes, and this is included in the example in the
> > KIP,
> > > > > > > > > > > > > > `transforms.flatten-latest` and
> > > > `transforms.flatten-old`.
> > > > > > > This
> > > > > > > > > will
> > > > > > > > > > > > > > work in the natural way, with the two instances
> > > having
> > > > a
> > > > > > > > > different
> > > > > > > > > > > > > > version if configured for that. If a user wants
> to
> > > pin
> > > > > the
> > > > > > > same
> > > > > > > > > > > > > > version of a plugin for every instance, they will
> > > need
> > > > to
> > > > > > > > provide
> > > > > > > > > > the
> > > > > > > > > > > > > > version config multiple times and keep them
> in-sync
> > > > > > > themselves.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I would have expected the error
> > > > > > > > > > > > > > > to only be attributed to the version property,
> > and
> > > > for
> > > > > > the
> > > > > > > > > class
> > > > > > > > > > > > property
> > > > > > > > > > > > > > > to be reported as valid.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I considered this, and then went back and changed
> > it.
> > > > To
> > > > > > > quote
> > > > > > > > > > > someone
> > > > > > > > > > > > > > else, this is to catch "misspelling cat as dog".
> > For
> > > > > > example
> > > > > > > a
> > > > > > > > > user
> > > > > > > > > > > > > > types DogConverter and meant to type
> CatConverter,
> > > and
> > > > > then
> > > > > > > > > types a
> > > > > > > > > > > > > > version which is valid for CatConverter,
> > conceptually
> > > > the
> > > > > > > error
> > > > > > > > > is
> > > > > > > > > > in
> > > > > > > > > > > > > > the class name and not the version.
> > > > > > > > > > > > > > That's a very contrived scenario, but I think
> > similar
> > > > > > > arguments
> > > > > > > > > are
> > > > > > > > > > > > > > used for attributing validation errors to
> > > > > > > > > endpoints/urls/hostnames
> > > > > > > > > > > > > > when the associated credentials are unable to log
> > > into
> > > > > the
> > > > > > > > remote
> > > > > > > > > > > > > > system. Did the user provide the correct testing
> > > > > > credentials,
> > > > > > > > but
> > > > > > > > > > > > > > accidentally typed the production endpoint? Even
> if
> > > the
> > > > > > > > > production
> > > > > > > > > > > > > > endpoint looks valid (it's a real hostname that
> is
> > > > > > reachable)
> > > > > > > > the
> > > > > > > > > > > > > > conceptual error is still in the hostname and
> > should
> > > > have
> > > > > > the
> > > > > > > > > error
> > > > > > > > > > > > > > attributed to it to draw the user's attention.
> > > > > > > > > > > > > > If that's not convincing, I think the alternative
> > of
> > > > only
> > > > > > > > > > attributing
> > > > > > > > > > > > > > version errors to the version property is also
> > > > > acceptable.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Wed, May 15, 2024 at 8:06 AM Chris Egerton
> > > > > > > > > > > <chrise@aiven.io.invalid
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for your responses! Continuations of
> > > existing
> > > > > > > > > discussions:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regarding crashing the worker on startup--yes,
> > > there
> > > > is
> > > > > > > also
> > > > > > > > a
> > > > > > > > > > risk
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > allowing it to join the cluster. But this risk
> is
> > > > > already
> > > > > > > > > present
> > > > > > > > > > > > with
> > > > > > > > > > > > > > > existing error cases, and I don't see anything
> > that
> > > > > > > justifies
> > > > > > > > > > > > changing
> > > > > > > > > > > > > > > existing behavior with an invalid converter
> > class,
> > > or
> > > > > > > > diverging
> > > > > > > > > > > from
> > > > > > > > > > > > it
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the case of invalid converter versions. I think
> > we
> > > > > should
> > > > > > > > keep
> > > > > > > > > > this
> > > > > > > > > > > > > > simple
> > > > > > > > > > > > > > > and not do anything different for worker
> startup
> > > than
> > > > > we
> > > > > > do
> > > > > > > > > > > already.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As far as REST API vs. metrics go--I think
> you're
> > > > right
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > > > > original
> > > > > > > > > > > > > > > version metrics were added as a "monitoring"
> > > detail.
> > > > > > > However,
> > > > > > > > > > this
> > > > > > > > > > > > was
> > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > when plugin versions were managed solely by
> > cluster
> > > > > > > > > > administrators.
> > > > > > > > > > > > With
> > > > > > > > > > > > > > > this KIP, connector users will be able to
> manage
> > > > plugin
> > > > > > > > > versions,
> > > > > > > > > > > > and CLI
> > > > > > > > > > > > > > > and programmatic UI developers will want to
> > develop
> > > > > their
> > > > > > > own
> > > > > > > > > > > tooling
> > > > > > > > > > > > > > > layers. I think focusing on the REST API as the
> > > > primary
> > > > > > > > > interface
> > > > > > > > > > > for
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > KIP would be best for these users.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > (All that said, I don't object to the metrics
> > that
> > > > are
> > > > > > > > proposed
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > KIP;
> > > > > > > > > > > > > > > I just think they make more sense in addition
> to
> > > new
> > > > > REST
> > > > > > > API
> > > > > > > > > > > > > > > functionality, as opposed to instead of it.)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regarding the GET
> > > /connector-plugins/<plugin>/config
> > > > > > > > endpoint,
> > > > > > > > > I
> > > > > > > > > > > was
> > > > > > > > > > > > > > > thinking about the response for non-connector
> > > > plugins,
> > > > > > > e.g.,
> > > > > > > > > > > > > > > GET /connector-plugins/RegexRouter/config.
> Would
> > a
> > > > > > > "version"
> > > > > > > > > > > property
> > > > > > > > > > > > > > > appear with recommended values?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > And new thoughts:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1) Regarding the recommended values for
> > > > > > > "connector.version",
> > > > > > > > > this
> > > > > > > > > > > > might
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > confusing since there could be differences
> > between
> > > > the
> > > > > > > > > ConfigDefs
> > > > > > > > > > > > for the
> > > > > > > > > > > > > > > latest version of the connector and prior
> > versions.
> > > > It
> > > > > > also
> > > > > > > > > makes
> > > > > > > > > > > the
> > > > > > > > > > > > > > flow
> > > > > > > > > > > > > > > a bit awkward for programmatic UI developers:
> if
> > a
> > > > user
> > > > > > > > changes
> > > > > > > > > > the
> > > > > > > > > > > > > > > connector version in, e.g., a dropdown menu,
> then
> > > the
> > > > > UI
> > > > > > > > either
> > > > > > > > > > has
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > re-fetch the ConfigDef for the new version, or
> > risk
> > > > > > > operating
> > > > > > > > > on
> > > > > > > > > > > > stale
> > > > > > > > > > > > > > > information. I'm starting to doubt that
> exposing
> > > the
> > > > > > range
> > > > > > > of
> > > > > > > > > > > > available
> > > > > > > > > > > > > > > versions via recommended values is the best way
> > to
> > > > > > proceed,
> > > > > > > > > > instead
> > > > > > > > > > > > of a
> > > > > > > > > > > > > > > more explicit approach like GET
> > > > > > > > > > > > /connector-plugins/<plugin>/versions, or
> > > > > > > > > > > > > > > the "Adding new REST API endpoints" rejected
> > > > > alternative.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2) I know that this is a bit heinous, but will
> it
> > > be
> > > > > > > possible
> > > > > > > > > to
> > > > > > > > > > > > > > configure
> > > > > > > > > > > > > > > a connector to use two instances of the same
> > > > transform
> > > > > or
> > > > > > > > > > > predicate,
> > > > > > > > > > > > but
> > > > > > > > > > > > > > > with different versions for each? (I don't
> think
> > > this
> > > > > is
> > > > > > > > worth
> > > > > > > > > > > > > > significant
> > > > > > > > > > > > > > > design/implementation effort, so if it would
> > > inflate
> > > > > > either
> > > > > > > > of
> > > > > > > > > > > those,
> > > > > > > > > > > > > > > please don't feel obligated to support it.)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3) Just out of curiosity, why double-ding
> during
> > > > config
> > > > > > > > > > validation
> > > > > > > > > > > > if the
> > > > > > > > > > > > > > > version for a plugin class can't be found? I
> > would
> > > > have
> > > > > > > > > expected
> > > > > > > > > > > the
> > > > > > > > > > > > > > error
> > > > > > > > > > > > > > > to only be attributed to the version property,
> > and
> > > > for
> > > > > > the
> > > > > > > > > class
> > > > > > > > > > > > property
> > > > > > > > > > > > > > > to be reported as valid.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, May 14, 2024 at 1:11 PM Greg Harris
> > > > > > > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hey Chris,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for your comments. I'm glad you like
> the
> > > > > > > > motivations,
> > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > wrote that part!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > the configuration syntax for the most basic
> > use
> > > > > case
> > > > > > of
> > > > > > > > > > > > > > > > > specifying a single desired version is
> pretty
> > > > > > > > > > counterintuitive.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I agree, and the "soft" requirement scheme is
> > > > > > something I
> > > > > > > > > > wasn't
> > > > > > > > > > > > > > > > explicitly looking for, but would be
> inherited
> > > from
> > > > > the
> > > > > > > > > > library.
> > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > fine with eliminating the soft requirement
> > > > semantics,
> > > > > > and
> > > > > > > > > > having
> > > > > > > > > > > > > > > > "1.0.0" behave the same as "[1.0.0]". I'm
> less
> > > > > inclined
> > > > > > > to
> > > > > > > > > > > include
> > > > > > > > > > > > > > > > "range" in the property name, or have two
> > > > properties.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Failing startup is drastic and has the
> > > potential
> > > > to
> > > > > > > > disrupt
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > availability of connectors that would
> > otherwise
> > > > be
> > > > > > able
> > > > > > > > to
> > > > > > > > > > run
> > > > > > > > > > > > > > healthily
> > > > > > > > > > > > > > > > > because they were explicitly configured to
> > use
> > > > > valid
> > > > > > > > > > converters
> > > > > > > > > > > > > > instead
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > the worker defaults.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think this argument cuts both ways. If
> > someone
> > > > > > > > > reconfigures a
> > > > > > > > > > > > worker
> > > > > > > > > > > > > > > > and adds an invalid ".version" string to the
> > > worker
> > > > > (or
> > > > > > > > > changes
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > plugin.path to make it invalid), it would be
> > > > > permitted
> > > > > > to
> > > > > > > > > enter
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > group, and accept work assignments. If those
> > work
> > > > > > > > assignments
> > > > > > > > > > > used
> > > > > > > > > > > > > > > > these configurations, a set of tasks could
> > > > transition
> > > > > > to
> > > > > > > > > FAILED
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > not be able to recover, because they would be
> > > > > restarted
> > > > > > > > again
> > > > > > > > > > on
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > same worker.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Why are metrics utilized to report
> > information
> > > > > about
> > > > > > > > plugin
> > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > utilized by connectors at runtime instead
> of
> > > > > > publishing
> > > > > > > > > this
> > > > > > > > > > > > info in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > REST API
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I was following the existing pattern for
> > exposing
> > > > > > runtime
> > > > > > > > > > > versions
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > connectors, and it did seem like a
> "monitoring"
> > > > > > feature.
> > > > > > > If
> > > > > > > > > > that
> > > > > > > > > > > > > > > > approach is flawed and should be phased out,
> I
> > > > think
> > > > > it
> > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > > a
> > > > > > > > > > > > > > > > good idea to reconsider the REST API rejected
> > > > > > > alternative.
> > > > > > > > > > > > > > > > We would need some additional design work to
> > spec
> > > > out
> > > > > > the
> > > > > > > > > REST
> > > > > > > > > > > API
> > > > > > > > > > > > > > > > interface, as I don't have anything in mind
> > > > > currently.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'm unclear on whether or not it'll be
> > possible
> > > > to
> > > > > > see
> > > > > > > > this
> > > > > > > > > > > > > > information
> > > > > > > > > > > > > > > > via the
> > > > > > > > > > > > > > > > > GET /connector-plugins/<plugin>/config
> > endpoint
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There is room in the API to add recommenders
> > for
> > > > > > > > > > "key.converter",
> > > > > > > > > > > > > > > > "value.converter", and "header.converter",
> but
> > > not
> > > > > for
> > > > > > > > > > transforms
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > predicates, as they include aliases that
> depend
> > > on
> > > > an
> > > > > > > > actual
> > > > > > > > > > > > > > > > configuration. We could explicitly say we're
> > > going
> > > > to
> > > > > > do
> > > > > > > > > that,
> > > > > > > > > > or
> > > > > > > > > > > > do
> > > > > > > > > > > > > > > > whatever is convenient during the
> > implementation
> > > > > phase,
> > > > > > > or
> > > > > > > > > > leave
> > > > > > > > > > > it
> > > > > > > > > > > > > > > > open to be improved later.
> > > > > > > > > > > > > > > > There will not be any recommenders for
> > ".version"
> > > > > > > > properties
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > `/config` endpoint, because those
> recommenders
> > > are
> > > > > > > dynamic
> > > > > > > > > and
> > > > > > > > > > > > depend
> > > > > > > > > > > > > > > > on an actual configuration.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 5) There are two relevant lines in the KIP:
> > "If a
> > > > > > > .version
> > > > > > > > > > > property
> > > > > > > > > > > > > > > > contains a hard requirement, select the
> latest
> > > > > > installed
> > > > > > > > > > version
> > > > > > > > > > > > which
> > > > > > > > > > > > > > > > satisfies the requirement." and "This
> > > configuration
> > > > > is
> > > > > > > > > > > re-evaluated
> > > > > > > > > > > > > > > > each time the connector or task are assigned
> > to a
> > > > new
> > > > > > > > > worker".
> > > > > > > > > > I
> > > > > > > > > > > > would
> > > > > > > > > > > > > > > > call this "eager" upgrade behavior, rather
> > than a
> > > > > > > "sticky"
> > > > > > > > or
> > > > > > > > > > > > "lazy"
> > > > > > > > > > > > > > > > upgrade behavior.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 6) Updated!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, May 14, 2024 at 9:14 AM Chris Egerton
> > > > > > > > > > > > <chrise@aiven.io.invalid
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks Greg for updating the KIP, and
> thanks
> > > > > > Snehashis
> > > > > > > > for
> > > > > > > > > > > > starting
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > work on this originally.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The motivation section makes a pretty
> > > convincing
> > > > > case
> > > > > > > for
> > > > > > > > > > this
> > > > > > > > > > > > kind
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > feature. My thoughts are mostly about
> > specific
> > > > > > details:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1) I like the support for version ranges
> (the
> > > > > example
> > > > > > > > > > > > demonstrating
> > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > avoid KAFKA-10574 with the header converter
> > was
> > > > > > > > > particularly
> > > > > > > > > > > > > > > > entertaining),
> > > > > > > > > > > > > > > > > but the configuration syntax for the most
> > basic
> > > > use
> > > > > > > case
> > > > > > > > of
> > > > > > > > > > > > > > specifying a
> > > > > > > > > > > > > > > > > single desired version is pretty
> > > > counterintuitive.
> > > > > > > People
> > > > > > > > > may
> > > > > > > > > > > get
> > > > > > > > > > > > > > bitten
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > at least frustrated if they put
> > > > > > connector.version=3.8.0
> > > > > > > > in
> > > > > > > > > a
> > > > > > > > > > > > config
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > then version 3.7.5 ends up running. I'd
> like
> > it
> > > > if
> > > > > we
> > > > > > > > could
> > > > > > > > > > > > either
> > > > > > > > > > > > > > > > > intentionally deviate from Maven ranges
> when
> > a
> > > > bare
> > > > > > > > version
> > > > > > > > > > is
> > > > > > > > > > > > > > present,
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > separate things out into two properties:
> > > > > foo.version
> > > > > > > > would
> > > > > > > > > be
> > > > > > > > > > > the
> > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > accepted version for the foo plugin, and
> > > > > > > > foo.version.range
> > > > > > > > > > > would
> > > > > > > > > > > > use
> > > > > > > > > > > > > > > > Maven
> > > > > > > > > > > > > > > > > range syntax. Open to other options too,
> just
> > > > > > > providing a
> > > > > > > > > > > couple
> > > > > > > > > > > > to
> > > > > > > > > > > > > > get
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > ball rolling.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2) Although the current behavior for a
> worker
> > > > with
> > > > > an
> > > > > > > > > invalid
> > > > > > > > > > > > > > > > > key/value/header converter class specified
> in
> > > its
> > > > > > > config
> > > > > > > > > file
> > > > > > > > > > > is
> > > > > > > > > > > > a
> > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > strange (I was surprised to learn that it
> > > > wouldn't
> > > > > > fail
> > > > > > > > on
> > > > > > > > > > > > startup),
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > don't see a good reason to deviate from
> this
> > > when
> > > > > an
> > > > > > > > > invalid
> > > > > > > > > > > > version
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > specified. Failing startup is drastic and
> has
> > > the
> > > > > > > > potential
> > > > > > > > > > to
> > > > > > > > > > > > > > disrupt
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > availability of connectors that would
> > otherwise
> > > > be
> > > > > > able
> > > > > > > > to
> > > > > > > > > > run
> > > > > > > > > > > > > > healthily
> > > > > > > > > > > > > > > > > because they were explicitly configured to
> > use
> > > > > valid
> > > > > > > > > > converters
> > > > > > > > > > > > > > instead
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > the worker defaults.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 3) Why are metrics utilized to report
> > > information
> > > > > > about
> > > > > > > > > > plugin
> > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > utilized by connectors at runtime instead
> of
> > > > > > publishing
> > > > > > > > > this
> > > > > > > > > > > > info in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > REST API? I saw that this was mentioned as
> a
> > > > > rejected
> > > > > > > > > > > > alternative,
> > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > didn't get a sense of why. It seems like
> the
> > > REST
> > > > > API
> > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > > > > > easier to
> > > > > > > > > > > > > > > > > access and more intuitive for most users
> than
> > > new
> > > > > > > > metrics.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 4) In the "Validation" section it's stated
> > that
> > > > > > "Users
> > > > > > > > can
> > > > > > > > > > use
> > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > recommenders to discover the valid plugin
> > > classes
> > > > > and
> > > > > > > > > > versions,
> > > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > > requiring an earlier call to GET
> > > > > > > > > > > > > > > > /connector-plugins?connectorsOnly=false."
> > > > > > > > > > > > > > > > > I really like the creativity and simplicity
> > of
> > > > > > reusing
> > > > > > > > the
> > > > > > > > > > > > > > recommender
> > > > > > > > > > > > > > > > > mechanism to expose available versions for
> > > > plugins
> > > > > > via
> > > > > > > > the
> > > > > > > > > > REST
> > > > > > > > > > > > API.
> > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > unclear on whether or not it'll be possible
> > to
> > > > see
> > > > > > this
> > > > > > > > > > > > information
> > > > > > > > > > > > > > via
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > GET /connector-plugins/<plugin>/config
> > > endpoint,
> > > > > > > though.
> > > > > > > > > It'd
> > > > > > > > > > > be
> > > > > > > > > > > > > > great if
> > > > > > > > > > > > > > > > > this were supported, since we learned in
> > > KIP-769
> > > > > [1]
> > > > > > > that
> > > > > > > > > > > people
> > > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > > want to be able to see configuration
> options
> > > for
> > > > > > > > connectors
> > > > > > > > > > and
> > > > > > > > > > > > their
> > > > > > > > > > > > > > > > > plugins via some kind of GET endpoint
> without
> > > > > having
> > > > > > to
> > > > > > > > > > > provide a
> > > > > > > > > > > > > > > > complete
> > > > > > > > > > > > > > > > > connector config for validation.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 5) In the Maven version range docs, it's
> > stated
> > > > > that
> > > > > > > > "Maven
> > > > > > > > > > > > picks the
> > > > > > > > > > > > > > > > > highest version of each project that
> > satisfies
> > > > all
> > > > > > the
> > > > > > > > hard
> > > > > > > > > > > > > > requirements
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > the dependencies on that project." I'm
> > guessing
> > > > > this
> > > > > > > > > behavior
> > > > > > > > > > > > will be
> > > > > > > > > > > > > > > > > retained for Connect; i.e., the
> > > highest-possible
> > > > > > > version
> > > > > > > > of
> > > > > > > > > > > each
> > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > that satisfies the user-specified version
> > > > > constraints
> > > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > run?
> > > > > > > > > > > > > > (An
> > > > > > > > > > > > > > > > > alternative approach could be to have some
> > kind
> > > > of
> > > > > > > > "sticky"
> > > > > > > > > > > logic
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > restarts connectors/tasks when their
> > > > currently-used
> > > > > > > > version
> > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > incompatible with the configured
> > constraints.)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 6) (Nit) It'd be nice to add a link to the
> > > > > > TestPlugins
> > > > > > > > > class
> > > > > > > > > > or
> > > > > > > > > > > > > > somewhere
> > > > > > > > > > > > > > > > > in its neighborhood to the testing plan;
> > > > unfamiliar
> > > > > > > > readers
> > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > won't
> > > > > > > > > > > > > > > > > get much out of what's there right now.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > [1] -
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, May 13, 2024 at 2:01 PM Snehashis <
> > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > That is much appreciated. No complaints
> on
> > > the
> > > > > > > > additional
> > > > > > > > > > > > scope, I
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > make some time out to work on this once
> we
> > > have
> > > > > > > > approval.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 9:28 PM Greg
> Harris
> > > > > > > > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hey Snehashis,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I'm glad to hear you're still
> interested
> > in
> > > > > this
> > > > > > > KIP!
> > > > > > > > > > > > > > > > > > > I'm happy to let you drive this, and I
> > > > > apologize
> > > > > > > for
> > > > > > > > > > > > increasing
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > scope of work so drastically. To make
> up
> > > for
> > > > > > that,
> > > > > > > > I'll
> > > > > > > > > > > > > > volunteer to
> > > > > > > > > > > > > > > > > > > be the primary PR reviewer to help get
> > this
> > > > > done
> > > > > > > > > quickly
> > > > > > > > > > > once
> > > > > > > > > > > > > > the KIP
> > > > > > > > > > > > > > > > > > > is approved.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 3:51 AM
> > Snehashis <
> > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for the follow up to my
> original
> > > > KIP,
> > > > > I
> > > > > > am
> > > > > > > > in
> > > > > > > > > > > > favour of
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > additions made to expand its scope,
> the
> > > > > > addition
> > > > > > > of
> > > > > > > > > > range
> > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > specifically make a lot of sense.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Apologies if I have not publicly
> worked
> > > on
> > > > > this
> > > > > > > KIP
> > > > > > > > > > for a
> > > > > > > > > > > > long
> > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > original work was done when the move
> to
> > > > > service
> > > > > > > > > loading
> > > > > > > > > > > > was in
> > > > > > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > > > > > and I wanted to loop back to this
> only
> > > > after
> > > > > > that
> > > > > > > > > work
> > > > > > > > > > > was
> > > > > > > > > > > > > > > > completed.
> > > > > > > > > > > > > > > > > > > Post
> > > > > > > > > > > > > > > > > > > > its conclusion, I have not been able
> to
> > > > take
> > > > > > this
> > > > > > > > up
> > > > > > > > > > due
> > > > > > > > > > > to
> > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > > priorities. If it's okay with you, I
> > > would
> > > > > > still
> > > > > > > > like
> > > > > > > > > > to
> > > > > > > > > > > > get
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > implemented myself, including the
> > > > additional
> > > > > > > scope.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks and regards
> > > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 12:45 AM Greg
> > > > Harris
> > > > > > > > > > > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I'd like to reboot the discussion
> on
> > > > > KIP-891:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I've made some changes, most
> notably:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. Specifying versions for all
> > plugins
> > > in
> > > > > > > > Connector
> > > > > > > > > > > > configs
> > > > > > > > > > > > > > > > > > > > > (converters, header converters,
> > > > transforms,
> > > > > > and
> > > > > > > > > > > > predicates)
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > > > connectors & tasks
> > > > > > > > > > > > > > > > > > > > > 2. Specifying a range of versions
> > > instead
> > > > > of
> > > > > > an
> > > > > > > > > exact
> > > > > > > > > > > > match
> > > > > > > > > > > > > > > > > > > > > 3. New metrics to observe what
> > versions
> > > > are
> > > > > > > > in-use
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks to Snehashis for the
> original
> > > KIP
> > > > > > idea!
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Jan 2, 2024 at 11:49 AM
> Greg
> > > > > Harris <
> > > > > > > > > > > > > > > > greg.harris@aiven.io>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Snehashis,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thank you for the KIP! This is
> > > > something
> > > > > > I've
> > > > > > > > > > wanted
> > > > > > > > > > > > for a
> > > > > > > > > > > > > > long
> > > > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I know the discussion has gone
> > cold,
> > > > are
> > > > > > you
> > > > > > > > > still
> > > > > > > > > > > > > > interested
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > pursuing this feature? I'll make
> > time
> > > > to
> > > > > > > review
> > > > > > > > > the
> > > > > > > > > > > > KIP if
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > > > still accepting comments.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM
> > > > > Snehashis
> > > > > > <
> > > > > > > > > > > > > > > > > > snehashisp1995@gmail.com
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for the points Sagar.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1) Should we update the GET
> > > > > /connectors
> > > > > > > > > > endpoint
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > include the
> > > > > > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > > > > > the plugin that is running?
> It
> > > > could
> > > > > be
> > > > > > > > > useful
> > > > > > > > > > to
> > > > > > > > > > > > > > figure
> > > > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > the plugin or I am assuming
> it
> > > gets
> > > > > > > > returned
> > > > > > > > > by
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > expand=info
> > > > > > > > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think this is good to have
> and
> > > > > possible
> > > > > > > > > future
> > > > > > > > > > > > > > > > enhancement. The
> > > > > > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > > > > info will be present in the
> > config
> > > of
> > > > > the
> > > > > > > > > > connector
> > > > > > > > > > > > if
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > > > > > specified the version.
> Otherwise
> > it
> > > > is
> > > > > > the
> > > > > > > > > latest
> > > > > > > > > > > > version
> > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > > > > > can find out from the
> > > > connector-plugin
> > > > > > > > > endpoint.
> > > > > > > > > > > The
> > > > > > > > > > > > > > > > information
> > > > > > > > > > > > > > > > > > > can be
> > > > > > > > > > > > > > > > > > > > > > > introduced to the response of
> the
> > > GET
> > > > > > > > > /connectors
> > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > > itself,
> > > > > > > > > > > > > > > > > > > > > however
> > > > > > > > > > > > > > > > > > > > > > > the most ideal way of doing
> this
> > > > would
> > > > > be
> > > > > > > to
> > > > > > > > > get
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > > > > > > instance of the connector and
> get
> > > the
> > > > > > > version
> > > > > > > > > > > > directly
> > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > there.
> > > > > > > > > > > > > > > > > > > > > This is
> > > > > > > > > > > > > > > > > > > > > > > slightly tricky as the
> connector
> > > > could
> > > > > be
> > > > > > > > > running
> > > > > > > > > > > in
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > node.
> > > > > > > > > > > > > > > > > > > > > > > One way to do this would be to
> > > > persist
> > > > > > the
> > > > > > > > > > version
> > > > > > > > > > > > > > > > information in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > status backing store during
> > > > > instantiation
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > requires
> > > > > > > > > > > > > > > > > > > > > > > some more thought and since the
> > > > version
> > > > > > is
> > > > > > > > part
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > configs if
> > > > > > > > > > > > > > > > > > > > > provided
> > > > > > > > > > > > > > > > > > > > > > > and evident otherwise, I have
> not
> > > > > > included
> > > > > > > it
> > > > > > > > > in
> > > > > > > > > > > this
> > > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2) I am not aware of this and
> > > hence
> > > > > > > asking,
> > > > > > > > > > can 2
> > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > > > > versions have the same name?
> > Does
> > > > the
> > > > > > > > plugin
> > > > > > > > > > > > isolation
> > > > > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > > > > > could have a bearing when
> using
> > > the
> > > > > > > > lifecycle
> > > > > > > > > > > > > > endpoints for
> > > > > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > All connectors in a cluster
> need
> > to
> > > > > have
> > > > > > > > > uniquire
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > names
> > > > > > > > > > > > > > > > > > > > > > > regardless of what version of
> the
> > > > > plugin
> > > > > > > the
> > > > > > > > > > > > connector is
> > > > > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > > > > > > underneath. This is something
> > > > enforced
> > > > > by
> > > > > > > the
> > > > > > > > > > > connect
> > > > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > > > itself.
> > > > > > > > > > > > > > > > > > > > > All
> > > > > > > > > > > > > > > > > > > > > > > connect CRUD operations are
> keyed
> > > on
> > > > > the
> > > > > > > > > > connector
> > > > > > > > > > > > name
> > > > > > > > > > > > > > so
> > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > be an issue.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM
> > > > Sagar <
> > > > > > > > > > > > > > > > sagarmeansocean@gmail.com
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hey Snehashsih,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP. It looks
> > > like a
> > > > > > very
> > > > > > > > > useful
> > > > > > > > > > > > > > feature.
> > > > > > > > > > > > > > > > Couple
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > small-ish points, let me know
> > > what
> > > > > you
> > > > > > > > think:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1) Should we update the GET
> > > > > /connectors
> > > > > > > > > > endpoint
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > include the
> > > > > > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > > > > > the plugin that is running?
> It
> > > > could
> > > > > be
> > > > > > > > > useful
> > > > > > > > > > to
> > > > > > > > > > > > > > figure
> > > > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > > > > > the plugin or I am assuming
> it
> > > gets
> > > > > > > > returned
> > > > > > > > > by
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > expand=info
> > > > > > > > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > > > > > > > > 2) I am not aware of this and
> > > hence
> > > > > > > asking,
> > > > > > > > > > can 2
> > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > > > > versions have the same name?
> > Does
> > > > the
> > > > > > > > plugin
> > > > > > > > > > > > isolation
> > > > > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > > > > > could have a bearing when
> using
> > > the
> > > > > > > > lifecycle
> > > > > > > > > > > > > > endpoints for
> > > > > > > > > > > > > > > > > > > > > connectors like
> > > > > > > > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > > > > > > Sagar.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 2:10
> PM
> > > > > Ashwin
> > > > > > > > > > > > > > > > > > > <apankaj@confluent.io.invalid
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > IIUC (please correct me
> if
> > I
> > > am
> > > > > > wrong
> > > > > > > > > > here),
> > > > > > > > > > > > what
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > a versioning scheme for a
> > > > connector
> > > > > > > > config
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > > different versions of a
> > > connector
> > > > > > > > plugin).
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Sorry for not being more
> > > precise
> > > > in
> > > > > > my
> > > > > > > > > > wording
> > > > > > > > > > > > - I
> > > > > > > > > > > > > > meant
> > > > > > > > > > > > > > > > > > > > > registering
> > > > > > > > > > > > > > > > > > > > > > > > > versions of schema for
> > > connector
> > > > > > > config.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Let's take the example of a
> > > > > fictional
> > > > > > > > > > connector
> > > > > > > > > > > > which
> > > > > > > > > > > > > > > > uses a
> > > > > > > > > > > > > > > > > > > > > fictional
> > > > > > > > > > > > > > > > > > > > > > > > AWS
> > > > > > > > > > > > > > > > > > > > > > > > > service.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Fictional Connector Config
> > > schema
> > > > > > > > > version:2.0
> > > > > > > > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > > > > > > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Fictional Connector config
> > > schema
> > > > > > > > > version:3.0
> > > > > > > > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > > "iam_role": {
> > > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > > > > > > > "iam_role"
> > > > > > > > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > The connector which
> supports
> > > > > > Fictional
> > > > > > > > > config
> > > > > > > > > > > > schema
> > > > > > > > > > > > > > 2.0
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > validate
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > access key and secret key.
> > > > > > > > > > > > > > > > > > > > > > > > > Whereas a connector which
> > > > supports
> > > > > > > config
> > > > > > > > > > with
> > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > 3.0
> > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > > > > > > > > validate the IAM role.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This is the alternative
> > which I
> > > > > > wanted
> > > > > > > to
> > > > > > > > > > > > suggest.
> > > > > > > > > > > > > > Each
> > > > > > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > > register the schema
> versions
> > of
> > > > > > > connector
> > > > > > > > > > > config
> > > > > > > > > > > > > > which it
> > > > > > > > > > > > > > > > > > > supports.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > The plugin paths may be
> > > > optionally
> > > > > > > > > different
> > > > > > > > > > > > i.e we
> > > > > > > > > > > > > > > > don't
> > > > > > > > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > > > > > > > > mandatorily add a new
> plugin
> > > path
> > > > > to
> > > > > > > > > support
> > > > > > > > > > a
> > > > > > > > > > > > new
> > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > version.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at
> 12:47
> > > PM
> > > > > > > > Snehashis
> > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the input
> > Ashwin.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on
> > the
> > > > > > > rejected
> > > > > > > > > > > > > > alternatives ?
> > > > > > > > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > > > config is versioned and
> > > has a
> > > > > > > schema.
> > > > > > > > > > Then
> > > > > > > > > > > a
> > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > > > > > > > dependencies have not
> > > > changed)
> > > > > > can
> > > > > > > > > handle
> > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > IIUC (please correct me
> if
> > I
> > > am
> > > > > > wrong
> > > > > > > > > > here),
> > > > > > > > > > > > what
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > a versioning scheme for a
> > > > > connector
> > > > > > > > > config
> > > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > > > connector (and
> > > > > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > > > different versions of a
> > > > connector
> > > > > > > > > plugin).
> > > > > > > > > > > > That is
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > somewhat
> > > > > > > > > > > > > > > > > > > > > > > > tangential
> > > > > > > > > > > > > > > > > > > > > > > > > > problem. While it is
> > > > definitely a
> > > > > > > > useful
> > > > > > > > > > > > feature to
> > > > > > > > > > > > > > > > have,
> > > > > > > > > > > > > > > > > > > like a
> > > > > > > > > > > > > > > > > > > > > log to
> > > > > > > > > > > > > > > > > > > > > > > > > > check what changes were
> > made
> > > > over
> > > > > > > time
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > > easier to do rollbacks,
> it
> > is
> > > > not
> > > > > > the
> > > > > > > > > focus
> > > > > > > > > > > > here.
> > > > > > > > > > > > > > Here
> > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > version we
> > > > > > > > > > > > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > > > > > > > > > > to say what underlying
> > > version
> > > > of
> > > > > > the
> > > > > > > > > > plugin
> > > > > > > > > > > > > > should the
> > > > > > > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > > > of the connector use.
> > Perhaps
> > > > it
> > > > > is
> > > > > > > > > better
> > > > > > > > > > to
> > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > name of
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > parameter from
> > > > connector.version
> > > > > to
> > > > > > > > > > > > > > > > > > connector.plugin.version
> > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > > > plugin.version if it was
> > > > > confusing.
> > > > > > > > wdyt?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Any plans to
> support
> > > > > assisted
> > > > > > > > > > migration
> > > > > > > > > > > > e.g
> > > > > > > > > > > > > > if a
> > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > connector/config?migrate=latest",
> > > > > > > the
> > > > > > > > > > > latest
> > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > > > > > > > transform the existing
> > > config
> > > > > to
> > > > > > > the
> > > > > > > > > > newer
> > > > > > > > > > > > > > version.
> > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > > > > adding a method like
> > > "boolean
> > > > > > > > > > > migrate(Version
> > > > > > > > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This is an enhancement we
> > can
> > > > > think
> > > > > > > of
> > > > > > > > > > doing
> > > > > > > > > > > in
> > > > > > > > > > > > > > future.
> > > > > > > > > > > > > > > > > > > Users can
> > > > > > > > > > > > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > > > > > > > > > a PUT call with the
> updated
> > > > > config
> > > > > > > > which
> > > > > > > > > > has
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > > > > > number.
> > > > > > > > > > > > > > > > > > > > > > > > > > The assisted mode could
> be
> > > > handy
> > > > > as
> > > > > > > the
> > > > > > > > > > user
> > > > > > > > > > > > does
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > know the
> > > > > > > > > > > > > > > > > > > > > > > > > > config but beyond this it
> > > does
> > > > > not
> > > > > > > seem
> > > > > > > > > to
> > > > > > > > > > > > justify
> > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > > > existence.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at
> > 10:50
> > > > AM
> > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > This is a really useful
> > > > feature
> > > > > > and
> > > > > > > > > > thanks
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > initiating
> > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I had the following
> > > > questions -
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on
> > the
> > > > > > > rejected
> > > > > > > > > > > > > > alternatives ?
> > > > > > > > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > > > config is versioned and
> > > has a
> > > > > > > schema.
> > > > > > > > > > Then
> > > > > > > > > > > a
> > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > > > > > > > dependencies have not
> > > > changed)
> > > > > > can
> > > > > > > > > handle
> > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Any plans to support
> > > > > assisted
> > > > > > > > > > migration
> > > > > > > > > > > > e.g
> > > > > > > > > > > > > > if a
> > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > connector/config?migrate=latest",
> > > > > > > the
> > > > > > > > > > > latest
> > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > > > > > > > transform the existing
> > > config
> > > > > to
> > > > > > > the
> > > > > > > > > > newer
> > > > > > > > > > > > > > version.
> > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > > > > adding a method like
> > > "boolean
> > > > > > > > > > > migrate(Version
> > > > > > > > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 21, 2022 at
> > > 2:27
> > > > PM
> > > > > > > > > > Snehashis <
> > > > > > > > > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > I'd like to start a
> > > > > discussion
> > > > > > > > thread
> > > > > > > > > > on
> > > > > > > > > > > > > > KIP-891:
> > > > > > > > > > > > > > > > > > Running
> > > > > > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > The KIP aims to add
> the
> > > > > ability
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > connect
> > > > > > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > > > > > > > versions of a
> > connector.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Please take a look
> and
> > > let
> > > > me
> > > > > > > know
> > > > > > > > > what
> > > > > > > > > > > you
> > > > > > > > > > > > > > think.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Hey Snehashis,
I updated the KIP to remove some stale mentions of the soft version
requirements, and the crashing workers on startup. I also added more detail
to the REST API.
IMHO we're ready to move to voting, so please open the thread if you also
believe it is ready.
Thanks!
Greg
On Fri, Aug 23, 2024 at 7:21 AM Snehashis <sn...@gmail.com> wrote:
> Hi Greg,
>
> Thanks for the clarification!
>
> I agree with not breaking compatibility and opting to not fail worker
> startup by validating converter versions.
>
> Let's also go with the unclosed variation of the requirements as a hard
> requirement i.e. 3.8 instead of [3.8]. I have updated the KIP and added a
> line there highlighting this.
>
> Regards
> Snehashis
>
> On Wed, Aug 21, 2024 at 9:24 PM Greg Harris <gr...@aiven.io.invalid>
> wrote:
>
> > Hey Snehashis,
> >
> > Thanks for your reply!
> >
> > > Deviating
> > > it from the spec seems unnecessary if we document it accordingly,
> however
> > > It's probably less intuitive and can lead to confusion. I would just
> keep
> > > it as is but making it simpler is also fine.
> >
> > It sounds like you don't have a strong opinion on this, similar to me.
> > Chris had a more firm stance on this. I think that you're right that we
> > could document this, but it would still be the single biggest foot-gun of
> > the feature, and I think we would regret it later.
> >
> > > Also for converter versions
> > > specified as part of the worker configs I believe we concluded that
> this
> > > step need not be fatal during worker startup if the required version is
> > not
> > > found but LMK if otherwise.
> >
> > Re-reading our earlier discussion, I think Chris had a strong opinion
> that
> > we shouldn't fail on startup because it would be inconsistent. I made an
> > offhand comment that if this was released in 4.0, we could change it so
> > both invalid classes and invalid versions cause the worker to fail, which
> > would be inconsistent but backwards incompatible.
> > I think in the interest of not breaking compatibility needlessly and
> > keeping consistent behavior, we should ignore invalid versions in the
> > worker config.
> >
> > Thanks,
> > Greg
> >
> > On Wed, Aug 21, 2024 at 1:58 AM Snehashis <sn...@gmail.com>
> > wrote:
> >
> > > Hi Greg,
> > >
> > > No issues, I have been caught up in a few things myself.
> > >
> > > I have added the points we discussed. In addition, I have added config
> > > providers as part of the set of plugins which will not support
> > versioning,
> > > for the same reason as to why it is not supported in the other plugins
> > that
> > > are initiated on startup.
> > >
> > > On whether to deviate from maven versioning for hard requirements as
> > > discussed between you and Chris.Whether to simply simply specify
> > > connector.version=1.1.1 as a hard requirement instead of [1.1.1].
> > Deviating
> > > it from the spec seems unnecessary if we document it accordingly,
> however
> > > It's probably less intuitive and can lead to confusion. I would just
> keep
> > > it as is but making it simpler is also fine. Also for converter
> versions
> > > specified as part of the worker configs I believe we concluded that
> this
> > > step need not be fatal during worker startup if the required version is
> > not
> > > found but LMK if otherwise.
> > >
> > > Regards
> > > Snehashis
> > >
> > >
> > >
> > > On Mon, Aug 19, 2024 at 9:24 PM Greg Harris
> <greg.harris@aiven.io.invalid
> > >
> > > wrote:
> > >
> > > > Hi Snehashis,
> > > >
> > > > Sorry for the late reply.
> > > >
> > > > > Heterogeneous dependencies in a multi cluster deployment is highly
> > > > discouraged
> > > >
> > > > Right, this remains unchanged in this KIP.
> > > >
> > > > > Let's add the version information for both connector and tasks in
> the
> > > > connector status itself
> > > > > once we add these two additions to the KIP (LMK if you want me to
> > take
> > > > that up).
> > > >
> > > > Could you make these additions?
> > > >
> > > > I'm interested to see if we can include this in 4.0.
> > > >
> > > > Thanks,
> > > > Greg
> > > >
> > > > On Tue, Jul 2, 2024 at 2:52 AM Snehashis <sn...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Greg,
> > > > >
> > > > > Thanks for taking a look at this, to conclude on the two points
> > above.
> > > > >
> > > > > 1. I'm okay with the status quo of leaving the dependency
> management
> > of
> > > > > plugins to systems outside of the connect runtime as it is now.
> Given
> > > > that
> > > > > the dependencies are homogenous across a connect cluster, it should
> > > > ensure
> > > > > that task and connector versions are uniform across a deployment,
> > after
> > > > it
> > > > > has stabilised post an upgrade operation. Heterogeneous
> dependencies
> > > in a
> > > > > multi cluster deployment is highly discouraged and we should point
> > out
> > > > this
> > > > > can lead to inconsistent behaviour with connectors and even
> > > validations.
> > > > >
> > > > > 2. Let's add the version information for both connector and tasks
> in
> > > the
> > > > > connector status itself, I don't see another endpoint being
> required
> > > for
> > > > > this.
> > > > >
> > > > > I don't have any further points, so if you are okay we can put this
> > to
> > > a
> > > > > vote, once we add these two additions to the KIP (LMK if you want
> me
> > to
> > > > > take that up).
> > > > >
> > > > > Regards
> > > > > Snehashis
> > > > >
> > > > > On Tue, Jul 2, 2024 at 12:08 AM Greg Harris
> > > <greg.harris@aiven.io.invalid
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hey Snehashis,
> > > > > >
> > > > > > Sorry for the late reply, and thanks for helping close out the
> > > > > discussion.
> > > > > >
> > > > > > > Note that if my assumptions are correct then
> > > > > > > this can happen with the existing framework as well, or is
> there
> > > some
> > > > > > > safeguard from this happening?
> > > > > >
> > > > > > Currently, if a cluster has a heterogeneous plugin installation,
> > each
> > > > > > worker may have a different definition of "latest". If you call
> to
> > > > > > validate a configuration, it will validate against whatever
> latest
> > > > > version
> > > > > > is present locally.
> > > > > > With this KIP, that behavior will continue when the version isn't
> > > > > > specified, when the version is a soft requirement, or when the
> > > version
> > > > > is a
> > > > > > range. For users that do not want to tolerate their tasks having
> > > > > > heterogeneous versions, the single-version hard requirements are
> > > there,
> > > > > but
> > > > > > come at the cost of micromanagement of upgrades.
> > > > > > Perhaps there is some room in the future for a connector
> configured
> > > > with
> > > > > a
> > > > > > range to "pin" the version that all of its tasks should use. Or
> > > > > > version-aware-scheduling can schedule connectors and tasks to
> > workers
> > > > > that
> > > > > > will resolve the same versions. The important primitive is being
> > able
> > > > to
> > > > > > enforce which version is being used, everything else looks more
> > like
> > > a
> > > > > > convenience feature to me.
> > > > > >
> > > > > > > So far, we could have pointed to the
> > > > > > > misconfigured cluster configuration and somewhat differ this
> > > problem
> > > > to
> > > > > > > something outside of connect runtime.
> > > > > >
> > > > > > I think this will still be the case. Even with this feature, we
> > will
> > > > > still
> > > > > > recommend homogeneous clusters that have the same set of plugins
> > > > > installed
> > > > > > on every worker. Version-aware scheduling is a rejected
> > alternative,
> > > > and
> > > > > > your example for performing validation requests on an arbitrary
> > > worker
> > > > > is a
> > > > > > great example of a complication for heterogeneous clusters that
> > we're
> > > > not
> > > > > > ready to solve.
> > > > > >
> > > > > > > We can introduce
> > > > > > > a new path under this for version
> > > > (/connector/connector-name/version),
> > > > > > but
> > > > > > > perhaps adding this as part of the status is a valid
> alternative.
> > > > > >
> > > > > > I think either is fine. Adding a new field to the existing
> > endpoints
> > > is
> > > > > > fine, and shouldn't require special compatibility such as a query
> > > > > > parameter.
> > > > > >
> > > > > > > Also, to go further I think
> > > > > > > version information for tasks could also be available,
> > > > > >
> > > > > > I completely agree. The version of each connector and each task
> > > should
> > > > be
> > > > > > visible individually, since they may be different during a
> rolling
> > > > > upgrade.
> > > > > >
> > > > > > Thanks!
> > > > > > Greg
> > > > > >
> > > > > > On Tue, Jun 18, 2024 at 5:18 AM Snehashis <
> > snehashisp1995@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Greg, Chris
> > > > > > >
> > > > > > > Thanks for the in-depth discussion, I have a couple of
> discussion
> > > > > points
> > > > > > > and would like your thoughts on this.
> > > > > > >
> > > > > > > 1) One concern I have with the new addition of 'soft' and
> 'hard'
> > > > > version
> > > > > > > requirements is that there could be a mismatch in the plugin
> > > version
> > > > > that
> > > > > > > two different tasks are running, if a soft requirement is
> > provided
> > > > and
> > > > > > the
> > > > > > > nodes a multi cluster deployment are not in sync w.r.t the
> plugin
> > > > > > versions
> > > > > > > that they are configured with. Note that if my assumptions are
> > > > correct
> > > > > > then
> > > > > > > this can happen with the existing framework as well, or is
> there
> > > some
> > > > > > > safeguard from this happening? So far, we could have pointed to
> > the
> > > > > > > misconfigured cluster configuration and somewhat differ this
> > > problem
> > > > to
> > > > > > > something outside of connect runtime. With this feature in
> place
> > > > > perhaps
> > > > > > > the expectation is more on connect to not be running with such
> > > > > > > inconsistency, especially if a connector version is specified.
> > This
> > > > is
> > > > > > also
> > > > > > > a problem with validation if different cluster have different
> > > > > > > configurations, as IIRC validations are local to the worker
> which
> > > > > > receives
> > > > > > > the rest call for validate. So, we might be validating with a
> > > certain
> > > > > > > version which is different from the one that will be used to
> > create
> > > > > > > connector and tasks. Again, this is likely how the current
> state
> > > is,
> > > > > but
> > > > > > > perhaps such inconsistencies warrant a deeper look with the
> > > addition
> > > > of
> > > > > > > this feature. The problems associated with them can be somewhat
> > > > > insidious
> > > > > > > and hard to diagnose.
> > > > > > >
> > > > > > > 2) There was some discussion on the need for a new REST
> endpoint
> > to
> > > > > > provide
> > > > > > > information on the versions of running connectors, and I think
> > > adding
> > > > > > this
> > > > > > > information via REST is a valuable addition. The way I see it
> the
> > > > > version
> > > > > > > is an intrinsic property of an instance of a running connector
> > and
> > > > > hence
> > > > > > > this should be part of the set of APIs under
> > > > > /connector/<connector-name>
> > > > > > > (also the /connectors API should also have this information as
> it
> > > is
> > > > an
> > > > > > > amalgamation of all the individual connector information). We
> can
> > > > > > introduce
> > > > > > > a new path under this for version
> > > > (/connector/connector-name/version),
> > > > > > but
> > > > > > > perhaps adding this as part of the status is a valid
> alternative.
> > > > This
> > > > > is
> > > > > > > mentioned as a rejected alternative right now. Also, to go
> > further
> > > I
> > > > > > think
> > > > > > > version information for tasks could also be available,
> especially
> > > if
> > > > we
> > > > > > > choose to not address the pitfalls discussed in my point 1),
> this
> > > > will
> > > > > > > at-least provide admins a quick and easy way to determine if
> such
> > > and
> > > > > > > inconsistent state exist in any of the connectors.
> > > > > > >
> > > > > > > Thanks again for reviving my original KIP and working to
> improve
> > > it.
> > > > > > > Looking forward to your thoughts on the points mentioned above.
> > > > > > > Regards
> > > > > > > Snehashis
> > > > > > >
> > > > > > >
> > > > > > > On Wed, May 29, 2024 at 9:59 PM Chris Egerton
> > > > <chrise@aiven.io.invalid
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Greg,
> > > > > > > >
> > > > > > > > First, an apology! I mistakenly assumed that each plugin
> > appeared
> > > > > only
> > > > > > > once
> > > > > > > > in the responses from GET
> > > /connector-plugins?connectorsOnly=false.
> > > > > > Thank
> > > > > > > > you for correcting me and pointing out that all versions of
> > each
> > > > > plugin
> > > > > > > > appear in that response, which does indeed satisfy my desire
> > for
> > > > > users
> > > > > > to
> > > > > > > > discover this information in at most two REST requests (and
> in
> > > > fact,
> > > > > > does
> > > > > > > > it in only one)!
> > > > > > > >
> > > > > > > > And secondly, with the revelation about recommenders, I agree
> > > that
> > > > > it's
> > > > > > > > best to leave the "version" property out of the lists of
> > > properties
> > > > > > > > returned from the GET /connector-plugins/<plugin>/config
> > > endpoint.
> > > > > > > >
> > > > > > > > With those two points settled, I think the only unresolved
> item
> > > is
> > > > > the
> > > > > > > > small change to version parsing added to the KIP (where raw
> > > version
> > > > > > > numbers
> > > > > > > > are treated as an exact match, instead of a best-effort match
> > > with
> > > > a
> > > > > > > > fallback on the default version). If the KIP is updated with
> > that
> > > > > then
> > > > > > > I'd
> > > > > > > > be ready to vote on it.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Wed, May 29, 2024 at 12:00 PM Greg Harris
> > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Chris,
> > > > > > > > >
> > > > > > > > > Thanks for your thoughts.
> > > > > > > > >
> > > > > > > > > > Won't it still only expose the
> > > > > > > > > > latest version for each plugin, instead of the range of
> > > > versions
> > > > > > > > > available?
> > > > > > > > >
> > > > > > > > > Here is a snippet of the current output of the GET
> > > > > > > > > /connector-plugins?connectorsOnly=false endpoint, after I
> > > > installed
> > > > > > two
> > > > > > > > > versions of the debezium PostgresConnector:
> > > > > > > > >
> > > > > > > > > {
> > > > > > > > > "class":
> > > > "io.debezium.connector.postgresql.PostgresConnector",
> > > > > > > > > "type": "source",
> > > > > > > > > "version": "2.0.1.Final"
> > > > > > > > > },
> > > > > > > > > {
> > > > > > > > > "class":
> > > > "io.debezium.connector.postgresql.PostgresConnector",
> > > > > > > > > "type": "source",
> > > > > > > > > "version": "2.6.1.Final"
> > > > > > > > > },
> > > > > > > > >
> > > > > > > > > I think this satisfies your requirement to learn about all
> > > > plugins
> > > > > > and
> > > > > > > > all
> > > > > > > > > versions in two or fewer REST calls.
> > > > > > > > >
> > > > > > > > > I tried to get an example of the output of `/config` by
> > > > hardcoding
> > > > > > the
> > > > > > > > > Recommender, and realized that Recommenders aren't executed
> > on
> > > > the
> > > > > > > > > `/config` endpoint at all: only during validation, when a
> > > > > > configuration
> > > > > > > > is
> > > > > > > > > actually present.
> > > > > > > > > And this led me to discover that the `/config` endpoint
> > > returns a
> > > > > > > > > List<ConfigKeyInfo>, and ConfigKeyInfo does not contain a
> > > > > > > > recommendedValues
> > > > > > > > > field. The ConfigValue field is the object which contains
> > > > > > > > > recommendedValues, and it is only generated during
> > validation.
> > > > > > > > > I think it's out of scope to start calling recommenders on
> > > empty
> > > > > > > configs
> > > > > > > > > that might throw exceptions, changing the existing REST
> > > entities,
> > > > > or
> > > > > > > > > changing the core ConfigDef implementation.
> > > > > > > > > Someone could add this functionality later, I don't think
> > it's
> > > > > > > necessary
> > > > > > > > > here.
> > > > > > > > >
> > > > > > > > > Then the question is: should "version" without recommenders
> > > > appear
> > > > > in
> > > > > > > > > non-connector plugins? I think I'd rather be consistent
> with
> > > > > > > "predicate"
> > > > > > > > > and "negate" on release, and let a later improvement add
> > them.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Greg
> > > > > > > > >
> > > > > > > > > On Wed, May 29, 2024 at 8:06 AM Chris Egerton
> > > > > > <chrise@aiven.io.invalid
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Greg,
> > > > > > > > > >
> > > > > > > > > > I'm confused about the behavior for GET
> > > > > > > > > > /connector-plugins?connectorsOnly=false. Won't it still
> > only
> > > > > expose
> > > > > > > the
> > > > > > > > > > latest version for each plugin, instead of the range of
> > > > versions
> > > > > > > > > available?
> > > > > > > > > >
> > > > > > > > > > I'm hoping we can provide a flow where people need at
> most
> > > two
> > > > > REST
> > > > > > > > calls
> > > > > > > > > > to discover 1) the complete set of plugins available on
> the
> > > > > worker
> > > > > > > > (which
> > > > > > > > > > is already possible with the endpoint under discussion)
> and
> > > 2)
> > > > > the
> > > > > > > set
> > > > > > > > of
> > > > > > > > > > versions available for a specific plugin on the worker
> > (which
> > > > > > doesn't
> > > > > > > > > > appear to be possible, at least for some plugin types).
> > This
> > > > > > wouldn't
> > > > > > > > > > require any out-of-band knowledge and would be valuable
> for
> > > > > > connector
> > > > > > > > > users
> > > > > > > > > > (who may want to, for example, know what their options
> are
> > > when
> > > > > > > > > considering
> > > > > > > > > > a plugin upgrade) and cluster administrators (who could
> use
> > > it
> > > > > as a
> > > > > > > > > sanity
> > > > > > > > > > check for the setup of their Kafka Connect clusters
> without
> > > > > having
> > > > > > to
> > > > > > > > > pore
> > > > > > > > > > over log files).
> > > > > > > > > >
> > > > > > > > > > As far as modifying the content of the GET
> > > > > > > > > > /connector-plugins/<plugin>/config endpoint to include a
> > > > > "version"
> > > > > > > > > property
> > > > > > > > > > goes, I think your point about returning that property
> from
> > > > > > requests
> > > > > > > to
> > > > > > > > > > that endpoint that include a version query parameter is
> > > > salient,
> > > > > > but
> > > > > > > it
> > > > > > > > > > also unfortunately applies to all types of plugin. I
> don't
> > > > think
> > > > > it
> > > > > > > > > should
> > > > > > > > > > completely disqualify that option. I also wasn't
> imagining
> > > > adding
> > > > > > > > > anything
> > > > > > > > > > besides that single property to that endpoint, so no new
> > > > > > "predicate"
> > > > > > > > > > property for SMTs, no new "negate" property for
> predicates,
> > > and
> > > > > no
> > > > > > > new
> > > > > > > > > > "type" property for either. I'm not necessarily opposed
> to
> > > > adding
> > > > > > > > those,
> > > > > > > > > > but it can be done without being pulled into the scope of
> > > this
> > > > > KIP.
> > > > > > > > > >
> > > > > > > > > > So to summarize how I imagine people might use the REST
> API
> > > > after
> > > > > > > this
> > > > > > > > > KIP:
> > > > > > > > > >
> > > > > > > > > > 1. Discover the set of plugins on the worker via GET
> > > > > > > > > > /connector-plugins?connectorsOnly=false
> > > > > > > > > > 2. For any of those plugins, discover the set of
> available
> > > > > versions
> > > > > > > by
> > > > > > > > > > examining the recommended values for the "version"
> property
> > > > after
> > > > > > > > hitting
> > > > > > > > > > the GET GET /connector-plugins/<plugin>/config endpoint
> > > > > > > > > >
> > > > > > > > > > I don't think this is necessarily optimal since plucking
> > the
> > > > > > > "version"
> > > > > > > > > > property out of the response from that endpoint might be
> a
> > > PITA
> > > > > for
> > > > > > > ad
> > > > > > > > > hoc
> > > > > > > > > > queries via, e.g., curl, but I think it'd be enough for
> > this
> > > > KIP.
> > > > > > > Still
> > > > > > > > > > open to other options though, and if I've misunderstood
> > > > anything
> > > > > in
> > > > > > > the
> > > > > > > > > > proposal, please let me know!
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
> > > > > > > > > > Chris
> > > > > > > > > >
> > > > > > > > > > On Wed, May 22, 2024 at 4:05 PM Greg Harris
> > > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey Chris,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your comments, and I'm glad that it seems
> like
> > > > we're
> > > > > > > > > > > aligning on the vision here.
> > > > > > > > > > >
> > > > > > > > > > > > An
> > > > > > > > > > > > alternative could be to change existing behavior to
> > fail
> > > > fast
> > > > > > on
> > > > > > > > any
> > > > > > > > > > > > invalid default converter configuration instead of
> just
> > > for
> > > > > > > invalid
> > > > > > > > > > > > versions
> > > > > > > > > > >
> > > > > > > > > > > I suppose if this is landing in 4.0, we have the
> > > opportunity
> > > > to
> > > > > > > break
> > > > > > > > > > > compatibility and strictly validate the worker class
> > > configs,
> > > > > and
> > > > > > > > > > > could have a new consistent behavior instead of
> > > inconsistent
> > > > > but
> > > > > > > > > > > backwards-compatible behavior.
> > > > > > > > > > >
> > > > > > > > > > > > no other part of the KIP requires this change.
> > > > > > > > > > >
> > > > > > > > > > > This is correct, and a compelling argument. I'm fine
> > > leaving
> > > > > the
> > > > > > > > > > > strict worker validation off of this configuration to
> > > > > potentially
> > > > > > > be
> > > > > > > > > > > added later. If a KIP was raised to perform strict
> > > validation
> > > > > of
> > > > > > > the
> > > > > > > > > > > worker config, it would include the version config, and
> > can
> > > > > > address
> > > > > > > > > > > backwards compatibility for both configs together.
> > > > > > > > > > >
> > > > > > > > > > > > RE exposing the version property in the
> > > > > > > > > > > /connector-plugins/<plugin>/config
> > > > > > > > > > > > endpoint, the behavior is inconsistent across plugin
> > > types.
> > > > > > > > > > >
> > > > > > > > > > > Yes it is, and that started in a bugfix, once people
> > > started
> > > > > > using
> > > > > > > > > > > this endpoint:
> > > > > https://issues.apache.org/jira/browse/KAFKA-14843
> > > > > > > . I
> > > > > > > > > > > wasn't planning on converging the behavior in this KIP,
> > as
> > > I
> > > > > > > > > > > considered it out-of-scope, and was going to follow the
> > > > current
> > > > > > > > > > > behavior. To summarize:
> > > > > > > > > > >
> > > > > > > > > > > GET /connector-plugins/<connector>/config will emit:
> > > > > > > > > > > * connector.class (already implemented)
> > > > > > > > > > > * connector.version (new)
> > > > > > > > > > > * key.converter (already implemented)
> > > > > > > > > > > * key.converter.version (new)
> > > > > > > > > > > * value.converter (already implemented)
> > > > > > > > > > > * value.converter.version (new)
> > > > > > > > > > > * header.converter (already implemented)
> > > > > > > > > > > * header.converter.version (new)
> > > > > > > > > > > But will NOT emit:
> > > > > > > > > > > * transforms.<alias>.type
> > > > > > > > > > > * transforms.<alias>.version
> > > > > > > > > > > * transforms.<alias>.predicate
> > > > > > > > > > > * predicates.<alias>.type
> > > > > > > > > > > * predicates.<alias>.version
> > > > > > > > > > > * predicates.<alias>.negate
> > > > > > > > > > >
> > > > > > > > > > > GET /connector-plugins/<converter>/config will NOT
> emit:
> > > > > > > > > > > * "" (there's not even a ".class" prefix!)
> > > > > > > > > > > * version
> > > > > > > > > > >
> > > > > > > > > > > GET /connector-plugins/<transform>/config will NOT
> emit:
> > > > > > > > > > > * type
> > > > > > > > > > > * version
> > > > > > > > > > > * predicate
> > > > > > > > > > >
> > > > > > > > > > > GET /connector-plugins/<predicate>/config will NOT
> emit:
> > > > > > > > > > > * type
> > > > > > > > > > > * version
> > > > > > > > > > > * negate
> > > > > > > > > > >
> > > > > > > > > > > Do you want the converter, transform, and predicate
> > > endpoints
> > > > > > > > changed?
> > > > > > > > > > > Do you want just "version", or do you want all of the
> > > > prefixed
> > > > > > > > configs
> > > > > > > > > > > including "type", "predicate" and "negate"? How would
> you
> > > > want
> > > > > to
> > > > > > > > > > > handle the converters?
> > > > > > > > > > > And when I say the configs are "not part of the plugin
> > > config
> > > > > > > itself"
> > > > > > > > > > > I mean that saying that GET
> > > > > > > > > > > /connector-plugins/Flatten$Key/config?version=3.8.0
> has a
> > > > > > "version"
> > > > > > > > > > > config that must be "3.8.0" is a little bit nonsense,
> as
> > > the
> > > > > > > version
> > > > > > > > > > > is already specified.
> > > > > > > > > > >
> > > > > > > > > > > > IMO
> > > > > > > > > > > > it's worth including this information somewhere
> > directly
> > > > > > > accessible
> > > > > > > > > > > without
> > > > > > > > > > > > having to provide a full connector config. FWIW I'd
> be
> > > fine
> > > > > > with
> > > > > > > > GET
> > > > > > > > > > > > /connector-plugins/<plugin>/versions as a first-class
> > > > > endpoint
> > > > > > > > > > >
> > > > > > > > > > > You don't have to provide a configuration to call GET
> > > > > > > > > > > /connector-plugins?connectorsOnly=false , is that
> > endpoint
> > > > not
> > > > > > > close
> > > > > > > > > > > enough to what you have in mind? See also the Rejected
> > > > > > Alternative
> > > > > > > > > > > "Adding new REST API endpoints"
> > > > > > > > > > >
> > > > > > > > > > > If you're calling
> /connector-plugins/<transform>/config,
> > > you
> > > > > know
> > > > > > > the
> > > > > > > > > > > name of a plugin right? That either comes from
> > out-of-band
> > > > > > > knowledge,
> > > > > > > > > > > validating a connector config, or calling GET
> > > > > > > > > > > /connector-plugins?connectorsOnly=false.
> > > > > > > > > > > * If you have out-of-band knowledge of plugin classes,
> > > > perhaps
> > > > > > you
> > > > > > > > > > > have out-of-band knowledge of versions too.
> > > > > > > > > > > * If you've just validated a connector config, there
> > should
> > > > be
> > > > > an
> > > > > > > > > > > accompanying "version" field there with an accurate
> > default
> > > > > value
> > > > > > > and
> > > > > > > > > > > recommenders.
> > > > > > > > > > > * If you've called GET
> > > > /connector-plugins?connectorsOnly=false,
> > > > > > > that
> > > > > > > > > > > endpoint includes version information.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Greg
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 22, 2024 at 11:05 AM Chris Egerton
> > > > > > > > <chrise@aiven.io.invalid
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > >
> > > > > > > > > > > > Hope you had a nice weekend! Gonna try to keep things
> > > > > concise.
> > > > > > > > > > > >
> > > > > > > > > > > > Concluded points:
> > > > > > > > > > > >
> > > > > > > > > > > > RE version recommenders, I agree it's likely that
> > > > > programmatic
> > > > > > > UIs
> > > > > > > > > will
> > > > > > > > > > > > already be able to handle dynamic configuration
> > > > definitions,
> > > > > > and
> > > > > > > > the
> > > > > > > > > > > detail
> > > > > > > > > > > > about SMTs is a great point. I still anticipate some
> > > > > > awkwardness
> > > > > > > > with
> > > > > > > > > > > > connector versions, though: if the latest version
> > > supports
> > > > > some
> > > > > > > new
> > > > > > > > > > > > properties, then a user switches to an earlier
> > version, a
> > > > UI
> > > > > > may
> > > > > > > > > > respond
> > > > > > > > > > > by
> > > > > > > > > > > > wiping values for these properties. I guess we can
> bite
> > > the
> > > > > > > bullet,
> > > > > > > > > > > though.
> > > > > > > > > > > >
> > > > > > > > > > > > RE double-dinging during preflight validation for
> > invalid
> > > > > > > > versions, I
> > > > > > > > > > > like
> > > > > > > > > > > > the analogy with login credentials. I'm convinced
> that
> > > the
> > > > > > > proposal
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > KIP is best 👍
> > > > > > > > > > > >
> > > > > > > > > > > > Continued points:
> > > > > > > > > > > >
> > > > > > > > > > > > RE failing on worker startup, sorry, I should be
> > clearer:
> > > > > there
> > > > > > > is
> > > > > > > > no
> > > > > > > > > > > _new_
> > > > > > > > > > > > justification for it that doesn't also apply to
> > existing
> > > > > > > behavior.
> > > > > > > > We
> > > > > > > > > > > > shouldn't diverge from existing behavior solely for
> > this
> > > > new
> > > > > > > case.
> > > > > > > > An
> > > > > > > > > > > > alternative could be to change existing behavior to
> > fail
> > > > fast
> > > > > > on
> > > > > > > > any
> > > > > > > > > > > > invalid default converter configuration instead of
> just
> > > for
> > > > > > > invalid
> > > > > > > > > > > > versions, but I'd vote to just stick to existing
> > behavior
> > > > and
> > > > > > not
> > > > > > > > > > > > complicate things, especially since no other part of
> > the
> > > > KIP
> > > > > > > > requires
> > > > > > > > > > > this
> > > > > > > > > > > > change.
> > > > > > > > > > > >
> > > > > > > > > > > > RE exposing the version property in the
> > > > > > > > > > > /connector-plugins/<plugin>/config
> > > > > > > > > > > > endpoint, the behavior is inconsistent across plugin
> > > types.
> > > > > > > Hitting
> > > > > > > > > the
> > > > > > > > > > > > endpoint for the FileStreamSinkConnector on version
> > 3.7.0
> > > > > > yields
> > > > > > > a
> > > > > > > > > > > response
> > > > > > > > > > > > that includes, among other things, the "topics",
> > > > > > "topics.regex",
> > > > > > > > and
> > > > > > > > > > > > "errors.tolerance" properties. I see that we don't do
> > > this
> > > > > > > > everywhere
> > > > > > > > > > > (the
> > > > > > > > > > > > examples you cite for SMT and converter properties
> are
> > > > > > accurate),
> > > > > > > > but
> > > > > > > > > > IMO
> > > > > > > > > > > > it's worth including this information somewhere
> > directly
> > > > > > > accessible
> > > > > > > > > > > without
> > > > > > > > > > > > having to provide a full connector config. FWIW I'd
> be
> > > fine
> > > > > > with
> > > > > > > > GET
> > > > > > > > > > > > /connector-plugins/<plugin>/versions as a first-class
> > > > > endpoint
> > > > > > > > either
> > > > > > > > > > > > instead of or in addition to adding recommended
> values
> > > for
> > > > > all
> > > > > > > > plugin
> > > > > > > > > > > > versions.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your continued work on this KIP, and with
> > the
> > > > > > progress
> > > > > > > > > we're
> > > > > > > > > > > > making I'm optimistic about its chances of appearing
> in
> > > > > 4.0.0.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > >
> > > > > > > > > > > > Chris
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, May 15, 2024 at 1:22 PM Greg Harris
> > > > > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hey Chris,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your quick follow up.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > But this risk is already present with
> > > > > > > > > > > > > > existing error cases, and I don't see anything
> that
> > > > > > justifies
> > > > > > > > > > > changing
> > > > > > > > > > > > > > existing behavior with an invalid converter
> class,
> > or
> > > > > > > diverging
> > > > > > > > > > from
> > > > > > > > > > > it
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > the case of invalid converter versions.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The justification is to fail-fast, and prevent REST
> > API
> > > > > users
> > > > > > > > from
> > > > > > > > > > > > > receiving errors from bad configs that they didn't
> > > write,
> > > > > or
> > > > > > > > maybe
> > > > > > > > > > > > > don't even know apply to them.
> > > > > > > > > > > > > Up until recently errors in these configurations
> > > surfaced
> > > > > as
> > > > > > > > > failures
> > > > > > > > > > > > > to create the connector, or failures to start, and
> > you
> > > > made
> > > > > > > them
> > > > > > > > > > > > > fail-fast during validation. I think this change is
> > in
> > > > the
> > > > > > same
> > > > > > > > > > > > > spirit, pulling the validation further forward and
> > not
> > > > > > letting
> > > > > > > > > errors
> > > > > > > > > > > > > lie dormant.
> > > > > > > > > > > > >
> > > > > > > > > > > > > And to call back to your original concern about
> > > > > interrupting
> > > > > > > > > > > > > connectors that explicitly provide these
> > configurations
> > > > and
> > > > > > > don't
> > > > > > > > > use
> > > > > > > > > > > > > the worker configs: I expect that operators with a
> > > > majority
> > > > > > of
> > > > > > > > > these
> > > > > > > > > > > > > sorts of clients aren't going to be setting the
> > worker
> > > > > > .version
> > > > > > > > > > > > > properties, because it would have no effect on the
> > > > majority
> > > > > > of
> > > > > > > > > their
> > > > > > > > > > > > > connectors. They would be able to rely on
> > > > > > > backwards-compatibility
> > > > > > > > > and
> > > > > > > > > > > > > continue to ignore the class properties.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > CLI
> > > > > > > > > > > > > > and programmatic UI developers will want to
> develop
> > > > their
> > > > > > own
> > > > > > > > > > tooling
> > > > > > > > > > > > > > layers.
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is a very compelling argument. Snehashis do
> you
> > > want
> > > > > to
> > > > > > > > figure
> > > > > > > > > > > > > out a REST API design for this use-case?
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding the GET
> > /connector-plugins/<plugin>/config
> > > > > > > endpoint,
> > > > > > > > I
> > > > > > > > > > was
> > > > > > > > > > > > > > thinking about the response for non-connector
> > > plugins,
> > > > > > e.g.,
> > > > > > > > > > > > > > GET /connector-plugins/RegexRouter/config. Would
> a
> > > > > > "version"
> > > > > > > > > > property
> > > > > > > > > > > > > > appear with recommended values?
> > > > > > > > > > > > >
> > > > > > > > > > > > > I intended for the ".version" property to be like
> > other
> > > > > > > framework
> > > > > > > > > > > > > configs (".class", ".type", ".predicate",
> ".negate")
> > > > where
> > > > > > they
> > > > > > > > are
> > > > > > > > > > > > > inside the plugin namespace, but not part of the
> > plugin
> > > > > > config
> > > > > > > > > > itself.
> > > > > > > > > > > > > Perhaps we can deviate from those configs because
> it
> > > > would
> > > > > > aid
> > > > > > > in
> > > > > > > > > > > > > discovering other valid `GET
> > > > > > > > > > > > > /connector-plugins/<plugin>/config?version=` calls,
> > > > without
> > > > > > > > calling
> > > > > > > > > > > > > `GET /connector-plugins?connectorsOnly=false`.
> > > > > > > > > > > > > I don't really feel strongly either way.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > if a user changes the
> > > > > > > > > > > > > > connector version in, e.g., a dropdown menu, then
> > the
> > > > UI
> > > > > > > either
> > > > > > > > > has
> > > > > > > > > > > to
> > > > > > > > > > > > > > re-fetch the ConfigDef for the new version, or
> risk
> > > > > > operating
> > > > > > > > on
> > > > > > > > > > > stale
> > > > > > > > > > > > > > information
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is a really interesting situation, thanks for
> > > > finding
> > > > > > > that!
> > > > > > > > > This
> > > > > > > > > > > > > is already a footgun with transformations and
> > > predicates;
> > > > > > Once
> > > > > > > > you
> > > > > > > > > > > > > fill out a transformation class (which includes a
> > > > > > recommender)
> > > > > > > > you
> > > > > > > > > > > > > will then see all of the transformations
> > > configurations.
> > > > > > > > > > > > > I think this means that UI developers should have
> > > already
> > > > > > > > developed
> > > > > > > > > > > > > the infrastructure for handling dynamic
> recommenders,
> > > or
> > > > > else
> > > > > > > > > they've
> > > > > > > > > > > > > had this bug since KIP-66 in 2017. It may require
> > some
> > > > > manual
> > > > > > > > > > > > > attention to roll-out support for the ".version"
> > > > properties
> > > > > > > > > > > > > specifically, but I don't think that should prevent
> > us
> > > > from
> > > > > > > > > providing
> > > > > > > > > > > > > this information in a natural place.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > but will it be possible to configure
> > > > > > > > > > > > > > a connector to use two instances of the same
> > > transform
> > > > or
> > > > > > > > > > predicate,
> > > > > > > > > > > but
> > > > > > > > > > > > > > with different versions for each?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, and this is included in the example in the
> KIP,
> > > > > > > > > > > > > `transforms.flatten-latest` and
> > > `transforms.flatten-old`.
> > > > > > This
> > > > > > > > will
> > > > > > > > > > > > > work in the natural way, with the two instances
> > having
> > > a
> > > > > > > > different
> > > > > > > > > > > > > version if configured for that. If a user wants to
> > pin
> > > > the
> > > > > > same
> > > > > > > > > > > > > version of a plugin for every instance, they will
> > need
> > > to
> > > > > > > provide
> > > > > > > > > the
> > > > > > > > > > > > > version config multiple times and keep them in-sync
> > > > > > themselves.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I would have expected the error
> > > > > > > > > > > > > > to only be attributed to the version property,
> and
> > > for
> > > > > the
> > > > > > > > class
> > > > > > > > > > > property
> > > > > > > > > > > > > > to be reported as valid.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I considered this, and then went back and changed
> it.
> > > To
> > > > > > quote
> > > > > > > > > > someone
> > > > > > > > > > > > > else, this is to catch "misspelling cat as dog".
> For
> > > > > example
> > > > > > a
> > > > > > > > user
> > > > > > > > > > > > > types DogConverter and meant to type CatConverter,
> > and
> > > > then
> > > > > > > > types a
> > > > > > > > > > > > > version which is valid for CatConverter,
> conceptually
> > > the
> > > > > > error
> > > > > > > > is
> > > > > > > > > in
> > > > > > > > > > > > > the class name and not the version.
> > > > > > > > > > > > > That's a very contrived scenario, but I think
> similar
> > > > > > arguments
> > > > > > > > are
> > > > > > > > > > > > > used for attributing validation errors to
> > > > > > > > endpoints/urls/hostnames
> > > > > > > > > > > > > when the associated credentials are unable to log
> > into
> > > > the
> > > > > > > remote
> > > > > > > > > > > > > system. Did the user provide the correct testing
> > > > > credentials,
> > > > > > > but
> > > > > > > > > > > > > accidentally typed the production endpoint? Even if
> > the
> > > > > > > > production
> > > > > > > > > > > > > endpoint looks valid (it's a real hostname that is
> > > > > reachable)
> > > > > > > the
> > > > > > > > > > > > > conceptual error is still in the hostname and
> should
> > > have
> > > > > the
> > > > > > > > error
> > > > > > > > > > > > > attributed to it to draw the user's attention.
> > > > > > > > > > > > > If that's not convincing, I think the alternative
> of
> > > only
> > > > > > > > > attributing
> > > > > > > > > > > > > version errors to the version property is also
> > > > acceptable.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Greg
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, May 15, 2024 at 8:06 AM Chris Egerton
> > > > > > > > > > <chrise@aiven.io.invalid
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for your responses! Continuations of
> > existing
> > > > > > > > discussions:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding crashing the worker on startup--yes,
> > there
> > > is
> > > > > > also
> > > > > > > a
> > > > > > > > > risk
> > > > > > > > > > > to
> > > > > > > > > > > > > > allowing it to join the cluster. But this risk is
> > > > already
> > > > > > > > present
> > > > > > > > > > > with
> > > > > > > > > > > > > > existing error cases, and I don't see anything
> that
> > > > > > justifies
> > > > > > > > > > > changing
> > > > > > > > > > > > > > existing behavior with an invalid converter
> class,
> > or
> > > > > > > diverging
> > > > > > > > > > from
> > > > > > > > > > > it
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > the case of invalid converter versions. I think
> we
> > > > should
> > > > > > > keep
> > > > > > > > > this
> > > > > > > > > > > > > simple
> > > > > > > > > > > > > > and not do anything different for worker startup
> > than
> > > > we
> > > > > do
> > > > > > > > > > already.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > As far as REST API vs. metrics go--I think you're
> > > right
> > > > > > that
> > > > > > > > the
> > > > > > > > > > > original
> > > > > > > > > > > > > > version metrics were added as a "monitoring"
> > detail.
> > > > > > However,
> > > > > > > > > this
> > > > > > > > > > > was
> > > > > > > > > > > > > back
> > > > > > > > > > > > > > when plugin versions were managed solely by
> cluster
> > > > > > > > > administrators.
> > > > > > > > > > > With
> > > > > > > > > > > > > > this KIP, connector users will be able to manage
> > > plugin
> > > > > > > > versions,
> > > > > > > > > > > and CLI
> > > > > > > > > > > > > > and programmatic UI developers will want to
> develop
> > > > their
> > > > > > own
> > > > > > > > > > tooling
> > > > > > > > > > > > > > layers. I think focusing on the REST API as the
> > > primary
> > > > > > > > interface
> > > > > > > > > > for
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > KIP would be best for these users.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > (All that said, I don't object to the metrics
> that
> > > are
> > > > > > > proposed
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > KIP;
> > > > > > > > > > > > > > I just think they make more sense in addition to
> > new
> > > > REST
> > > > > > API
> > > > > > > > > > > > > > functionality, as opposed to instead of it.)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding the GET
> > /connector-plugins/<plugin>/config
> > > > > > > endpoint,
> > > > > > > > I
> > > > > > > > > > was
> > > > > > > > > > > > > > thinking about the response for non-connector
> > > plugins,
> > > > > > e.g.,
> > > > > > > > > > > > > > GET /connector-plugins/RegexRouter/config. Would
> a
> > > > > > "version"
> > > > > > > > > > property
> > > > > > > > > > > > > > appear with recommended values?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > And new thoughts:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1) Regarding the recommended values for
> > > > > > "connector.version",
> > > > > > > > this
> > > > > > > > > > > might
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > confusing since there could be differences
> between
> > > the
> > > > > > > > ConfigDefs
> > > > > > > > > > > for the
> > > > > > > > > > > > > > latest version of the connector and prior
> versions.
> > > It
> > > > > also
> > > > > > > > makes
> > > > > > > > > > the
> > > > > > > > > > > > > flow
> > > > > > > > > > > > > > a bit awkward for programmatic UI developers: if
> a
> > > user
> > > > > > > changes
> > > > > > > > > the
> > > > > > > > > > > > > > connector version in, e.g., a dropdown menu, then
> > the
> > > > UI
> > > > > > > either
> > > > > > > > > has
> > > > > > > > > > > to
> > > > > > > > > > > > > > re-fetch the ConfigDef for the new version, or
> risk
> > > > > > operating
> > > > > > > > on
> > > > > > > > > > > stale
> > > > > > > > > > > > > > information. I'm starting to doubt that exposing
> > the
> > > > > range
> > > > > > of
> > > > > > > > > > > available
> > > > > > > > > > > > > > versions via recommended values is the best way
> to
> > > > > proceed,
> > > > > > > > > instead
> > > > > > > > > > > of a
> > > > > > > > > > > > > > more explicit approach like GET
> > > > > > > > > > > /connector-plugins/<plugin>/versions, or
> > > > > > > > > > > > > > the "Adding new REST API endpoints" rejected
> > > > alternative.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2) I know that this is a bit heinous, but will it
> > be
> > > > > > possible
> > > > > > > > to
> > > > > > > > > > > > > configure
> > > > > > > > > > > > > > a connector to use two instances of the same
> > > transform
> > > > or
> > > > > > > > > > predicate,
> > > > > > > > > > > but
> > > > > > > > > > > > > > with different versions for each? (I don't think
> > this
> > > > is
> > > > > > > worth
> > > > > > > > > > > > > significant
> > > > > > > > > > > > > > design/implementation effort, so if it would
> > inflate
> > > > > either
> > > > > > > of
> > > > > > > > > > those,
> > > > > > > > > > > > > > please don't feel obligated to support it.)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3) Just out of curiosity, why double-ding during
> > > config
> > > > > > > > > validation
> > > > > > > > > > > if the
> > > > > > > > > > > > > > version for a plugin class can't be found? I
> would
> > > have
> > > > > > > > expected
> > > > > > > > > > the
> > > > > > > > > > > > > error
> > > > > > > > > > > > > > to only be attributed to the version property,
> and
> > > for
> > > > > the
> > > > > > > > class
> > > > > > > > > > > property
> > > > > > > > > > > > > > to be reported as valid.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, May 14, 2024 at 1:11 PM Greg Harris
> > > > > > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hey Chris,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for your comments. I'm glad you like the
> > > > > > > motivations,
> > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > wrote that part!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > the configuration syntax for the most basic
> use
> > > > case
> > > > > of
> > > > > > > > > > > > > > > > specifying a single desired version is pretty
> > > > > > > > > counterintuitive.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I agree, and the "soft" requirement scheme is
> > > > > something I
> > > > > > > > > wasn't
> > > > > > > > > > > > > > > explicitly looking for, but would be inherited
> > from
> > > > the
> > > > > > > > > library.
> > > > > > > > > > > I'm
> > > > > > > > > > > > > > > fine with eliminating the soft requirement
> > > semantics,
> > > > > and
> > > > > > > > > having
> > > > > > > > > > > > > > > "1.0.0" behave the same as "[1.0.0]". I'm less
> > > > inclined
> > > > > > to
> > > > > > > > > > include
> > > > > > > > > > > > > > > "range" in the property name, or have two
> > > properties.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Failing startup is drastic and has the
> > potential
> > > to
> > > > > > > disrupt
> > > > > > > > > the
> > > > > > > > > > > > > > > > availability of connectors that would
> otherwise
> > > be
> > > > > able
> > > > > > > to
> > > > > > > > > run
> > > > > > > > > > > > > healthily
> > > > > > > > > > > > > > > > because they were explicitly configured to
> use
> > > > valid
> > > > > > > > > converters
> > > > > > > > > > > > > instead
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the worker defaults.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think this argument cuts both ways. If
> someone
> > > > > > > > reconfigures a
> > > > > > > > > > > worker
> > > > > > > > > > > > > > > and adds an invalid ".version" string to the
> > worker
> > > > (or
> > > > > > > > changes
> > > > > > > > > > the
> > > > > > > > > > > > > > > plugin.path to make it invalid), it would be
> > > > permitted
> > > > > to
> > > > > > > > enter
> > > > > > > > > > the
> > > > > > > > > > > > > > > group, and accept work assignments. If those
> work
> > > > > > > assignments
> > > > > > > > > > used
> > > > > > > > > > > > > > > these configurations, a set of tasks could
> > > transition
> > > > > to
> > > > > > > > FAILED
> > > > > > > > > > and
> > > > > > > > > > > > > > > not be able to recover, because they would be
> > > > restarted
> > > > > > > again
> > > > > > > > > on
> > > > > > > > > > > the
> > > > > > > > > > > > > > > same worker.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Why are metrics utilized to report
> information
> > > > about
> > > > > > > plugin
> > > > > > > > > > > versions
> > > > > > > > > > > > > > > > utilized by connectors at runtime instead of
> > > > > publishing
> > > > > > > > this
> > > > > > > > > > > info in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > REST API
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I was following the existing pattern for
> exposing
> > > > > runtime
> > > > > > > > > > versions
> > > > > > > > > > > for
> > > > > > > > > > > > > > > connectors, and it did seem like a "monitoring"
> > > > > feature.
> > > > > > If
> > > > > > > > > that
> > > > > > > > > > > > > > > approach is flawed and should be phased out, I
> > > think
> > > > it
> > > > > > > would
> > > > > > > > > be
> > > > > > > > > > a
> > > > > > > > > > > > > > > good idea to reconsider the REST API rejected
> > > > > > alternative.
> > > > > > > > > > > > > > > We would need some additional design work to
> spec
> > > out
> > > > > the
> > > > > > > > REST
> > > > > > > > > > API
> > > > > > > > > > > > > > > interface, as I don't have anything in mind
> > > > currently.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'm unclear on whether or not it'll be
> possible
> > > to
> > > > > see
> > > > > > > this
> > > > > > > > > > > > > information
> > > > > > > > > > > > > > > via the
> > > > > > > > > > > > > > > > GET /connector-plugins/<plugin>/config
> endpoint
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There is room in the API to add recommenders
> for
> > > > > > > > > "key.converter",
> > > > > > > > > > > > > > > "value.converter", and "header.converter", but
> > not
> > > > for
> > > > > > > > > transforms
> > > > > > > > > > > and
> > > > > > > > > > > > > > > predicates, as they include aliases that depend
> > on
> > > an
> > > > > > > actual
> > > > > > > > > > > > > > > configuration. We could explicitly say we're
> > going
> > > to
> > > > > do
> > > > > > > > that,
> > > > > > > > > or
> > > > > > > > > > > do
> > > > > > > > > > > > > > > whatever is convenient during the
> implementation
> > > > phase,
> > > > > > or
> > > > > > > > > leave
> > > > > > > > > > it
> > > > > > > > > > > > > > > open to be improved later.
> > > > > > > > > > > > > > > There will not be any recommenders for
> ".version"
> > > > > > > properties
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > > `/config` endpoint, because those recommenders
> > are
> > > > > > dynamic
> > > > > > > > and
> > > > > > > > > > > depend
> > > > > > > > > > > > > > > on an actual configuration.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 5) There are two relevant lines in the KIP:
> "If a
> > > > > > .version
> > > > > > > > > > property
> > > > > > > > > > > > > > > contains a hard requirement, select the latest
> > > > > installed
> > > > > > > > > version
> > > > > > > > > > > which
> > > > > > > > > > > > > > > satisfies the requirement." and "This
> > configuration
> > > > is
> > > > > > > > > > re-evaluated
> > > > > > > > > > > > > > > each time the connector or task are assigned
> to a
> > > new
> > > > > > > > worker".
> > > > > > > > > I
> > > > > > > > > > > would
> > > > > > > > > > > > > > > call this "eager" upgrade behavior, rather
> than a
> > > > > > "sticky"
> > > > > > > or
> > > > > > > > > > > "lazy"
> > > > > > > > > > > > > > > upgrade behavior.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 6) Updated!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, May 14, 2024 at 9:14 AM Chris Egerton
> > > > > > > > > > > <chrise@aiven.io.invalid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks Greg for updating the KIP, and thanks
> > > > > Snehashis
> > > > > > > for
> > > > > > > > > > > starting
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > work on this originally.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The motivation section makes a pretty
> > convincing
> > > > case
> > > > > > for
> > > > > > > > > this
> > > > > > > > > > > kind
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > feature. My thoughts are mostly about
> specific
> > > > > details:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1) I like the support for version ranges (the
> > > > example
> > > > > > > > > > > demonstrating
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > avoid KAFKA-10574 with the header converter
> was
> > > > > > > > particularly
> > > > > > > > > > > > > > > entertaining),
> > > > > > > > > > > > > > > > but the configuration syntax for the most
> basic
> > > use
> > > > > > case
> > > > > > > of
> > > > > > > > > > > > > specifying a
> > > > > > > > > > > > > > > > single desired version is pretty
> > > counterintuitive.
> > > > > > People
> > > > > > > > may
> > > > > > > > > > get
> > > > > > > > > > > > > bitten
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > at least frustrated if they put
> > > > > connector.version=3.8.0
> > > > > > > in
> > > > > > > > a
> > > > > > > > > > > config
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > then version 3.7.5 ends up running. I'd like
> it
> > > if
> > > > we
> > > > > > > could
> > > > > > > > > > > either
> > > > > > > > > > > > > > > > intentionally deviate from Maven ranges when
> a
> > > bare
> > > > > > > version
> > > > > > > > > is
> > > > > > > > > > > > > present,
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > separate things out into two properties:
> > > > foo.version
> > > > > > > would
> > > > > > > > be
> > > > > > > > > > the
> > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > accepted version for the foo plugin, and
> > > > > > > foo.version.range
> > > > > > > > > > would
> > > > > > > > > > > use
> > > > > > > > > > > > > > > Maven
> > > > > > > > > > > > > > > > range syntax. Open to other options too, just
> > > > > > providing a
> > > > > > > > > > couple
> > > > > > > > > > > to
> > > > > > > > > > > > > get
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > ball rolling.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2) Although the current behavior for a worker
> > > with
> > > > an
> > > > > > > > invalid
> > > > > > > > > > > > > > > > key/value/header converter class specified in
> > its
> > > > > > config
> > > > > > > > file
> > > > > > > > > > is
> > > > > > > > > > > a
> > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > strange (I was surprised to learn that it
> > > wouldn't
> > > > > fail
> > > > > > > on
> > > > > > > > > > > startup),
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > don't see a good reason to deviate from this
> > when
> > > > an
> > > > > > > > invalid
> > > > > > > > > > > version
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > specified. Failing startup is drastic and has
> > the
> > > > > > > potential
> > > > > > > > > to
> > > > > > > > > > > > > disrupt
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > availability of connectors that would
> otherwise
> > > be
> > > > > able
> > > > > > > to
> > > > > > > > > run
> > > > > > > > > > > > > healthily
> > > > > > > > > > > > > > > > because they were explicitly configured to
> use
> > > > valid
> > > > > > > > > converters
> > > > > > > > > > > > > instead
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the worker defaults.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 3) Why are metrics utilized to report
> > information
> > > > > about
> > > > > > > > > plugin
> > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > utilized by connectors at runtime instead of
> > > > > publishing
> > > > > > > > this
> > > > > > > > > > > info in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > REST API? I saw that this was mentioned as a
> > > > rejected
> > > > > > > > > > > alternative,
> > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > didn't get a sense of why. It seems like the
> > REST
> > > > API
> > > > > > > would
> > > > > > > > > be
> > > > > > > > > > > > > easier to
> > > > > > > > > > > > > > > > access and more intuitive for most users than
> > new
> > > > > > > metrics.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 4) In the "Validation" section it's stated
> that
> > > > > "Users
> > > > > > > can
> > > > > > > > > use
> > > > > > > > > > > these
> > > > > > > > > > > > > > > > recommenders to discover the valid plugin
> > classes
> > > > and
> > > > > > > > > versions,
> > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > requiring an earlier call to GET
> > > > > > > > > > > > > > > /connector-plugins?connectorsOnly=false."
> > > > > > > > > > > > > > > > I really like the creativity and simplicity
> of
> > > > > reusing
> > > > > > > the
> > > > > > > > > > > > > recommender
> > > > > > > > > > > > > > > > mechanism to expose available versions for
> > > plugins
> > > > > via
> > > > > > > the
> > > > > > > > > REST
> > > > > > > > > > > API.
> > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > unclear on whether or not it'll be possible
> to
> > > see
> > > > > this
> > > > > > > > > > > information
> > > > > > > > > > > > > via
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > GET /connector-plugins/<plugin>/config
> > endpoint,
> > > > > > though.
> > > > > > > > It'd
> > > > > > > > > > be
> > > > > > > > > > > > > great if
> > > > > > > > > > > > > > > > this were supported, since we learned in
> > KIP-769
> > > > [1]
> > > > > > that
> > > > > > > > > > people
> > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > want to be able to see configuration options
> > for
> > > > > > > connectors
> > > > > > > > > and
> > > > > > > > > > > their
> > > > > > > > > > > > > > > > plugins via some kind of GET endpoint without
> > > > having
> > > > > to
> > > > > > > > > > provide a
> > > > > > > > > > > > > > > complete
> > > > > > > > > > > > > > > > connector config for validation.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 5) In the Maven version range docs, it's
> stated
> > > > that
> > > > > > > "Maven
> > > > > > > > > > > picks the
> > > > > > > > > > > > > > > > highest version of each project that
> satisfies
> > > all
> > > > > the
> > > > > > > hard
> > > > > > > > > > > > > requirements
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the dependencies on that project." I'm
> guessing
> > > > this
> > > > > > > > behavior
> > > > > > > > > > > will be
> > > > > > > > > > > > > > > > retained for Connect; i.e., the
> > highest-possible
> > > > > > version
> > > > > > > of
> > > > > > > > > > each
> > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > that satisfies the user-specified version
> > > > constraints
> > > > > > > will
> > > > > > > > be
> > > > > > > > > > > run?
> > > > > > > > > > > > > (An
> > > > > > > > > > > > > > > > alternative approach could be to have some
> kind
> > > of
> > > > > > > "sticky"
> > > > > > > > > > logic
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > restarts connectors/tasks when their
> > > currently-used
> > > > > > > version
> > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > incompatible with the configured
> constraints.)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 6) (Nit) It'd be nice to add a link to the
> > > > > TestPlugins
> > > > > > > > class
> > > > > > > > > or
> > > > > > > > > > > > > somewhere
> > > > > > > > > > > > > > > > in its neighborhood to the testing plan;
> > > unfamiliar
> > > > > > > readers
> > > > > > > > > > > probably
> > > > > > > > > > > > > > > won't
> > > > > > > > > > > > > > > > get much out of what's there right now.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > [1] -
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, May 13, 2024 at 2:01 PM Snehashis <
> > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > That is much appreciated. No complaints on
> > the
> > > > > > > additional
> > > > > > > > > > > scope, I
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > make some time out to work on this once we
> > have
> > > > > > > approval.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> > > > > > > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hey Snehashis,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'm glad to hear you're still interested
> in
> > > > this
> > > > > > KIP!
> > > > > > > > > > > > > > > > > > I'm happy to let you drive this, and I
> > > > apologize
> > > > > > for
> > > > > > > > > > > increasing
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > scope of work so drastically. To make up
> > for
> > > > > that,
> > > > > > > I'll
> > > > > > > > > > > > > volunteer to
> > > > > > > > > > > > > > > > > > be the primary PR reviewer to help get
> this
> > > > done
> > > > > > > > quickly
> > > > > > > > > > once
> > > > > > > > > > > > > the KIP
> > > > > > > > > > > > > > > > > > is approved.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 3:51 AM
> Snehashis <
> > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for the follow up to my original
> > > KIP,
> > > > I
> > > > > am
> > > > > > > in
> > > > > > > > > > > favour of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > additions made to expand its scope, the
> > > > > addition
> > > > > > of
> > > > > > > > > range
> > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > specifically make a lot of sense.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Apologies if I have not publicly worked
> > on
> > > > this
> > > > > > KIP
> > > > > > > > > for a
> > > > > > > > > > > long
> > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > original work was done when the move to
> > > > service
> > > > > > > > loading
> > > > > > > > > > > was in
> > > > > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > > > > and I wanted to loop back to this only
> > > after
> > > > > that
> > > > > > > > work
> > > > > > > > > > was
> > > > > > > > > > > > > > > completed.
> > > > > > > > > > > > > > > > > > Post
> > > > > > > > > > > > > > > > > > > its conclusion, I have not been able to
> > > take
> > > > > this
> > > > > > > up
> > > > > > > > > due
> > > > > > > > > > to
> > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > priorities. If it's okay with you, I
> > would
> > > > > still
> > > > > > > like
> > > > > > > > > to
> > > > > > > > > > > get
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > implemented myself, including the
> > > additional
> > > > > > scope.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks and regards
> > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 12:45 AM Greg
> > > Harris
> > > > > > > > > > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I'd like to reboot the discussion on
> > > > KIP-891:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I've made some changes, most notably:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. Specifying versions for all
> plugins
> > in
> > > > > > > Connector
> > > > > > > > > > > configs
> > > > > > > > > > > > > > > > > > > > (converters, header converters,
> > > transforms,
> > > > > and
> > > > > > > > > > > predicates)
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > > connectors & tasks
> > > > > > > > > > > > > > > > > > > > 2. Specifying a range of versions
> > instead
> > > > of
> > > > > an
> > > > > > > > exact
> > > > > > > > > > > match
> > > > > > > > > > > > > > > > > > > > 3. New metrics to observe what
> versions
> > > are
> > > > > > > in-use
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks to Snehashis for the original
> > KIP
> > > > > idea!
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg
> > > > Harris <
> > > > > > > > > > > > > > > greg.harris@aiven.io>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Snehashis,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thank you for the KIP! This is
> > > something
> > > > > I've
> > > > > > > > > wanted
> > > > > > > > > > > for a
> > > > > > > > > > > > > long
> > > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I know the discussion has gone
> cold,
> > > are
> > > > > you
> > > > > > > > still
> > > > > > > > > > > > > interested
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > pursuing this feature? I'll make
> time
> > > to
> > > > > > review
> > > > > > > > the
> > > > > > > > > > > KIP if
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > > still accepting comments.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM
> > > > Snehashis
> > > > > <
> > > > > > > > > > > > > > > > > snehashisp1995@gmail.com
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for the points Sagar.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1) Should we update the GET
> > > > /connectors
> > > > > > > > > endpoint
> > > > > > > > > > to
> > > > > > > > > > > > > > > include the
> > > > > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > > > > the plugin that is running? It
> > > could
> > > > be
> > > > > > > > useful
> > > > > > > > > to
> > > > > > > > > > > > > figure
> > > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > the plugin or I am assuming it
> > gets
> > > > > > > returned
> > > > > > > > by
> > > > > > > > > > the
> > > > > > > > > > > > > > > expand=info
> > > > > > > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think this is good to have and
> > > > possible
> > > > > > > > future
> > > > > > > > > > > > > > > enhancement. The
> > > > > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > > > info will be present in the
> config
> > of
> > > > the
> > > > > > > > > connector
> > > > > > > > > > > if
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > > > > specified the version. Otherwise
> it
> > > is
> > > > > the
> > > > > > > > latest
> > > > > > > > > > > version
> > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > > > > can find out from the
> > > connector-plugin
> > > > > > > > endpoint.
> > > > > > > > > > The
> > > > > > > > > > > > > > > information
> > > > > > > > > > > > > > > > > > can be
> > > > > > > > > > > > > > > > > > > > > > introduced to the response of the
> > GET
> > > > > > > > /connectors
> > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > itself,
> > > > > > > > > > > > > > > > > > > > however
> > > > > > > > > > > > > > > > > > > > > > the most ideal way of doing this
> > > would
> > > > be
> > > > > > to
> > > > > > > > get
> > > > > > > > > > the
> > > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > > > > > instance of the connector and get
> > the
> > > > > > version
> > > > > > > > > > > directly
> > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > there.
> > > > > > > > > > > > > > > > > > > > This is
> > > > > > > > > > > > > > > > > > > > > > slightly tricky as the connector
> > > could
> > > > be
> > > > > > > > running
> > > > > > > > > > in
> > > > > > > > > > > a
> > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > node.
> > > > > > > > > > > > > > > > > > > > > > One way to do this would be to
> > > persist
> > > > > the
> > > > > > > > > version
> > > > > > > > > > > > > > > information in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > status backing store during
> > > > instantiation
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > > connector.
> > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > requires
> > > > > > > > > > > > > > > > > > > > > > some more thought and since the
> > > version
> > > > > is
> > > > > > > part
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > > > configs if
> > > > > > > > > > > > > > > > > > > > provided
> > > > > > > > > > > > > > > > > > > > > > and evident otherwise, I have not
> > > > > included
> > > > > > it
> > > > > > > > in
> > > > > > > > > > this
> > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2) I am not aware of this and
> > hence
> > > > > > asking,
> > > > > > > > > can 2
> > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > > > versions have the same name?
> Does
> > > the
> > > > > > > plugin
> > > > > > > > > > > isolation
> > > > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > > > > could have a bearing when using
> > the
> > > > > > > lifecycle
> > > > > > > > > > > > > endpoints for
> > > > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > All connectors in a cluster need
> to
> > > > have
> > > > > > > > uniquire
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > names
> > > > > > > > > > > > > > > > > > > > > > regardless of what version of the
> > > > plugin
> > > > > > the
> > > > > > > > > > > connector is
> > > > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > > > > > underneath. This is something
> > > enforced
> > > > by
> > > > > > the
> > > > > > > > > > connect
> > > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > > itself.
> > > > > > > > > > > > > > > > > > > > All
> > > > > > > > > > > > > > > > > > > > > > connect CRUD operations are keyed
> > on
> > > > the
> > > > > > > > > connector
> > > > > > > > > > > name
> > > > > > > > > > > > > so
> > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > be an issue.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM
> > > Sagar <
> > > > > > > > > > > > > > > sagarmeansocean@gmail.com
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hey Snehashsih,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP. It looks
> > like a
> > > > > very
> > > > > > > > useful
> > > > > > > > > > > > > feature.
> > > > > > > > > > > > > > > Couple
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > small-ish points, let me know
> > what
> > > > you
> > > > > > > think:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1) Should we update the GET
> > > > /connectors
> > > > > > > > > endpoint
> > > > > > > > > > to
> > > > > > > > > > > > > > > include the
> > > > > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > > > > the plugin that is running? It
> > > could
> > > > be
> > > > > > > > useful
> > > > > > > > > to
> > > > > > > > > > > > > figure
> > > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > > > > the plugin or I am assuming it
> > gets
> > > > > > > returned
> > > > > > > > by
> > > > > > > > > > the
> > > > > > > > > > > > > > > expand=info
> > > > > > > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > > > > > > > 2) I am not aware of this and
> > hence
> > > > > > asking,
> > > > > > > > > can 2
> > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > > > versions have the same name?
> Does
> > > the
> > > > > > > plugin
> > > > > > > > > > > isolation
> > > > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > > > > could have a bearing when using
> > the
> > > > > > > lifecycle
> > > > > > > > > > > > > endpoints for
> > > > > > > > > > > > > > > > > > > > connectors like
> > > > > > > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > > > > > Sagar.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM
> > > > Ashwin
> > > > > > > > > > > > > > > > > > <apankaj@confluent.io.invalid
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > IIUC (please correct me if
> I
> > am
> > > > > wrong
> > > > > > > > > here),
> > > > > > > > > > > what
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > a versioning scheme for a
> > > connector
> > > > > > > config
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > different versions of a
> > connector
> > > > > > > plugin).
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Sorry for not being more
> > precise
> > > in
> > > > > my
> > > > > > > > > wording
> > > > > > > > > > > - I
> > > > > > > > > > > > > meant
> > > > > > > > > > > > > > > > > > > > registering
> > > > > > > > > > > > > > > > > > > > > > > > versions of schema for
> > connector
> > > > > > config.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Let's take the example of a
> > > > fictional
> > > > > > > > > connector
> > > > > > > > > > > which
> > > > > > > > > > > > > > > uses a
> > > > > > > > > > > > > > > > > > > > fictional
> > > > > > > > > > > > > > > > > > > > > > > AWS
> > > > > > > > > > > > > > > > > > > > > > > > service.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Fictional Connector Config
> > schema
> > > > > > > > version:2.0
> > > > > > > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > > > > > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Fictional Connector config
> > schema
> > > > > > > > version:3.0
> > > > > > > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > "iam_role": {
> > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > > > > > > "iam_role"
> > > > > > > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The connector which supports
> > > > > Fictional
> > > > > > > > config
> > > > > > > > > > > schema
> > > > > > > > > > > > > 2.0
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > validate
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > access key and secret key.
> > > > > > > > > > > > > > > > > > > > > > > > Whereas a connector which
> > > supports
> > > > > > config
> > > > > > > > > with
> > > > > > > > > > > schema
> > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > 3.0
> > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > > > > > > > validate the IAM role.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This is the alternative
> which I
> > > > > wanted
> > > > > > to
> > > > > > > > > > > suggest.
> > > > > > > > > > > > > Each
> > > > > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > register the schema versions
> of
> > > > > > connector
> > > > > > > > > > config
> > > > > > > > > > > > > which it
> > > > > > > > > > > > > > > > > > supports.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The plugin paths may be
> > > optionally
> > > > > > > > different
> > > > > > > > > > > i.e we
> > > > > > > > > > > > > > > don't
> > > > > > > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > > > > > > > mandatorily add a new plugin
> > path
> > > > to
> > > > > > > > support
> > > > > > > > > a
> > > > > > > > > > > new
> > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > version.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:47
> > PM
> > > > > > > Snehashis
> > > > > > > > <
> > > > > > > > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the input
> Ashwin.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on
> the
> > > > > > rejected
> > > > > > > > > > > > > alternatives ?
> > > > > > > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > > config is versioned and
> > has a
> > > > > > schema.
> > > > > > > > > Then
> > > > > > > > > > a
> > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > > > > > > dependencies have not
> > > changed)
> > > > > can
> > > > > > > > handle
> > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > IIUC (please correct me if
> I
> > am
> > > > > wrong
> > > > > > > > > here),
> > > > > > > > > > > what
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > a versioning scheme for a
> > > > connector
> > > > > > > > config
> > > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > > connector (and
> > > > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > > different versions of a
> > > connector
> > > > > > > > plugin).
> > > > > > > > > > > That is
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > somewhat
> > > > > > > > > > > > > > > > > > > > > > > tangential
> > > > > > > > > > > > > > > > > > > > > > > > > problem. While it is
> > > definitely a
> > > > > > > useful
> > > > > > > > > > > feature to
> > > > > > > > > > > > > > > have,
> > > > > > > > > > > > > > > > > > like a
> > > > > > > > > > > > > > > > > > > > log to
> > > > > > > > > > > > > > > > > > > > > > > > > check what changes were
> made
> > > over
> > > > > > time
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > > config
> > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > easier to do rollbacks, it
> is
> > > not
> > > > > the
> > > > > > > > focus
> > > > > > > > > > > here.
> > > > > > > > > > > > > Here
> > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > version we
> > > > > > > > > > > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > > > > > > > > > to say what underlying
> > version
> > > of
> > > > > the
> > > > > > > > > plugin
> > > > > > > > > > > > > should the
> > > > > > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > > of the connector use.
> Perhaps
> > > it
> > > > is
> > > > > > > > better
> > > > > > > > > to
> > > > > > > > > > > > > change
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > name of
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > parameter from
> > > connector.version
> > > > to
> > > > > > > > > > > > > > > > > connector.plugin.version
> > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > > plugin.version if it was
> > > > confusing.
> > > > > > > wdyt?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2. Any plans to support
> > > > assisted
> > > > > > > > > migration
> > > > > > > > > > > e.g
> > > > > > > > > > > > > if a
> > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > connector/config?migrate=latest",
> > > > > > the
> > > > > > > > > > latest
> > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > > > > > > transform the existing
> > config
> > > > to
> > > > > > the
> > > > > > > > > newer
> > > > > > > > > > > > > version.
> > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > > > adding a method like
> > "boolean
> > > > > > > > > > migrate(Version
> > > > > > > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This is an enhancement we
> can
> > > > think
> > > > > > of
> > > > > > > > > doing
> > > > > > > > > > in
> > > > > > > > > > > > > future.
> > > > > > > > > > > > > > > > > > Users can
> > > > > > > > > > > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > > > > > > > > a PUT call with the updated
> > > > config
> > > > > > > which
> > > > > > > > > has
> > > > > > > > > > > the
> > > > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > > > > number.
> > > > > > > > > > > > > > > > > > > > > > > > > The assisted mode could be
> > > handy
> > > > as
> > > > > > the
> > > > > > > > > user
> > > > > > > > > > > does
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > know the
> > > > > > > > > > > > > > > > > > > > > > > > > config but beyond this it
> > does
> > > > not
> > > > > > seem
> > > > > > > > to
> > > > > > > > > > > justify
> > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > > existence.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at
> 10:50
> > > AM
> > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This is a really useful
> > > feature
> > > > > and
> > > > > > > > > thanks
> > > > > > > > > > > for
> > > > > > > > > > > > > > > initiating
> > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I had the following
> > > questions -
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on
> the
> > > > > > rejected
> > > > > > > > > > > > > alternatives ?
> > > > > > > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > > config is versioned and
> > has a
> > > > > > schema.
> > > > > > > > > Then
> > > > > > > > > > a
> > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > > > > > > dependencies have not
> > > changed)
> > > > > can
> > > > > > > > handle
> > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2. Any plans to support
> > > > assisted
> > > > > > > > > migration
> > > > > > > > > > > e.g
> > > > > > > > > > > > > if a
> > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > connector/config?migrate=latest",
> > > > > > the
> > > > > > > > > > latest
> > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > > > > > > transform the existing
> > config
> > > > to
> > > > > > the
> > > > > > > > > newer
> > > > > > > > > > > > > version.
> > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > > > adding a method like
> > "boolean
> > > > > > > > > > migrate(Version
> > > > > > > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 21, 2022 at
> > 2:27
> > > PM
> > > > > > > > > Snehashis <
> > > > > > > > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I'd like to start a
> > > > discussion
> > > > > > > thread
> > > > > > > > > on
> > > > > > > > > > > > > KIP-891:
> > > > > > > > > > > > > > > > > Running
> > > > > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > The KIP aims to add the
> > > > ability
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > connect
> > > > > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > > > > > > versions of a
> connector.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Please take a look and
> > let
> > > me
> > > > > > know
> > > > > > > > what
> > > > > > > > > > you
> > > > > > > > > > > > > think.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Snehashis <sn...@gmail.com>.
Hi Greg,
Thanks for the clarification!
I agree with not breaking compatibility and opting to not fail worker
startup by validating converter versions.
Let's also go with the unclosed variation of the requirements as a hard
requirement i.e. 3.8 instead of [3.8]. I have updated the KIP and added a
line there highlighting this.
Regards
Snehashis
On Wed, Aug 21, 2024 at 9:24 PM Greg Harris <gr...@aiven.io.invalid>
wrote:
> Hey Snehashis,
>
> Thanks for your reply!
>
> > Deviating
> > it from the spec seems unnecessary if we document it accordingly, however
> > It's probably less intuitive and can lead to confusion. I would just keep
> > it as is but making it simpler is also fine.
>
> It sounds like you don't have a strong opinion on this, similar to me.
> Chris had a more firm stance on this. I think that you're right that we
> could document this, but it would still be the single biggest foot-gun of
> the feature, and I think we would regret it later.
>
> > Also for converter versions
> > specified as part of the worker configs I believe we concluded that this
> > step need not be fatal during worker startup if the required version is
> not
> > found but LMK if otherwise.
>
> Re-reading our earlier discussion, I think Chris had a strong opinion that
> we shouldn't fail on startup because it would be inconsistent. I made an
> offhand comment that if this was released in 4.0, we could change it so
> both invalid classes and invalid versions cause the worker to fail, which
> would be inconsistent but backwards incompatible.
> I think in the interest of not breaking compatibility needlessly and
> keeping consistent behavior, we should ignore invalid versions in the
> worker config.
>
> Thanks,
> Greg
>
> On Wed, Aug 21, 2024 at 1:58 AM Snehashis <sn...@gmail.com>
> wrote:
>
> > Hi Greg,
> >
> > No issues, I have been caught up in a few things myself.
> >
> > I have added the points we discussed. In addition, I have added config
> > providers as part of the set of plugins which will not support
> versioning,
> > for the same reason as to why it is not supported in the other plugins
> that
> > are initiated on startup.
> >
> > On whether to deviate from maven versioning for hard requirements as
> > discussed between you and Chris.Whether to simply simply specify
> > connector.version=1.1.1 as a hard requirement instead of [1.1.1].
> Deviating
> > it from the spec seems unnecessary if we document it accordingly, however
> > It's probably less intuitive and can lead to confusion. I would just keep
> > it as is but making it simpler is also fine. Also for converter versions
> > specified as part of the worker configs I believe we concluded that this
> > step need not be fatal during worker startup if the required version is
> not
> > found but LMK if otherwise.
> >
> > Regards
> > Snehashis
> >
> >
> >
> > On Mon, Aug 19, 2024 at 9:24 PM Greg Harris <greg.harris@aiven.io.invalid
> >
> > wrote:
> >
> > > Hi Snehashis,
> > >
> > > Sorry for the late reply.
> > >
> > > > Heterogeneous dependencies in a multi cluster deployment is highly
> > > discouraged
> > >
> > > Right, this remains unchanged in this KIP.
> > >
> > > > Let's add the version information for both connector and tasks in the
> > > connector status itself
> > > > once we add these two additions to the KIP (LMK if you want me to
> take
> > > that up).
> > >
> > > Could you make these additions?
> > >
> > > I'm interested to see if we can include this in 4.0.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Tue, Jul 2, 2024 at 2:52 AM Snehashis <sn...@gmail.com>
> > wrote:
> > >
> > > > Hi Greg,
> > > >
> > > > Thanks for taking a look at this, to conclude on the two points
> above.
> > > >
> > > > 1. I'm okay with the status quo of leaving the dependency management
> of
> > > > plugins to systems outside of the connect runtime as it is now. Given
> > > that
> > > > the dependencies are homogenous across a connect cluster, it should
> > > ensure
> > > > that task and connector versions are uniform across a deployment,
> after
> > > it
> > > > has stabilised post an upgrade operation. Heterogeneous dependencies
> > in a
> > > > multi cluster deployment is highly discouraged and we should point
> out
> > > this
> > > > can lead to inconsistent behaviour with connectors and even
> > validations.
> > > >
> > > > 2. Let's add the version information for both connector and tasks in
> > the
> > > > connector status itself, I don't see another endpoint being required
> > for
> > > > this.
> > > >
> > > > I don't have any further points, so if you are okay we can put this
> to
> > a
> > > > vote, once we add these two additions to the KIP (LMK if you want me
> to
> > > > take that up).
> > > >
> > > > Regards
> > > > Snehashis
> > > >
> > > > On Tue, Jul 2, 2024 at 12:08 AM Greg Harris
> > <greg.harris@aiven.io.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hey Snehashis,
> > > > >
> > > > > Sorry for the late reply, and thanks for helping close out the
> > > > discussion.
> > > > >
> > > > > > Note that if my assumptions are correct then
> > > > > > this can happen with the existing framework as well, or is there
> > some
> > > > > > safeguard from this happening?
> > > > >
> > > > > Currently, if a cluster has a heterogeneous plugin installation,
> each
> > > > > worker may have a different definition of "latest". If you call to
> > > > > validate a configuration, it will validate against whatever latest
> > > > version
> > > > > is present locally.
> > > > > With this KIP, that behavior will continue when the version isn't
> > > > > specified, when the version is a soft requirement, or when the
> > version
> > > > is a
> > > > > range. For users that do not want to tolerate their tasks having
> > > > > heterogeneous versions, the single-version hard requirements are
> > there,
> > > > but
> > > > > come at the cost of micromanagement of upgrades.
> > > > > Perhaps there is some room in the future for a connector configured
> > > with
> > > > a
> > > > > range to "pin" the version that all of its tasks should use. Or
> > > > > version-aware-scheduling can schedule connectors and tasks to
> workers
> > > > that
> > > > > will resolve the same versions. The important primitive is being
> able
> > > to
> > > > > enforce which version is being used, everything else looks more
> like
> > a
> > > > > convenience feature to me.
> > > > >
> > > > > > So far, we could have pointed to the
> > > > > > misconfigured cluster configuration and somewhat differ this
> > problem
> > > to
> > > > > > something outside of connect runtime.
> > > > >
> > > > > I think this will still be the case. Even with this feature, we
> will
> > > > still
> > > > > recommend homogeneous clusters that have the same set of plugins
> > > > installed
> > > > > on every worker. Version-aware scheduling is a rejected
> alternative,
> > > and
> > > > > your example for performing validation requests on an arbitrary
> > worker
> > > > is a
> > > > > great example of a complication for heterogeneous clusters that
> we're
> > > not
> > > > > ready to solve.
> > > > >
> > > > > > We can introduce
> > > > > > a new path under this for version
> > > (/connector/connector-name/version),
> > > > > but
> > > > > > perhaps adding this as part of the status is a valid alternative.
> > > > >
> > > > > I think either is fine. Adding a new field to the existing
> endpoints
> > is
> > > > > fine, and shouldn't require special compatibility such as a query
> > > > > parameter.
> > > > >
> > > > > > Also, to go further I think
> > > > > > version information for tasks could also be available,
> > > > >
> > > > > I completely agree. The version of each connector and each task
> > should
> > > be
> > > > > visible individually, since they may be different during a rolling
> > > > upgrade.
> > > > >
> > > > > Thanks!
> > > > > Greg
> > > > >
> > > > > On Tue, Jun 18, 2024 at 5:18 AM Snehashis <
> snehashisp1995@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Greg, Chris
> > > > > >
> > > > > > Thanks for the in-depth discussion, I have a couple of discussion
> > > > points
> > > > > > and would like your thoughts on this.
> > > > > >
> > > > > > 1) One concern I have with the new addition of 'soft' and 'hard'
> > > > version
> > > > > > requirements is that there could be a mismatch in the plugin
> > version
> > > > that
> > > > > > two different tasks are running, if a soft requirement is
> provided
> > > and
> > > > > the
> > > > > > nodes a multi cluster deployment are not in sync w.r.t the plugin
> > > > > versions
> > > > > > that they are configured with. Note that if my assumptions are
> > > correct
> > > > > then
> > > > > > this can happen with the existing framework as well, or is there
> > some
> > > > > > safeguard from this happening? So far, we could have pointed to
> the
> > > > > > misconfigured cluster configuration and somewhat differ this
> > problem
> > > to
> > > > > > something outside of connect runtime. With this feature in place
> > > > perhaps
> > > > > > the expectation is more on connect to not be running with such
> > > > > > inconsistency, especially if a connector version is specified.
> This
> > > is
> > > > > also
> > > > > > a problem with validation if different cluster have different
> > > > > > configurations, as IIRC validations are local to the worker which
> > > > > receives
> > > > > > the rest call for validate. So, we might be validating with a
> > certain
> > > > > > version which is different from the one that will be used to
> create
> > > > > > connector and tasks. Again, this is likely how the current state
> > is,
> > > > but
> > > > > > perhaps such inconsistencies warrant a deeper look with the
> > addition
> > > of
> > > > > > this feature. The problems associated with them can be somewhat
> > > > insidious
> > > > > > and hard to diagnose.
> > > > > >
> > > > > > 2) There was some discussion on the need for a new REST endpoint
> to
> > > > > provide
> > > > > > information on the versions of running connectors, and I think
> > adding
> > > > > this
> > > > > > information via REST is a valuable addition. The way I see it the
> > > > version
> > > > > > is an intrinsic property of an instance of a running connector
> and
> > > > hence
> > > > > > this should be part of the set of APIs under
> > > > /connector/<connector-name>
> > > > > > (also the /connectors API should also have this information as it
> > is
> > > an
> > > > > > amalgamation of all the individual connector information). We can
> > > > > introduce
> > > > > > a new path under this for version
> > > (/connector/connector-name/version),
> > > > > but
> > > > > > perhaps adding this as part of the status is a valid alternative.
> > > This
> > > > is
> > > > > > mentioned as a rejected alternative right now. Also, to go
> further
> > I
> > > > > think
> > > > > > version information for tasks could also be available, especially
> > if
> > > we
> > > > > > choose to not address the pitfalls discussed in my point 1), this
> > > will
> > > > > > at-least provide admins a quick and easy way to determine if such
> > and
> > > > > > inconsistent state exist in any of the connectors.
> > > > > >
> > > > > > Thanks again for reviving my original KIP and working to improve
> > it.
> > > > > > Looking forward to your thoughts on the points mentioned above.
> > > > > > Regards
> > > > > > Snehashis
> > > > > >
> > > > > >
> > > > > > On Wed, May 29, 2024 at 9:59 PM Chris Egerton
> > > <chrise@aiven.io.invalid
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Greg,
> > > > > > >
> > > > > > > First, an apology! I mistakenly assumed that each plugin
> appeared
> > > > only
> > > > > > once
> > > > > > > in the responses from GET
> > /connector-plugins?connectorsOnly=false.
> > > > > Thank
> > > > > > > you for correcting me and pointing out that all versions of
> each
> > > > plugin
> > > > > > > appear in that response, which does indeed satisfy my desire
> for
> > > > users
> > > > > to
> > > > > > > discover this information in at most two REST requests (and in
> > > fact,
> > > > > does
> > > > > > > it in only one)!
> > > > > > >
> > > > > > > And secondly, with the revelation about recommenders, I agree
> > that
> > > > it's
> > > > > > > best to leave the "version" property out of the lists of
> > properties
> > > > > > > returned from the GET /connector-plugins/<plugin>/config
> > endpoint.
> > > > > > >
> > > > > > > With those two points settled, I think the only unresolved item
> > is
> > > > the
> > > > > > > small change to version parsing added to the KIP (where raw
> > version
> > > > > > numbers
> > > > > > > are treated as an exact match, instead of a best-effort match
> > with
> > > a
> > > > > > > fallback on the default version). If the KIP is updated with
> that
> > > > then
> > > > > > I'd
> > > > > > > be ready to vote on it.
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Wed, May 29, 2024 at 12:00 PM Greg Harris
> > > > > > <greg.harris@aiven.io.invalid
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey Chris,
> > > > > > > >
> > > > > > > > Thanks for your thoughts.
> > > > > > > >
> > > > > > > > > Won't it still only expose the
> > > > > > > > > latest version for each plugin, instead of the range of
> > > versions
> > > > > > > > available?
> > > > > > > >
> > > > > > > > Here is a snippet of the current output of the GET
> > > > > > > > /connector-plugins?connectorsOnly=false endpoint, after I
> > > installed
> > > > > two
> > > > > > > > versions of the debezium PostgresConnector:
> > > > > > > >
> > > > > > > > {
> > > > > > > > "class":
> > > "io.debezium.connector.postgresql.PostgresConnector",
> > > > > > > > "type": "source",
> > > > > > > > "version": "2.0.1.Final"
> > > > > > > > },
> > > > > > > > {
> > > > > > > > "class":
> > > "io.debezium.connector.postgresql.PostgresConnector",
> > > > > > > > "type": "source",
> > > > > > > > "version": "2.6.1.Final"
> > > > > > > > },
> > > > > > > >
> > > > > > > > I think this satisfies your requirement to learn about all
> > > plugins
> > > > > and
> > > > > > > all
> > > > > > > > versions in two or fewer REST calls.
> > > > > > > >
> > > > > > > > I tried to get an example of the output of `/config` by
> > > hardcoding
> > > > > the
> > > > > > > > Recommender, and realized that Recommenders aren't executed
> on
> > > the
> > > > > > > > `/config` endpoint at all: only during validation, when a
> > > > > configuration
> > > > > > > is
> > > > > > > > actually present.
> > > > > > > > And this led me to discover that the `/config` endpoint
> > returns a
> > > > > > > > List<ConfigKeyInfo>, and ConfigKeyInfo does not contain a
> > > > > > > recommendedValues
> > > > > > > > field. The ConfigValue field is the object which contains
> > > > > > > > recommendedValues, and it is only generated during
> validation.
> > > > > > > > I think it's out of scope to start calling recommenders on
> > empty
> > > > > > configs
> > > > > > > > that might throw exceptions, changing the existing REST
> > entities,
> > > > or
> > > > > > > > changing the core ConfigDef implementation.
> > > > > > > > Someone could add this functionality later, I don't think
> it's
> > > > > > necessary
> > > > > > > > here.
> > > > > > > >
> > > > > > > > Then the question is: should "version" without recommenders
> > > appear
> > > > in
> > > > > > > > non-connector plugins? I think I'd rather be consistent with
> > > > > > "predicate"
> > > > > > > > and "negate" on release, and let a later improvement add
> them.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Greg
> > > > > > > >
> > > > > > > > On Wed, May 29, 2024 at 8:06 AM Chris Egerton
> > > > > <chrise@aiven.io.invalid
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Greg,
> > > > > > > > >
> > > > > > > > > I'm confused about the behavior for GET
> > > > > > > > > /connector-plugins?connectorsOnly=false. Won't it still
> only
> > > > expose
> > > > > > the
> > > > > > > > > latest version for each plugin, instead of the range of
> > > versions
> > > > > > > > available?
> > > > > > > > >
> > > > > > > > > I'm hoping we can provide a flow where people need at most
> > two
> > > > REST
> > > > > > > calls
> > > > > > > > > to discover 1) the complete set of plugins available on the
> > > > worker
> > > > > > > (which
> > > > > > > > > is already possible with the endpoint under discussion) and
> > 2)
> > > > the
> > > > > > set
> > > > > > > of
> > > > > > > > > versions available for a specific plugin on the worker
> (which
> > > > > doesn't
> > > > > > > > > appear to be possible, at least for some plugin types).
> This
> > > > > wouldn't
> > > > > > > > > require any out-of-band knowledge and would be valuable for
> > > > > connector
> > > > > > > > users
> > > > > > > > > (who may want to, for example, know what their options are
> > when
> > > > > > > > considering
> > > > > > > > > a plugin upgrade) and cluster administrators (who could use
> > it
> > > > as a
> > > > > > > > sanity
> > > > > > > > > check for the setup of their Kafka Connect clusters without
> > > > having
> > > > > to
> > > > > > > > pore
> > > > > > > > > over log files).
> > > > > > > > >
> > > > > > > > > As far as modifying the content of the GET
> > > > > > > > > /connector-plugins/<plugin>/config endpoint to include a
> > > > "version"
> > > > > > > > property
> > > > > > > > > goes, I think your point about returning that property from
> > > > > requests
> > > > > > to
> > > > > > > > > that endpoint that include a version query parameter is
> > > salient,
> > > > > but
> > > > > > it
> > > > > > > > > also unfortunately applies to all types of plugin. I don't
> > > think
> > > > it
> > > > > > > > should
> > > > > > > > > completely disqualify that option. I also wasn't imagining
> > > adding
> > > > > > > > anything
> > > > > > > > > besides that single property to that endpoint, so no new
> > > > > "predicate"
> > > > > > > > > property for SMTs, no new "negate" property for predicates,
> > and
> > > > no
> > > > > > new
> > > > > > > > > "type" property for either. I'm not necessarily opposed to
> > > adding
> > > > > > > those,
> > > > > > > > > but it can be done without being pulled into the scope of
> > this
> > > > KIP.
> > > > > > > > >
> > > > > > > > > So to summarize how I imagine people might use the REST API
> > > after
> > > > > > this
> > > > > > > > KIP:
> > > > > > > > >
> > > > > > > > > 1. Discover the set of plugins on the worker via GET
> > > > > > > > > /connector-plugins?connectorsOnly=false
> > > > > > > > > 2. For any of those plugins, discover the set of available
> > > > versions
> > > > > > by
> > > > > > > > > examining the recommended values for the "version" property
> > > after
> > > > > > > hitting
> > > > > > > > > the GET GET /connector-plugins/<plugin>/config endpoint
> > > > > > > > >
> > > > > > > > > I don't think this is necessarily optimal since plucking
> the
> > > > > > "version"
> > > > > > > > > property out of the response from that endpoint might be a
> > PITA
> > > > for
> > > > > > ad
> > > > > > > > hoc
> > > > > > > > > queries via, e.g., curl, but I think it'd be enough for
> this
> > > KIP.
> > > > > > Still
> > > > > > > > > open to other options though, and if I've misunderstood
> > > anything
> > > > in
> > > > > > the
> > > > > > > > > proposal, please let me know!
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Wed, May 22, 2024 at 4:05 PM Greg Harris
> > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hey Chris,
> > > > > > > > > >
> > > > > > > > > > Thanks for your comments, and I'm glad that it seems like
> > > we're
> > > > > > > > > > aligning on the vision here.
> > > > > > > > > >
> > > > > > > > > > > An
> > > > > > > > > > > alternative could be to change existing behavior to
> fail
> > > fast
> > > > > on
> > > > > > > any
> > > > > > > > > > > invalid default converter configuration instead of just
> > for
> > > > > > invalid
> > > > > > > > > > > versions
> > > > > > > > > >
> > > > > > > > > > I suppose if this is landing in 4.0, we have the
> > opportunity
> > > to
> > > > > > break
> > > > > > > > > > compatibility and strictly validate the worker class
> > configs,
> > > > and
> > > > > > > > > > could have a new consistent behavior instead of
> > inconsistent
> > > > but
> > > > > > > > > > backwards-compatible behavior.
> > > > > > > > > >
> > > > > > > > > > > no other part of the KIP requires this change.
> > > > > > > > > >
> > > > > > > > > > This is correct, and a compelling argument. I'm fine
> > leaving
> > > > the
> > > > > > > > > > strict worker validation off of this configuration to
> > > > potentially
> > > > > > be
> > > > > > > > > > added later. If a KIP was raised to perform strict
> > validation
> > > > of
> > > > > > the
> > > > > > > > > > worker config, it would include the version config, and
> can
> > > > > address
> > > > > > > > > > backwards compatibility for both configs together.
> > > > > > > > > >
> > > > > > > > > > > RE exposing the version property in the
> > > > > > > > > > /connector-plugins/<plugin>/config
> > > > > > > > > > > endpoint, the behavior is inconsistent across plugin
> > types.
> > > > > > > > > >
> > > > > > > > > > Yes it is, and that started in a bugfix, once people
> > started
> > > > > using
> > > > > > > > > > this endpoint:
> > > > https://issues.apache.org/jira/browse/KAFKA-14843
> > > > > > . I
> > > > > > > > > > wasn't planning on converging the behavior in this KIP,
> as
> > I
> > > > > > > > > > considered it out-of-scope, and was going to follow the
> > > current
> > > > > > > > > > behavior. To summarize:
> > > > > > > > > >
> > > > > > > > > > GET /connector-plugins/<connector>/config will emit:
> > > > > > > > > > * connector.class (already implemented)
> > > > > > > > > > * connector.version (new)
> > > > > > > > > > * key.converter (already implemented)
> > > > > > > > > > * key.converter.version (new)
> > > > > > > > > > * value.converter (already implemented)
> > > > > > > > > > * value.converter.version (new)
> > > > > > > > > > * header.converter (already implemented)
> > > > > > > > > > * header.converter.version (new)
> > > > > > > > > > But will NOT emit:
> > > > > > > > > > * transforms.<alias>.type
> > > > > > > > > > * transforms.<alias>.version
> > > > > > > > > > * transforms.<alias>.predicate
> > > > > > > > > > * predicates.<alias>.type
> > > > > > > > > > * predicates.<alias>.version
> > > > > > > > > > * predicates.<alias>.negate
> > > > > > > > > >
> > > > > > > > > > GET /connector-plugins/<converter>/config will NOT emit:
> > > > > > > > > > * "" (there's not even a ".class" prefix!)
> > > > > > > > > > * version
> > > > > > > > > >
> > > > > > > > > > GET /connector-plugins/<transform>/config will NOT emit:
> > > > > > > > > > * type
> > > > > > > > > > * version
> > > > > > > > > > * predicate
> > > > > > > > > >
> > > > > > > > > > GET /connector-plugins/<predicate>/config will NOT emit:
> > > > > > > > > > * type
> > > > > > > > > > * version
> > > > > > > > > > * negate
> > > > > > > > > >
> > > > > > > > > > Do you want the converter, transform, and predicate
> > endpoints
> > > > > > > changed?
> > > > > > > > > > Do you want just "version", or do you want all of the
> > > prefixed
> > > > > > > configs
> > > > > > > > > > including "type", "predicate" and "negate"? How would you
> > > want
> > > > to
> > > > > > > > > > handle the converters?
> > > > > > > > > > And when I say the configs are "not part of the plugin
> > config
> > > > > > itself"
> > > > > > > > > > I mean that saying that GET
> > > > > > > > > > /connector-plugins/Flatten$Key/config?version=3.8.0 has a
> > > > > "version"
> > > > > > > > > > config that must be "3.8.0" is a little bit nonsense, as
> > the
> > > > > > version
> > > > > > > > > > is already specified.
> > > > > > > > > >
> > > > > > > > > > > IMO
> > > > > > > > > > > it's worth including this information somewhere
> directly
> > > > > > accessible
> > > > > > > > > > without
> > > > > > > > > > > having to provide a full connector config. FWIW I'd be
> > fine
> > > > > with
> > > > > > > GET
> > > > > > > > > > > /connector-plugins/<plugin>/versions as a first-class
> > > > endpoint
> > > > > > > > > >
> > > > > > > > > > You don't have to provide a configuration to call GET
> > > > > > > > > > /connector-plugins?connectorsOnly=false , is that
> endpoint
> > > not
> > > > > > close
> > > > > > > > > > enough to what you have in mind? See also the Rejected
> > > > > Alternative
> > > > > > > > > > "Adding new REST API endpoints"
> > > > > > > > > >
> > > > > > > > > > If you're calling /connector-plugins/<transform>/config,
> > you
> > > > know
> > > > > > the
> > > > > > > > > > name of a plugin right? That either comes from
> out-of-band
> > > > > > knowledge,
> > > > > > > > > > validating a connector config, or calling GET
> > > > > > > > > > /connector-plugins?connectorsOnly=false.
> > > > > > > > > > * If you have out-of-band knowledge of plugin classes,
> > > perhaps
> > > > > you
> > > > > > > > > > have out-of-band knowledge of versions too.
> > > > > > > > > > * If you've just validated a connector config, there
> should
> > > be
> > > > an
> > > > > > > > > > accompanying "version" field there with an accurate
> default
> > > > value
> > > > > > and
> > > > > > > > > > recommenders.
> > > > > > > > > > * If you've called GET
> > > /connector-plugins?connectorsOnly=false,
> > > > > > that
> > > > > > > > > > endpoint includes version information.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Greg
> > > > > > > > > >
> > > > > > > > > > On Wed, May 22, 2024 at 11:05 AM Chris Egerton
> > > > > > > <chrise@aiven.io.invalid
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Greg,
> > > > > > > > > > >
> > > > > > > > > > > Hope you had a nice weekend! Gonna try to keep things
> > > > concise.
> > > > > > > > > > >
> > > > > > > > > > > Concluded points:
> > > > > > > > > > >
> > > > > > > > > > > RE version recommenders, I agree it's likely that
> > > > programmatic
> > > > > > UIs
> > > > > > > > will
> > > > > > > > > > > already be able to handle dynamic configuration
> > > definitions,
> > > > > and
> > > > > > > the
> > > > > > > > > > detail
> > > > > > > > > > > about SMTs is a great point. I still anticipate some
> > > > > awkwardness
> > > > > > > with
> > > > > > > > > > > connector versions, though: if the latest version
> > supports
> > > > some
> > > > > > new
> > > > > > > > > > > properties, then a user switches to an earlier
> version, a
> > > UI
> > > > > may
> > > > > > > > > respond
> > > > > > > > > > by
> > > > > > > > > > > wiping values for these properties. I guess we can bite
> > the
> > > > > > bullet,
> > > > > > > > > > though.
> > > > > > > > > > >
> > > > > > > > > > > RE double-dinging during preflight validation for
> invalid
> > > > > > > versions, I
> > > > > > > > > > like
> > > > > > > > > > > the analogy with login credentials. I'm convinced that
> > the
> > > > > > proposal
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > KIP is best 👍
> > > > > > > > > > >
> > > > > > > > > > > Continued points:
> > > > > > > > > > >
> > > > > > > > > > > RE failing on worker startup, sorry, I should be
> clearer:
> > > > there
> > > > > > is
> > > > > > > no
> > > > > > > > > > _new_
> > > > > > > > > > > justification for it that doesn't also apply to
> existing
> > > > > > behavior.
> > > > > > > We
> > > > > > > > > > > shouldn't diverge from existing behavior solely for
> this
> > > new
> > > > > > case.
> > > > > > > An
> > > > > > > > > > > alternative could be to change existing behavior to
> fail
> > > fast
> > > > > on
> > > > > > > any
> > > > > > > > > > > invalid default converter configuration instead of just
> > for
> > > > > > invalid
> > > > > > > > > > > versions, but I'd vote to just stick to existing
> behavior
> > > and
> > > > > not
> > > > > > > > > > > complicate things, especially since no other part of
> the
> > > KIP
> > > > > > > requires
> > > > > > > > > > this
> > > > > > > > > > > change.
> > > > > > > > > > >
> > > > > > > > > > > RE exposing the version property in the
> > > > > > > > > > /connector-plugins/<plugin>/config
> > > > > > > > > > > endpoint, the behavior is inconsistent across plugin
> > types.
> > > > > > Hitting
> > > > > > > > the
> > > > > > > > > > > endpoint for the FileStreamSinkConnector on version
> 3.7.0
> > > > > yields
> > > > > > a
> > > > > > > > > > response
> > > > > > > > > > > that includes, among other things, the "topics",
> > > > > "topics.regex",
> > > > > > > and
> > > > > > > > > > > "errors.tolerance" properties. I see that we don't do
> > this
> > > > > > > everywhere
> > > > > > > > > > (the
> > > > > > > > > > > examples you cite for SMT and converter properties are
> > > > > accurate),
> > > > > > > but
> > > > > > > > > IMO
> > > > > > > > > > > it's worth including this information somewhere
> directly
> > > > > > accessible
> > > > > > > > > > without
> > > > > > > > > > > having to provide a full connector config. FWIW I'd be
> > fine
> > > > > with
> > > > > > > GET
> > > > > > > > > > > /connector-plugins/<plugin>/versions as a first-class
> > > > endpoint
> > > > > > > either
> > > > > > > > > > > instead of or in addition to adding recommended values
> > for
> > > > all
> > > > > > > plugin
> > > > > > > > > > > versions.
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your continued work on this KIP, and with
> the
> > > > > progress
> > > > > > > > we're
> > > > > > > > > > > making I'm optimistic about its chances of appearing in
> > > > 4.0.0.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 15, 2024 at 1:22 PM Greg Harris
> > > > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hey Chris,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your quick follow up.
> > > > > > > > > > > >
> > > > > > > > > > > > > But this risk is already present with
> > > > > > > > > > > > > existing error cases, and I don't see anything that
> > > > > justifies
> > > > > > > > > > changing
> > > > > > > > > > > > > existing behavior with an invalid converter class,
> or
> > > > > > diverging
> > > > > > > > > from
> > > > > > > > > > it
> > > > > > > > > > > > in
> > > > > > > > > > > > > the case of invalid converter versions.
> > > > > > > > > > > >
> > > > > > > > > > > > The justification is to fail-fast, and prevent REST
> API
> > > > users
> > > > > > > from
> > > > > > > > > > > > receiving errors from bad configs that they didn't
> > write,
> > > > or
> > > > > > > maybe
> > > > > > > > > > > > don't even know apply to them.
> > > > > > > > > > > > Up until recently errors in these configurations
> > surfaced
> > > > as
> > > > > > > > failures
> > > > > > > > > > > > to create the connector, or failures to start, and
> you
> > > made
> > > > > > them
> > > > > > > > > > > > fail-fast during validation. I think this change is
> in
> > > the
> > > > > same
> > > > > > > > > > > > spirit, pulling the validation further forward and
> not
> > > > > letting
> > > > > > > > errors
> > > > > > > > > > > > lie dormant.
> > > > > > > > > > > >
> > > > > > > > > > > > And to call back to your original concern about
> > > > interrupting
> > > > > > > > > > > > connectors that explicitly provide these
> configurations
> > > and
> > > > > > don't
> > > > > > > > use
> > > > > > > > > > > > the worker configs: I expect that operators with a
> > > majority
> > > > > of
> > > > > > > > these
> > > > > > > > > > > > sorts of clients aren't going to be setting the
> worker
> > > > > .version
> > > > > > > > > > > > properties, because it would have no effect on the
> > > majority
> > > > > of
> > > > > > > > their
> > > > > > > > > > > > connectors. They would be able to rely on
> > > > > > backwards-compatibility
> > > > > > > > and
> > > > > > > > > > > > continue to ignore the class properties.
> > > > > > > > > > > >
> > > > > > > > > > > > > CLI
> > > > > > > > > > > > > and programmatic UI developers will want to develop
> > > their
> > > > > own
> > > > > > > > > tooling
> > > > > > > > > > > > > layers.
> > > > > > > > > > > >
> > > > > > > > > > > > This is a very compelling argument. Snehashis do you
> > want
> > > > to
> > > > > > > figure
> > > > > > > > > > > > out a REST API design for this use-case?
> > > > > > > > > > > >
> > > > > > > > > > > > > Regarding the GET
> /connector-plugins/<plugin>/config
> > > > > > endpoint,
> > > > > > > I
> > > > > > > > > was
> > > > > > > > > > > > > thinking about the response for non-connector
> > plugins,
> > > > > e.g.,
> > > > > > > > > > > > > GET /connector-plugins/RegexRouter/config. Would a
> > > > > "version"
> > > > > > > > > property
> > > > > > > > > > > > > appear with recommended values?
> > > > > > > > > > > >
> > > > > > > > > > > > I intended for the ".version" property to be like
> other
> > > > > > framework
> > > > > > > > > > > > configs (".class", ".type", ".predicate", ".negate")
> > > where
> > > > > they
> > > > > > > are
> > > > > > > > > > > > inside the plugin namespace, but not part of the
> plugin
> > > > > config
> > > > > > > > > itself.
> > > > > > > > > > > > Perhaps we can deviate from those configs because it
> > > would
> > > > > aid
> > > > > > in
> > > > > > > > > > > > discovering other valid `GET
> > > > > > > > > > > > /connector-plugins/<plugin>/config?version=` calls,
> > > without
> > > > > > > calling
> > > > > > > > > > > > `GET /connector-plugins?connectorsOnly=false`.
> > > > > > > > > > > > I don't really feel strongly either way.
> > > > > > > > > > > >
> > > > > > > > > > > > > if a user changes the
> > > > > > > > > > > > > connector version in, e.g., a dropdown menu, then
> the
> > > UI
> > > > > > either
> > > > > > > > has
> > > > > > > > > > to
> > > > > > > > > > > > > re-fetch the ConfigDef for the new version, or risk
> > > > > operating
> > > > > > > on
> > > > > > > > > > stale
> > > > > > > > > > > > > information
> > > > > > > > > > > >
> > > > > > > > > > > > This is a really interesting situation, thanks for
> > > finding
> > > > > > that!
> > > > > > > > This
> > > > > > > > > > > > is already a footgun with transformations and
> > predicates;
> > > > > Once
> > > > > > > you
> > > > > > > > > > > > fill out a transformation class (which includes a
> > > > > recommender)
> > > > > > > you
> > > > > > > > > > > > will then see all of the transformations
> > configurations.
> > > > > > > > > > > > I think this means that UI developers should have
> > already
> > > > > > > developed
> > > > > > > > > > > > the infrastructure for handling dynamic recommenders,
> > or
> > > > else
> > > > > > > > they've
> > > > > > > > > > > > had this bug since KIP-66 in 2017. It may require
> some
> > > > manual
> > > > > > > > > > > > attention to roll-out support for the ".version"
> > > properties
> > > > > > > > > > > > specifically, but I don't think that should prevent
> us
> > > from
> > > > > > > > providing
> > > > > > > > > > > > this information in a natural place.
> > > > > > > > > > > >
> > > > > > > > > > > > > but will it be possible to configure
> > > > > > > > > > > > > a connector to use two instances of the same
> > transform
> > > or
> > > > > > > > > predicate,
> > > > > > > > > > but
> > > > > > > > > > > > > with different versions for each?
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, and this is included in the example in the KIP,
> > > > > > > > > > > > `transforms.flatten-latest` and
> > `transforms.flatten-old`.
> > > > > This
> > > > > > > will
> > > > > > > > > > > > work in the natural way, with the two instances
> having
> > a
> > > > > > > different
> > > > > > > > > > > > version if configured for that. If a user wants to
> pin
> > > the
> > > > > same
> > > > > > > > > > > > version of a plugin for every instance, they will
> need
> > to
> > > > > > provide
> > > > > > > > the
> > > > > > > > > > > > version config multiple times and keep them in-sync
> > > > > themselves.
> > > > > > > > > > > >
> > > > > > > > > > > > > I would have expected the error
> > > > > > > > > > > > > to only be attributed to the version property, and
> > for
> > > > the
> > > > > > > class
> > > > > > > > > > property
> > > > > > > > > > > > > to be reported as valid.
> > > > > > > > > > > >
> > > > > > > > > > > > I considered this, and then went back and changed it.
> > To
> > > > > quote
> > > > > > > > > someone
> > > > > > > > > > > > else, this is to catch "misspelling cat as dog". For
> > > > example
> > > > > a
> > > > > > > user
> > > > > > > > > > > > types DogConverter and meant to type CatConverter,
> and
> > > then
> > > > > > > types a
> > > > > > > > > > > > version which is valid for CatConverter, conceptually
> > the
> > > > > error
> > > > > > > is
> > > > > > > > in
> > > > > > > > > > > > the class name and not the version.
> > > > > > > > > > > > That's a very contrived scenario, but I think similar
> > > > > arguments
> > > > > > > are
> > > > > > > > > > > > used for attributing validation errors to
> > > > > > > endpoints/urls/hostnames
> > > > > > > > > > > > when the associated credentials are unable to log
> into
> > > the
> > > > > > remote
> > > > > > > > > > > > system. Did the user provide the correct testing
> > > > credentials,
> > > > > > but
> > > > > > > > > > > > accidentally typed the production endpoint? Even if
> the
> > > > > > > production
> > > > > > > > > > > > endpoint looks valid (it's a real hostname that is
> > > > reachable)
> > > > > > the
> > > > > > > > > > > > conceptual error is still in the hostname and should
> > have
> > > > the
> > > > > > > error
> > > > > > > > > > > > attributed to it to draw the user's attention.
> > > > > > > > > > > > If that's not convincing, I think the alternative of
> > only
> > > > > > > > attributing
> > > > > > > > > > > > version errors to the version property is also
> > > acceptable.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Greg
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, May 15, 2024 at 8:06 AM Chris Egerton
> > > > > > > > > <chrise@aiven.io.invalid
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your responses! Continuations of
> existing
> > > > > > > discussions:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regarding crashing the worker on startup--yes,
> there
> > is
> > > > > also
> > > > > > a
> > > > > > > > risk
> > > > > > > > > > to
> > > > > > > > > > > > > allowing it to join the cluster. But this risk is
> > > already
> > > > > > > present
> > > > > > > > > > with
> > > > > > > > > > > > > existing error cases, and I don't see anything that
> > > > > justifies
> > > > > > > > > > changing
> > > > > > > > > > > > > existing behavior with an invalid converter class,
> or
> > > > > > diverging
> > > > > > > > > from
> > > > > > > > > > it
> > > > > > > > > > > > in
> > > > > > > > > > > > > the case of invalid converter versions. I think we
> > > should
> > > > > > keep
> > > > > > > > this
> > > > > > > > > > > > simple
> > > > > > > > > > > > > and not do anything different for worker startup
> than
> > > we
> > > > do
> > > > > > > > > already.
> > > > > > > > > > > > >
> > > > > > > > > > > > > As far as REST API vs. metrics go--I think you're
> > right
> > > > > that
> > > > > > > the
> > > > > > > > > > original
> > > > > > > > > > > > > version metrics were added as a "monitoring"
> detail.
> > > > > However,
> > > > > > > > this
> > > > > > > > > > was
> > > > > > > > > > > > back
> > > > > > > > > > > > > when plugin versions were managed solely by cluster
> > > > > > > > administrators.
> > > > > > > > > > With
> > > > > > > > > > > > > this KIP, connector users will be able to manage
> > plugin
> > > > > > > versions,
> > > > > > > > > > and CLI
> > > > > > > > > > > > > and programmatic UI developers will want to develop
> > > their
> > > > > own
> > > > > > > > > tooling
> > > > > > > > > > > > > layers. I think focusing on the REST API as the
> > primary
> > > > > > > interface
> > > > > > > > > for
> > > > > > > > > > > > this
> > > > > > > > > > > > > KIP would be best for these users.
> > > > > > > > > > > > >
> > > > > > > > > > > > > (All that said, I don't object to the metrics that
> > are
> > > > > > proposed
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > KIP;
> > > > > > > > > > > > > I just think they make more sense in addition to
> new
> > > REST
> > > > > API
> > > > > > > > > > > > > functionality, as opposed to instead of it.)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regarding the GET
> /connector-plugins/<plugin>/config
> > > > > > endpoint,
> > > > > > > I
> > > > > > > > > was
> > > > > > > > > > > > > thinking about the response for non-connector
> > plugins,
> > > > > e.g.,
> > > > > > > > > > > > > GET /connector-plugins/RegexRouter/config. Would a
> > > > > "version"
> > > > > > > > > property
> > > > > > > > > > > > > appear with recommended values?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > And new thoughts:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1) Regarding the recommended values for
> > > > > "connector.version",
> > > > > > > this
> > > > > > > > > > might
> > > > > > > > > > > > be
> > > > > > > > > > > > > confusing since there could be differences between
> > the
> > > > > > > ConfigDefs
> > > > > > > > > > for the
> > > > > > > > > > > > > latest version of the connector and prior versions.
> > It
> > > > also
> > > > > > > makes
> > > > > > > > > the
> > > > > > > > > > > > flow
> > > > > > > > > > > > > a bit awkward for programmatic UI developers: if a
> > user
> > > > > > changes
> > > > > > > > the
> > > > > > > > > > > > > connector version in, e.g., a dropdown menu, then
> the
> > > UI
> > > > > > either
> > > > > > > > has
> > > > > > > > > > to
> > > > > > > > > > > > > re-fetch the ConfigDef for the new version, or risk
> > > > > operating
> > > > > > > on
> > > > > > > > > > stale
> > > > > > > > > > > > > information. I'm starting to doubt that exposing
> the
> > > > range
> > > > > of
> > > > > > > > > > available
> > > > > > > > > > > > > versions via recommended values is the best way to
> > > > proceed,
> > > > > > > > instead
> > > > > > > > > > of a
> > > > > > > > > > > > > more explicit approach like GET
> > > > > > > > > > /connector-plugins/<plugin>/versions, or
> > > > > > > > > > > > > the "Adding new REST API endpoints" rejected
> > > alternative.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2) I know that this is a bit heinous, but will it
> be
> > > > > possible
> > > > > > > to
> > > > > > > > > > > > configure
> > > > > > > > > > > > > a connector to use two instances of the same
> > transform
> > > or
> > > > > > > > > predicate,
> > > > > > > > > > but
> > > > > > > > > > > > > with different versions for each? (I don't think
> this
> > > is
> > > > > > worth
> > > > > > > > > > > > significant
> > > > > > > > > > > > > design/implementation effort, so if it would
> inflate
> > > > either
> > > > > > of
> > > > > > > > > those,
> > > > > > > > > > > > > please don't feel obligated to support it.)
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3) Just out of curiosity, why double-ding during
> > config
> > > > > > > > validation
> > > > > > > > > > if the
> > > > > > > > > > > > > version for a plugin class can't be found? I would
> > have
> > > > > > > expected
> > > > > > > > > the
> > > > > > > > > > > > error
> > > > > > > > > > > > > to only be attributed to the version property, and
> > for
> > > > the
> > > > > > > class
> > > > > > > > > > property
> > > > > > > > > > > > > to be reported as valid.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Chris
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, May 14, 2024 at 1:11 PM Greg Harris
> > > > > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hey Chris,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for your comments. I'm glad you like the
> > > > > > motivations,
> > > > > > > > > > Snehashis
> > > > > > > > > > > > > > wrote that part!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > the configuration syntax for the most basic use
> > > case
> > > > of
> > > > > > > > > > > > > > > specifying a single desired version is pretty
> > > > > > > > counterintuitive.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I agree, and the "soft" requirement scheme is
> > > > something I
> > > > > > > > wasn't
> > > > > > > > > > > > > > explicitly looking for, but would be inherited
> from
> > > the
> > > > > > > > library.
> > > > > > > > > > I'm
> > > > > > > > > > > > > > fine with eliminating the soft requirement
> > semantics,
> > > > and
> > > > > > > > having
> > > > > > > > > > > > > > "1.0.0" behave the same as "[1.0.0]". I'm less
> > > inclined
> > > > > to
> > > > > > > > > include
> > > > > > > > > > > > > > "range" in the property name, or have two
> > properties.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Failing startup is drastic and has the
> potential
> > to
> > > > > > disrupt
> > > > > > > > the
> > > > > > > > > > > > > > > availability of connectors that would otherwise
> > be
> > > > able
> > > > > > to
> > > > > > > > run
> > > > > > > > > > > > healthily
> > > > > > > > > > > > > > > because they were explicitly configured to use
> > > valid
> > > > > > > > converters
> > > > > > > > > > > > instead
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the worker defaults.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think this argument cuts both ways. If someone
> > > > > > > reconfigures a
> > > > > > > > > > worker
> > > > > > > > > > > > > > and adds an invalid ".version" string to the
> worker
> > > (or
> > > > > > > changes
> > > > > > > > > the
> > > > > > > > > > > > > > plugin.path to make it invalid), it would be
> > > permitted
> > > > to
> > > > > > > enter
> > > > > > > > > the
> > > > > > > > > > > > > > group, and accept work assignments. If those work
> > > > > > assignments
> > > > > > > > > used
> > > > > > > > > > > > > > these configurations, a set of tasks could
> > transition
> > > > to
> > > > > > > FAILED
> > > > > > > > > and
> > > > > > > > > > > > > > not be able to recover, because they would be
> > > restarted
> > > > > > again
> > > > > > > > on
> > > > > > > > > > the
> > > > > > > > > > > > > > same worker.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Why are metrics utilized to report information
> > > about
> > > > > > plugin
> > > > > > > > > > versions
> > > > > > > > > > > > > > > utilized by connectors at runtime instead of
> > > > publishing
> > > > > > > this
> > > > > > > > > > info in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > REST API
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I was following the existing pattern for exposing
> > > > runtime
> > > > > > > > > versions
> > > > > > > > > > for
> > > > > > > > > > > > > > connectors, and it did seem like a "monitoring"
> > > > feature.
> > > > > If
> > > > > > > > that
> > > > > > > > > > > > > > approach is flawed and should be phased out, I
> > think
> > > it
> > > > > > would
> > > > > > > > be
> > > > > > > > > a
> > > > > > > > > > > > > > good idea to reconsider the REST API rejected
> > > > > alternative.
> > > > > > > > > > > > > > We would need some additional design work to spec
> > out
> > > > the
> > > > > > > REST
> > > > > > > > > API
> > > > > > > > > > > > > > interface, as I don't have anything in mind
> > > currently.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm unclear on whether or not it'll be possible
> > to
> > > > see
> > > > > > this
> > > > > > > > > > > > information
> > > > > > > > > > > > > > via the
> > > > > > > > > > > > > > > GET /connector-plugins/<plugin>/config endpoint
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > There is room in the API to add recommenders for
> > > > > > > > "key.converter",
> > > > > > > > > > > > > > "value.converter", and "header.converter", but
> not
> > > for
> > > > > > > > transforms
> > > > > > > > > > and
> > > > > > > > > > > > > > predicates, as they include aliases that depend
> on
> > an
> > > > > > actual
> > > > > > > > > > > > > > configuration. We could explicitly say we're
> going
> > to
> > > > do
> > > > > > > that,
> > > > > > > > or
> > > > > > > > > > do
> > > > > > > > > > > > > > whatever is convenient during the implementation
> > > phase,
> > > > > or
> > > > > > > > leave
> > > > > > > > > it
> > > > > > > > > > > > > > open to be improved later.
> > > > > > > > > > > > > > There will not be any recommenders for ".version"
> > > > > > properties
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > `/config` endpoint, because those recommenders
> are
> > > > > dynamic
> > > > > > > and
> > > > > > > > > > depend
> > > > > > > > > > > > > > on an actual configuration.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 5) There are two relevant lines in the KIP: "If a
> > > > > .version
> > > > > > > > > property
> > > > > > > > > > > > > > contains a hard requirement, select the latest
> > > > installed
> > > > > > > > version
> > > > > > > > > > which
> > > > > > > > > > > > > > satisfies the requirement." and "This
> configuration
> > > is
> > > > > > > > > re-evaluated
> > > > > > > > > > > > > > each time the connector or task are assigned to a
> > new
> > > > > > > worker".
> > > > > > > > I
> > > > > > > > > > would
> > > > > > > > > > > > > > call this "eager" upgrade behavior, rather than a
> > > > > "sticky"
> > > > > > or
> > > > > > > > > > "lazy"
> > > > > > > > > > > > > > upgrade behavior.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 6) Updated!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, May 14, 2024 at 9:14 AM Chris Egerton
> > > > > > > > > > <chrise@aiven.io.invalid
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks Greg for updating the KIP, and thanks
> > > > Snehashis
> > > > > > for
> > > > > > > > > > starting
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > work on this originally.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The motivation section makes a pretty
> convincing
> > > case
> > > > > for
> > > > > > > > this
> > > > > > > > > > kind
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > feature. My thoughts are mostly about specific
> > > > details:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1) I like the support for version ranges (the
> > > example
> > > > > > > > > > demonstrating
> > > > > > > > > > > > how
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > avoid KAFKA-10574 with the header converter was
> > > > > > > particularly
> > > > > > > > > > > > > > entertaining),
> > > > > > > > > > > > > > > but the configuration syntax for the most basic
> > use
> > > > > case
> > > > > > of
> > > > > > > > > > > > specifying a
> > > > > > > > > > > > > > > single desired version is pretty
> > counterintuitive.
> > > > > People
> > > > > > > may
> > > > > > > > > get
> > > > > > > > > > > > bitten
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > at least frustrated if they put
> > > > connector.version=3.8.0
> > > > > > in
> > > > > > > a
> > > > > > > > > > config
> > > > > > > > > > > > but
> > > > > > > > > > > > > > > then version 3.7.5 ends up running. I'd like it
> > if
> > > we
> > > > > > could
> > > > > > > > > > either
> > > > > > > > > > > > > > > intentionally deviate from Maven ranges when a
> > bare
> > > > > > version
> > > > > > > > is
> > > > > > > > > > > > present,
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > separate things out into two properties:
> > > foo.version
> > > > > > would
> > > > > > > be
> > > > > > > > > the
> > > > > > > > > > > > single
> > > > > > > > > > > > > > > accepted version for the foo plugin, and
> > > > > > foo.version.range
> > > > > > > > > would
> > > > > > > > > > use
> > > > > > > > > > > > > > Maven
> > > > > > > > > > > > > > > range syntax. Open to other options too, just
> > > > > providing a
> > > > > > > > > couple
> > > > > > > > > > to
> > > > > > > > > > > > get
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > ball rolling.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2) Although the current behavior for a worker
> > with
> > > an
> > > > > > > invalid
> > > > > > > > > > > > > > > key/value/header converter class specified in
> its
> > > > > config
> > > > > > > file
> > > > > > > > > is
> > > > > > > > > > a
> > > > > > > > > > > > little
> > > > > > > > > > > > > > > strange (I was surprised to learn that it
> > wouldn't
> > > > fail
> > > > > > on
> > > > > > > > > > startup),
> > > > > > > > > > > > I
> > > > > > > > > > > > > > > don't see a good reason to deviate from this
> when
> > > an
> > > > > > > invalid
> > > > > > > > > > version
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > specified. Failing startup is drastic and has
> the
> > > > > > potential
> > > > > > > > to
> > > > > > > > > > > > disrupt
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > availability of connectors that would otherwise
> > be
> > > > able
> > > > > > to
> > > > > > > > run
> > > > > > > > > > > > healthily
> > > > > > > > > > > > > > > because they were explicitly configured to use
> > > valid
> > > > > > > > converters
> > > > > > > > > > > > instead
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the worker defaults.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3) Why are metrics utilized to report
> information
> > > > about
> > > > > > > > plugin
> > > > > > > > > > > > versions
> > > > > > > > > > > > > > > utilized by connectors at runtime instead of
> > > > publishing
> > > > > > > this
> > > > > > > > > > info in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > REST API? I saw that this was mentioned as a
> > > rejected
> > > > > > > > > > alternative,
> > > > > > > > > > > > but I
> > > > > > > > > > > > > > > didn't get a sense of why. It seems like the
> REST
> > > API
> > > > > > would
> > > > > > > > be
> > > > > > > > > > > > easier to
> > > > > > > > > > > > > > > access and more intuitive for most users than
> new
> > > > > > metrics.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 4) In the "Validation" section it's stated that
> > > > "Users
> > > > > > can
> > > > > > > > use
> > > > > > > > > > these
> > > > > > > > > > > > > > > recommenders to discover the valid plugin
> classes
> > > and
> > > > > > > > versions,
> > > > > > > > > > > > without
> > > > > > > > > > > > > > > requiring an earlier call to GET
> > > > > > > > > > > > > > /connector-plugins?connectorsOnly=false."
> > > > > > > > > > > > > > > I really like the creativity and simplicity of
> > > > reusing
> > > > > > the
> > > > > > > > > > > > recommender
> > > > > > > > > > > > > > > mechanism to expose available versions for
> > plugins
> > > > via
> > > > > > the
> > > > > > > > REST
> > > > > > > > > > API.
> > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > unclear on whether or not it'll be possible to
> > see
> > > > this
> > > > > > > > > > information
> > > > > > > > > > > > via
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > GET /connector-plugins/<plugin>/config
> endpoint,
> > > > > though.
> > > > > > > It'd
> > > > > > > > > be
> > > > > > > > > > > > great if
> > > > > > > > > > > > > > > this were supported, since we learned in
> KIP-769
> > > [1]
> > > > > that
> > > > > > > > > people
> > > > > > > > > > > > really
> > > > > > > > > > > > > > > want to be able to see configuration options
> for
> > > > > > connectors
> > > > > > > > and
> > > > > > > > > > their
> > > > > > > > > > > > > > > plugins via some kind of GET endpoint without
> > > having
> > > > to
> > > > > > > > > provide a
> > > > > > > > > > > > > > complete
> > > > > > > > > > > > > > > connector config for validation.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 5) In the Maven version range docs, it's stated
> > > that
> > > > > > "Maven
> > > > > > > > > > picks the
> > > > > > > > > > > > > > > highest version of each project that satisfies
> > all
> > > > the
> > > > > > hard
> > > > > > > > > > > > requirements
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the dependencies on that project." I'm guessing
> > > this
> > > > > > > behavior
> > > > > > > > > > will be
> > > > > > > > > > > > > > > retained for Connect; i.e., the
> highest-possible
> > > > > version
> > > > > > of
> > > > > > > > > each
> > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > that satisfies the user-specified version
> > > constraints
> > > > > > will
> > > > > > > be
> > > > > > > > > > run?
> > > > > > > > > > > > (An
> > > > > > > > > > > > > > > alternative approach could be to have some kind
> > of
> > > > > > "sticky"
> > > > > > > > > logic
> > > > > > > > > > > > that
> > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > restarts connectors/tasks when their
> > currently-used
> > > > > > version
> > > > > > > > > > becomes
> > > > > > > > > > > > > > > incompatible with the configured constraints.)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 6) (Nit) It'd be nice to add a link to the
> > > > TestPlugins
> > > > > > > class
> > > > > > > > or
> > > > > > > > > > > > somewhere
> > > > > > > > > > > > > > > in its neighborhood to the testing plan;
> > unfamiliar
> > > > > > readers
> > > > > > > > > > probably
> > > > > > > > > > > > > > won't
> > > > > > > > > > > > > > > get much out of what's there right now.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > [1] -
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, May 13, 2024 at 2:01 PM Snehashis <
> > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > That is much appreciated. No complaints on
> the
> > > > > > additional
> > > > > > > > > > scope, I
> > > > > > > > > > > > will
> > > > > > > > > > > > > > > > make some time out to work on this once we
> have
> > > > > > approval.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> > > > > > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hey Snehashis,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'm glad to hear you're still interested in
> > > this
> > > > > KIP!
> > > > > > > > > > > > > > > > > I'm happy to let you drive this, and I
> > > apologize
> > > > > for
> > > > > > > > > > increasing
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > scope of work so drastically. To make up
> for
> > > > that,
> > > > > > I'll
> > > > > > > > > > > > volunteer to
> > > > > > > > > > > > > > > > > be the primary PR reviewer to help get this
> > > done
> > > > > > > quickly
> > > > > > > > > once
> > > > > > > > > > > > the KIP
> > > > > > > > > > > > > > > > > is approved.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 3:51 AM Snehashis <
> > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for the follow up to my original
> > KIP,
> > > I
> > > > am
> > > > > > in
> > > > > > > > > > favour of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > additions made to expand its scope, the
> > > > addition
> > > > > of
> > > > > > > > range
> > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > specifically make a lot of sense.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Apologies if I have not publicly worked
> on
> > > this
> > > > > KIP
> > > > > > > > for a
> > > > > > > > > > long
> > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > original work was done when the move to
> > > service
> > > > > > > loading
> > > > > > > > > > was in
> > > > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > > > and I wanted to loop back to this only
> > after
> > > > that
> > > > > > > work
> > > > > > > > > was
> > > > > > > > > > > > > > completed.
> > > > > > > > > > > > > > > > > Post
> > > > > > > > > > > > > > > > > > its conclusion, I have not been able to
> > take
> > > > this
> > > > > > up
> > > > > > > > due
> > > > > > > > > to
> > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > priorities. If it's okay with you, I
> would
> > > > still
> > > > > > like
> > > > > > > > to
> > > > > > > > > > get
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > implemented myself, including the
> > additional
> > > > > scope.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks and regards
> > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 12:45 AM Greg
> > Harris
> > > > > > > > > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I'd like to reboot the discussion on
> > > KIP-891:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I've made some changes, most notably:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. Specifying versions for all plugins
> in
> > > > > > Connector
> > > > > > > > > > configs
> > > > > > > > > > > > > > > > > > > (converters, header converters,
> > transforms,
> > > > and
> > > > > > > > > > predicates)
> > > > > > > > > > > > not
> > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > connectors & tasks
> > > > > > > > > > > > > > > > > > > 2. Specifying a range of versions
> instead
> > > of
> > > > an
> > > > > > > exact
> > > > > > > > > > match
> > > > > > > > > > > > > > > > > > > 3. New metrics to observe what versions
> > are
> > > > > > in-use
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks to Snehashis for the original
> KIP
> > > > idea!
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg
> > > Harris <
> > > > > > > > > > > > > > greg.harris@aiven.io>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Snehashis,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thank you for the KIP! This is
> > something
> > > > I've
> > > > > > > > wanted
> > > > > > > > > > for a
> > > > > > > > > > > > long
> > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I know the discussion has gone cold,
> > are
> > > > you
> > > > > > > still
> > > > > > > > > > > > interested
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > pursuing this feature? I'll make time
> > to
> > > > > review
> > > > > > > the
> > > > > > > > > > KIP if
> > > > > > > > > > > > you
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > still accepting comments.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM
> > > Snehashis
> > > > <
> > > > > > > > > > > > > > > > snehashisp1995@gmail.com
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for the points Sagar.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1) Should we update the GET
> > > /connectors
> > > > > > > > endpoint
> > > > > > > > > to
> > > > > > > > > > > > > > include the
> > > > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > > > the plugin that is running? It
> > could
> > > be
> > > > > > > useful
> > > > > > > > to
> > > > > > > > > > > > figure
> > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > the plugin or I am assuming it
> gets
> > > > > > returned
> > > > > > > by
> > > > > > > > > the
> > > > > > > > > > > > > > expand=info
> > > > > > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think this is good to have and
> > > possible
> > > > > > > future
> > > > > > > > > > > > > > enhancement. The
> > > > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > > info will be present in the config
> of
> > > the
> > > > > > > > connector
> > > > > > > > > > if
> > > > > > > > > > > > the
> > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > > > specified the version. Otherwise it
> > is
> > > > the
> > > > > > > latest
> > > > > > > > > > version
> > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > > > can find out from the
> > connector-plugin
> > > > > > > endpoint.
> > > > > > > > > The
> > > > > > > > > > > > > > information
> > > > > > > > > > > > > > > > > can be
> > > > > > > > > > > > > > > > > > > > > introduced to the response of the
> GET
> > > > > > > /connectors
> > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > itself,
> > > > > > > > > > > > > > > > > > > however
> > > > > > > > > > > > > > > > > > > > > the most ideal way of doing this
> > would
> > > be
> > > > > to
> > > > > > > get
> > > > > > > > > the
> > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > > > > instance of the connector and get
> the
> > > > > version
> > > > > > > > > > directly
> > > > > > > > > > > > from
> > > > > > > > > > > > > > > > there.
> > > > > > > > > > > > > > > > > > > This is
> > > > > > > > > > > > > > > > > > > > > slightly tricky as the connector
> > could
> > > be
> > > > > > > running
> > > > > > > > > in
> > > > > > > > > > a
> > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > node.
> > > > > > > > > > > > > > > > > > > > > One way to do this would be to
> > persist
> > > > the
> > > > > > > > version
> > > > > > > > > > > > > > information in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > status backing store during
> > > instantiation
> > > > > of
> > > > > > > the
> > > > > > > > > > > > connector.
> > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > requires
> > > > > > > > > > > > > > > > > > > > > some more thought and since the
> > version
> > > > is
> > > > > > part
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > > configs if
> > > > > > > > > > > > > > > > > > > provided
> > > > > > > > > > > > > > > > > > > > > and evident otherwise, I have not
> > > > included
> > > > > it
> > > > > > > in
> > > > > > > > > this
> > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2) I am not aware of this and
> hence
> > > > > asking,
> > > > > > > > can 2
> > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > > versions have the same name? Does
> > the
> > > > > > plugin
> > > > > > > > > > isolation
> > > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > > > could have a bearing when using
> the
> > > > > > lifecycle
> > > > > > > > > > > > endpoints for
> > > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > All connectors in a cluster need to
> > > have
> > > > > > > uniquire
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > names
> > > > > > > > > > > > > > > > > > > > > regardless of what version of the
> > > plugin
> > > > > the
> > > > > > > > > > connector is
> > > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > > > > underneath. This is something
> > enforced
> > > by
> > > > > the
> > > > > > > > > connect
> > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > itself.
> > > > > > > > > > > > > > > > > > > All
> > > > > > > > > > > > > > > > > > > > > connect CRUD operations are keyed
> on
> > > the
> > > > > > > > connector
> > > > > > > > > > name
> > > > > > > > > > > > so
> > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > be an issue.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM
> > Sagar <
> > > > > > > > > > > > > > sagarmeansocean@gmail.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hey Snehashsih,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP. It looks
> like a
> > > > very
> > > > > > > useful
> > > > > > > > > > > > feature.
> > > > > > > > > > > > > > Couple
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > small-ish points, let me know
> what
> > > you
> > > > > > think:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1) Should we update the GET
> > > /connectors
> > > > > > > > endpoint
> > > > > > > > > to
> > > > > > > > > > > > > > include the
> > > > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > > > the plugin that is running? It
> > could
> > > be
> > > > > > > useful
> > > > > > > > to
> > > > > > > > > > > > figure
> > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > > > the plugin or I am assuming it
> gets
> > > > > > returned
> > > > > > > by
> > > > > > > > > the
> > > > > > > > > > > > > > expand=info
> > > > > > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > > > > > > 2) I am not aware of this and
> hence
> > > > > asking,
> > > > > > > > can 2
> > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > > versions have the same name? Does
> > the
> > > > > > plugin
> > > > > > > > > > isolation
> > > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > > > could have a bearing when using
> the
> > > > > > lifecycle
> > > > > > > > > > > > endpoints for
> > > > > > > > > > > > > > > > > > > connectors like
> > > > > > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > > > > Sagar.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM
> > > Ashwin
> > > > > > > > > > > > > > > > > <apankaj@confluent.io.invalid
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > IIUC (please correct me if I
> am
> > > > wrong
> > > > > > > > here),
> > > > > > > > > > what
> > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > a versioning scheme for a
> > connector
> > > > > > config
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > different versions of a
> connector
> > > > > > plugin).
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Sorry for not being more
> precise
> > in
> > > > my
> > > > > > > > wording
> > > > > > > > > > - I
> > > > > > > > > > > > meant
> > > > > > > > > > > > > > > > > > > registering
> > > > > > > > > > > > > > > > > > > > > > > versions of schema for
> connector
> > > > > config.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Let's take the example of a
> > > fictional
> > > > > > > > connector
> > > > > > > > > > which
> > > > > > > > > > > > > > uses a
> > > > > > > > > > > > > > > > > > > fictional
> > > > > > > > > > > > > > > > > > > > > > AWS
> > > > > > > > > > > > > > > > > > > > > > > service.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Fictional Connector Config
> schema
> > > > > > > version:2.0
> > > > > > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > > > > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Fictional Connector config
> schema
> > > > > > > version:3.0
> > > > > > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > "iam_role": {
> > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > > > > > "iam_role"
> > > > > > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The connector which supports
> > > > Fictional
> > > > > > > config
> > > > > > > > > > schema
> > > > > > > > > > > > 2.0
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > validate
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > access key and secret key.
> > > > > > > > > > > > > > > > > > > > > > > Whereas a connector which
> > supports
> > > > > config
> > > > > > > > with
> > > > > > > > > > schema
> > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > 3.0
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > > > > > > validate the IAM role.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This is the alternative which I
> > > > wanted
> > > > > to
> > > > > > > > > > suggest.
> > > > > > > > > > > > Each
> > > > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > register the schema versions of
> > > > > connector
> > > > > > > > > config
> > > > > > > > > > > > which it
> > > > > > > > > > > > > > > > > supports.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The plugin paths may be
> > optionally
> > > > > > > different
> > > > > > > > > > i.e we
> > > > > > > > > > > > > > don't
> > > > > > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > > > > > > mandatorily add a new plugin
> path
> > > to
> > > > > > > support
> > > > > > > > a
> > > > > > > > > > new
> > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > version.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:47
> PM
> > > > > > Snehashis
> > > > > > > <
> > > > > > > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on the
> > > > > rejected
> > > > > > > > > > > > alternatives ?
> > > > > > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > config is versioned and
> has a
> > > > > schema.
> > > > > > > > Then
> > > > > > > > > a
> > > > > > > > > > > > single
> > > > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > > > > > dependencies have not
> > changed)
> > > > can
> > > > > > > handle
> > > > > > > > > > > > multiple
> > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > IIUC (please correct me if I
> am
> > > > wrong
> > > > > > > > here),
> > > > > > > > > > what
> > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > a versioning scheme for a
> > > connector
> > > > > > > config
> > > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > connector (and
> > > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > different versions of a
> > connector
> > > > > > > plugin).
> > > > > > > > > > That is
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > somewhat
> > > > > > > > > > > > > > > > > > > > > > tangential
> > > > > > > > > > > > > > > > > > > > > > > > problem. While it is
> > definitely a
> > > > > > useful
> > > > > > > > > > feature to
> > > > > > > > > > > > > > have,
> > > > > > > > > > > > > > > > > like a
> > > > > > > > > > > > > > > > > > > log to
> > > > > > > > > > > > > > > > > > > > > > > > check what changes were made
> > over
> > > > > time
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > config
> > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > easier to do rollbacks, it is
> > not
> > > > the
> > > > > > > focus
> > > > > > > > > > here.
> > > > > > > > > > > > Here
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > version we
> > > > > > > > > > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > > > > > > > > to say what underlying
> version
> > of
> > > > the
> > > > > > > > plugin
> > > > > > > > > > > > should the
> > > > > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > of the connector use. Perhaps
> > it
> > > is
> > > > > > > better
> > > > > > > > to
> > > > > > > > > > > > change
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > name of
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > parameter from
> > connector.version
> > > to
> > > > > > > > > > > > > > > > connector.plugin.version
> > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > plugin.version if it was
> > > confusing.
> > > > > > wdyt?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2. Any plans to support
> > > assisted
> > > > > > > > migration
> > > > > > > > > > e.g
> > > > > > > > > > > > if a
> > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > connector/config?migrate=latest",
> > > > > the
> > > > > > > > > latest
> > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > > > > > transform the existing
> config
> > > to
> > > > > the
> > > > > > > > newer
> > > > > > > > > > > > version.
> > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > > adding a method like
> "boolean
> > > > > > > > > migrate(Version
> > > > > > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This is an enhancement we can
> > > think
> > > > > of
> > > > > > > > doing
> > > > > > > > > in
> > > > > > > > > > > > future.
> > > > > > > > > > > > > > > > > Users can
> > > > > > > > > > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > > > > > > > a PUT call with the updated
> > > config
> > > > > > which
> > > > > > > > has
> > > > > > > > > > the
> > > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > > > number.
> > > > > > > > > > > > > > > > > > > > > > > > The assisted mode could be
> > handy
> > > as
> > > > > the
> > > > > > > > user
> > > > > > > > > > does
> > > > > > > > > > > > not
> > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > know the
> > > > > > > > > > > > > > > > > > > > > > > > config but beyond this it
> does
> > > not
> > > > > seem
> > > > > > > to
> > > > > > > > > > justify
> > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > existence.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 10:50
> > AM
> > > > > Ashwin
> > > > > > > > > > > > > > > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This is a really useful
> > feature
> > > > and
> > > > > > > > thanks
> > > > > > > > > > for
> > > > > > > > > > > > > > initiating
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I had the following
> > questions -
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on the
> > > > > rejected
> > > > > > > > > > > > alternatives ?
> > > > > > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > config is versioned and
> has a
> > > > > schema.
> > > > > > > > Then
> > > > > > > > > a
> > > > > > > > > > > > single
> > > > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > > > > > dependencies have not
> > changed)
> > > > can
> > > > > > > handle
> > > > > > > > > > > > multiple
> > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2. Any plans to support
> > > assisted
> > > > > > > > migration
> > > > > > > > > > e.g
> > > > > > > > > > > > if a
> > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > connector/config?migrate=latest",
> > > > > the
> > > > > > > > > latest
> > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > > > > > transform the existing
> config
> > > to
> > > > > the
> > > > > > > > newer
> > > > > > > > > > > > version.
> > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > > adding a method like
> "boolean
> > > > > > > > > migrate(Version
> > > > > > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 21, 2022 at
> 2:27
> > PM
> > > > > > > > Snehashis <
> > > > > > > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I'd like to start a
> > > discussion
> > > > > > thread
> > > > > > > > on
> > > > > > > > > > > > KIP-891:
> > > > > > > > > > > > > > > > Running
> > > > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > The KIP aims to add the
> > > ability
> > > > > for
> > > > > > > the
> > > > > > > > > > connect
> > > > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > > > > > versions of a connector.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Please take a look and
> let
> > me
> > > > > know
> > > > > > > what
> > > > > > > > > you
> > > > > > > > > > > > think.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Hey Snehashis,
Thanks for your reply!
> Deviating
> it from the spec seems unnecessary if we document it accordingly, however
> It's probably less intuitive and can lead to confusion. I would just keep
> it as is but making it simpler is also fine.
It sounds like you don't have a strong opinion on this, similar to me.
Chris had a more firm stance on this. I think that you're right that we
could document this, but it would still be the single biggest foot-gun of
the feature, and I think we would regret it later.
> Also for converter versions
> specified as part of the worker configs I believe we concluded that this
> step need not be fatal during worker startup if the required version is
not
> found but LMK if otherwise.
Re-reading our earlier discussion, I think Chris had a strong opinion that
we shouldn't fail on startup because it would be inconsistent. I made an
offhand comment that if this was released in 4.0, we could change it so
both invalid classes and invalid versions cause the worker to fail, which
would be inconsistent but backwards incompatible.
I think in the interest of not breaking compatibility needlessly and
keeping consistent behavior, we should ignore invalid versions in the
worker config.
Thanks,
Greg
On Wed, Aug 21, 2024 at 1:58 AM Snehashis <sn...@gmail.com> wrote:
> Hi Greg,
>
> No issues, I have been caught up in a few things myself.
>
> I have added the points we discussed. In addition, I have added config
> providers as part of the set of plugins which will not support versioning,
> for the same reason as to why it is not supported in the other plugins that
> are initiated on startup.
>
> On whether to deviate from maven versioning for hard requirements as
> discussed between you and Chris.Whether to simply simply specify
> connector.version=1.1.1 as a hard requirement instead of [1.1.1]. Deviating
> it from the spec seems unnecessary if we document it accordingly, however
> It's probably less intuitive and can lead to confusion. I would just keep
> it as is but making it simpler is also fine. Also for converter versions
> specified as part of the worker configs I believe we concluded that this
> step need not be fatal during worker startup if the required version is not
> found but LMK if otherwise.
>
> Regards
> Snehashis
>
>
>
> On Mon, Aug 19, 2024 at 9:24 PM Greg Harris <gr...@aiven.io.invalid>
> wrote:
>
> > Hi Snehashis,
> >
> > Sorry for the late reply.
> >
> > > Heterogeneous dependencies in a multi cluster deployment is highly
> > discouraged
> >
> > Right, this remains unchanged in this KIP.
> >
> > > Let's add the version information for both connector and tasks in the
> > connector status itself
> > > once we add these two additions to the KIP (LMK if you want me to take
> > that up).
> >
> > Could you make these additions?
> >
> > I'm interested to see if we can include this in 4.0.
> >
> > Thanks,
> > Greg
> >
> > On Tue, Jul 2, 2024 at 2:52 AM Snehashis <sn...@gmail.com>
> wrote:
> >
> > > Hi Greg,
> > >
> > > Thanks for taking a look at this, to conclude on the two points above.
> > >
> > > 1. I'm okay with the status quo of leaving the dependency management of
> > > plugins to systems outside of the connect runtime as it is now. Given
> > that
> > > the dependencies are homogenous across a connect cluster, it should
> > ensure
> > > that task and connector versions are uniform across a deployment, after
> > it
> > > has stabilised post an upgrade operation. Heterogeneous dependencies
> in a
> > > multi cluster deployment is highly discouraged and we should point out
> > this
> > > can lead to inconsistent behaviour with connectors and even
> validations.
> > >
> > > 2. Let's add the version information for both connector and tasks in
> the
> > > connector status itself, I don't see another endpoint being required
> for
> > > this.
> > >
> > > I don't have any further points, so if you are okay we can put this to
> a
> > > vote, once we add these two additions to the KIP (LMK if you want me to
> > > take that up).
> > >
> > > Regards
> > > Snehashis
> > >
> > > On Tue, Jul 2, 2024 at 12:08 AM Greg Harris
> <greg.harris@aiven.io.invalid
> > >
> > > wrote:
> > >
> > > > Hey Snehashis,
> > > >
> > > > Sorry for the late reply, and thanks for helping close out the
> > > discussion.
> > > >
> > > > > Note that if my assumptions are correct then
> > > > > this can happen with the existing framework as well, or is there
> some
> > > > > safeguard from this happening?
> > > >
> > > > Currently, if a cluster has a heterogeneous plugin installation, each
> > > > worker may have a different definition of "latest". If you call to
> > > > validate a configuration, it will validate against whatever latest
> > > version
> > > > is present locally.
> > > > With this KIP, that behavior will continue when the version isn't
> > > > specified, when the version is a soft requirement, or when the
> version
> > > is a
> > > > range. For users that do not want to tolerate their tasks having
> > > > heterogeneous versions, the single-version hard requirements are
> there,
> > > but
> > > > come at the cost of micromanagement of upgrades.
> > > > Perhaps there is some room in the future for a connector configured
> > with
> > > a
> > > > range to "pin" the version that all of its tasks should use. Or
> > > > version-aware-scheduling can schedule connectors and tasks to workers
> > > that
> > > > will resolve the same versions. The important primitive is being able
> > to
> > > > enforce which version is being used, everything else looks more like
> a
> > > > convenience feature to me.
> > > >
> > > > > So far, we could have pointed to the
> > > > > misconfigured cluster configuration and somewhat differ this
> problem
> > to
> > > > > something outside of connect runtime.
> > > >
> > > > I think this will still be the case. Even with this feature, we will
> > > still
> > > > recommend homogeneous clusters that have the same set of plugins
> > > installed
> > > > on every worker. Version-aware scheduling is a rejected alternative,
> > and
> > > > your example for performing validation requests on an arbitrary
> worker
> > > is a
> > > > great example of a complication for heterogeneous clusters that we're
> > not
> > > > ready to solve.
> > > >
> > > > > We can introduce
> > > > > a new path under this for version
> > (/connector/connector-name/version),
> > > > but
> > > > > perhaps adding this as part of the status is a valid alternative.
> > > >
> > > > I think either is fine. Adding a new field to the existing endpoints
> is
> > > > fine, and shouldn't require special compatibility such as a query
> > > > parameter.
> > > >
> > > > > Also, to go further I think
> > > > > version information for tasks could also be available,
> > > >
> > > > I completely agree. The version of each connector and each task
> should
> > be
> > > > visible individually, since they may be different during a rolling
> > > upgrade.
> > > >
> > > > Thanks!
> > > > Greg
> > > >
> > > > On Tue, Jun 18, 2024 at 5:18 AM Snehashis <sn...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Greg, Chris
> > > > >
> > > > > Thanks for the in-depth discussion, I have a couple of discussion
> > > points
> > > > > and would like your thoughts on this.
> > > > >
> > > > > 1) One concern I have with the new addition of 'soft' and 'hard'
> > > version
> > > > > requirements is that there could be a mismatch in the plugin
> version
> > > that
> > > > > two different tasks are running, if a soft requirement is provided
> > and
> > > > the
> > > > > nodes a multi cluster deployment are not in sync w.r.t the plugin
> > > > versions
> > > > > that they are configured with. Note that if my assumptions are
> > correct
> > > > then
> > > > > this can happen with the existing framework as well, or is there
> some
> > > > > safeguard from this happening? So far, we could have pointed to the
> > > > > misconfigured cluster configuration and somewhat differ this
> problem
> > to
> > > > > something outside of connect runtime. With this feature in place
> > > perhaps
> > > > > the expectation is more on connect to not be running with such
> > > > > inconsistency, especially if a connector version is specified. This
> > is
> > > > also
> > > > > a problem with validation if different cluster have different
> > > > > configurations, as IIRC validations are local to the worker which
> > > > receives
> > > > > the rest call for validate. So, we might be validating with a
> certain
> > > > > version which is different from the one that will be used to create
> > > > > connector and tasks. Again, this is likely how the current state
> is,
> > > but
> > > > > perhaps such inconsistencies warrant a deeper look with the
> addition
> > of
> > > > > this feature. The problems associated with them can be somewhat
> > > insidious
> > > > > and hard to diagnose.
> > > > >
> > > > > 2) There was some discussion on the need for a new REST endpoint to
> > > > provide
> > > > > information on the versions of running connectors, and I think
> adding
> > > > this
> > > > > information via REST is a valuable addition. The way I see it the
> > > version
> > > > > is an intrinsic property of an instance of a running connector and
> > > hence
> > > > > this should be part of the set of APIs under
> > > /connector/<connector-name>
> > > > > (also the /connectors API should also have this information as it
> is
> > an
> > > > > amalgamation of all the individual connector information). We can
> > > > introduce
> > > > > a new path under this for version
> > (/connector/connector-name/version),
> > > > but
> > > > > perhaps adding this as part of the status is a valid alternative.
> > This
> > > is
> > > > > mentioned as a rejected alternative right now. Also, to go further
> I
> > > > think
> > > > > version information for tasks could also be available, especially
> if
> > we
> > > > > choose to not address the pitfalls discussed in my point 1), this
> > will
> > > > > at-least provide admins a quick and easy way to determine if such
> and
> > > > > inconsistent state exist in any of the connectors.
> > > > >
> > > > > Thanks again for reviving my original KIP and working to improve
> it.
> > > > > Looking forward to your thoughts on the points mentioned above.
> > > > > Regards
> > > > > Snehashis
> > > > >
> > > > >
> > > > > On Wed, May 29, 2024 at 9:59 PM Chris Egerton
> > <chrise@aiven.io.invalid
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Greg,
> > > > > >
> > > > > > First, an apology! I mistakenly assumed that each plugin appeared
> > > only
> > > > > once
> > > > > > in the responses from GET
> /connector-plugins?connectorsOnly=false.
> > > > Thank
> > > > > > you for correcting me and pointing out that all versions of each
> > > plugin
> > > > > > appear in that response, which does indeed satisfy my desire for
> > > users
> > > > to
> > > > > > discover this information in at most two REST requests (and in
> > fact,
> > > > does
> > > > > > it in only one)!
> > > > > >
> > > > > > And secondly, with the revelation about recommenders, I agree
> that
> > > it's
> > > > > > best to leave the "version" property out of the lists of
> properties
> > > > > > returned from the GET /connector-plugins/<plugin>/config
> endpoint.
> > > > > >
> > > > > > With those two points settled, I think the only unresolved item
> is
> > > the
> > > > > > small change to version parsing added to the KIP (where raw
> version
> > > > > numbers
> > > > > > are treated as an exact match, instead of a best-effort match
> with
> > a
> > > > > > fallback on the default version). If the KIP is updated with that
> > > then
> > > > > I'd
> > > > > > be ready to vote on it.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Wed, May 29, 2024 at 12:00 PM Greg Harris
> > > > > <greg.harris@aiven.io.invalid
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hey Chris,
> > > > > > >
> > > > > > > Thanks for your thoughts.
> > > > > > >
> > > > > > > > Won't it still only expose the
> > > > > > > > latest version for each plugin, instead of the range of
> > versions
> > > > > > > available?
> > > > > > >
> > > > > > > Here is a snippet of the current output of the GET
> > > > > > > /connector-plugins?connectorsOnly=false endpoint, after I
> > installed
> > > > two
> > > > > > > versions of the debezium PostgresConnector:
> > > > > > >
> > > > > > > {
> > > > > > > "class":
> > "io.debezium.connector.postgresql.PostgresConnector",
> > > > > > > "type": "source",
> > > > > > > "version": "2.0.1.Final"
> > > > > > > },
> > > > > > > {
> > > > > > > "class":
> > "io.debezium.connector.postgresql.PostgresConnector",
> > > > > > > "type": "source",
> > > > > > > "version": "2.6.1.Final"
> > > > > > > },
> > > > > > >
> > > > > > > I think this satisfies your requirement to learn about all
> > plugins
> > > > and
> > > > > > all
> > > > > > > versions in two or fewer REST calls.
> > > > > > >
> > > > > > > I tried to get an example of the output of `/config` by
> > hardcoding
> > > > the
> > > > > > > Recommender, and realized that Recommenders aren't executed on
> > the
> > > > > > > `/config` endpoint at all: only during validation, when a
> > > > configuration
> > > > > > is
> > > > > > > actually present.
> > > > > > > And this led me to discover that the `/config` endpoint
> returns a
> > > > > > > List<ConfigKeyInfo>, and ConfigKeyInfo does not contain a
> > > > > > recommendedValues
> > > > > > > field. The ConfigValue field is the object which contains
> > > > > > > recommendedValues, and it is only generated during validation.
> > > > > > > I think it's out of scope to start calling recommenders on
> empty
> > > > > configs
> > > > > > > that might throw exceptions, changing the existing REST
> entities,
> > > or
> > > > > > > changing the core ConfigDef implementation.
> > > > > > > Someone could add this functionality later, I don't think it's
> > > > > necessary
> > > > > > > here.
> > > > > > >
> > > > > > > Then the question is: should "version" without recommenders
> > appear
> > > in
> > > > > > > non-connector plugins? I think I'd rather be consistent with
> > > > > "predicate"
> > > > > > > and "negate" on release, and let a later improvement add them.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Greg
> > > > > > >
> > > > > > > On Wed, May 29, 2024 at 8:06 AM Chris Egerton
> > > > <chrise@aiven.io.invalid
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Greg,
> > > > > > > >
> > > > > > > > I'm confused about the behavior for GET
> > > > > > > > /connector-plugins?connectorsOnly=false. Won't it still only
> > > expose
> > > > > the
> > > > > > > > latest version for each plugin, instead of the range of
> > versions
> > > > > > > available?
> > > > > > > >
> > > > > > > > I'm hoping we can provide a flow where people need at most
> two
> > > REST
> > > > > > calls
> > > > > > > > to discover 1) the complete set of plugins available on the
> > > worker
> > > > > > (which
> > > > > > > > is already possible with the endpoint under discussion) and
> 2)
> > > the
> > > > > set
> > > > > > of
> > > > > > > > versions available for a specific plugin on the worker (which
> > > > doesn't
> > > > > > > > appear to be possible, at least for some plugin types). This
> > > > wouldn't
> > > > > > > > require any out-of-band knowledge and would be valuable for
> > > > connector
> > > > > > > users
> > > > > > > > (who may want to, for example, know what their options are
> when
> > > > > > > considering
> > > > > > > > a plugin upgrade) and cluster administrators (who could use
> it
> > > as a
> > > > > > > sanity
> > > > > > > > check for the setup of their Kafka Connect clusters without
> > > having
> > > > to
> > > > > > > pore
> > > > > > > > over log files).
> > > > > > > >
> > > > > > > > As far as modifying the content of the GET
> > > > > > > > /connector-plugins/<plugin>/config endpoint to include a
> > > "version"
> > > > > > > property
> > > > > > > > goes, I think your point about returning that property from
> > > > requests
> > > > > to
> > > > > > > > that endpoint that include a version query parameter is
> > salient,
> > > > but
> > > > > it
> > > > > > > > also unfortunately applies to all types of plugin. I don't
> > think
> > > it
> > > > > > > should
> > > > > > > > completely disqualify that option. I also wasn't imagining
> > adding
> > > > > > > anything
> > > > > > > > besides that single property to that endpoint, so no new
> > > > "predicate"
> > > > > > > > property for SMTs, no new "negate" property for predicates,
> and
> > > no
> > > > > new
> > > > > > > > "type" property for either. I'm not necessarily opposed to
> > adding
> > > > > > those,
> > > > > > > > but it can be done without being pulled into the scope of
> this
> > > KIP.
> > > > > > > >
> > > > > > > > So to summarize how I imagine people might use the REST API
> > after
> > > > > this
> > > > > > > KIP:
> > > > > > > >
> > > > > > > > 1. Discover the set of plugins on the worker via GET
> > > > > > > > /connector-plugins?connectorsOnly=false
> > > > > > > > 2. For any of those plugins, discover the set of available
> > > versions
> > > > > by
> > > > > > > > examining the recommended values for the "version" property
> > after
> > > > > > hitting
> > > > > > > > the GET GET /connector-plugins/<plugin>/config endpoint
> > > > > > > >
> > > > > > > > I don't think this is necessarily optimal since plucking the
> > > > > "version"
> > > > > > > > property out of the response from that endpoint might be a
> PITA
> > > for
> > > > > ad
> > > > > > > hoc
> > > > > > > > queries via, e.g., curl, but I think it'd be enough for this
> > KIP.
> > > > > Still
> > > > > > > > open to other options though, and if I've misunderstood
> > anything
> > > in
> > > > > the
> > > > > > > > proposal, please let me know!
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Wed, May 22, 2024 at 4:05 PM Greg Harris
> > > > > > <greg.harris@aiven.io.invalid
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Chris,
> > > > > > > > >
> > > > > > > > > Thanks for your comments, and I'm glad that it seems like
> > we're
> > > > > > > > > aligning on the vision here.
> > > > > > > > >
> > > > > > > > > > An
> > > > > > > > > > alternative could be to change existing behavior to fail
> > fast
> > > > on
> > > > > > any
> > > > > > > > > > invalid default converter configuration instead of just
> for
> > > > > invalid
> > > > > > > > > > versions
> > > > > > > > >
> > > > > > > > > I suppose if this is landing in 4.0, we have the
> opportunity
> > to
> > > > > break
> > > > > > > > > compatibility and strictly validate the worker class
> configs,
> > > and
> > > > > > > > > could have a new consistent behavior instead of
> inconsistent
> > > but
> > > > > > > > > backwards-compatible behavior.
> > > > > > > > >
> > > > > > > > > > no other part of the KIP requires this change.
> > > > > > > > >
> > > > > > > > > This is correct, and a compelling argument. I'm fine
> leaving
> > > the
> > > > > > > > > strict worker validation off of this configuration to
> > > potentially
> > > > > be
> > > > > > > > > added later. If a KIP was raised to perform strict
> validation
> > > of
> > > > > the
> > > > > > > > > worker config, it would include the version config, and can
> > > > address
> > > > > > > > > backwards compatibility for both configs together.
> > > > > > > > >
> > > > > > > > > > RE exposing the version property in the
> > > > > > > > > /connector-plugins/<plugin>/config
> > > > > > > > > > endpoint, the behavior is inconsistent across plugin
> types.
> > > > > > > > >
> > > > > > > > > Yes it is, and that started in a bugfix, once people
> started
> > > > using
> > > > > > > > > this endpoint:
> > > https://issues.apache.org/jira/browse/KAFKA-14843
> > > > > . I
> > > > > > > > > wasn't planning on converging the behavior in this KIP, as
> I
> > > > > > > > > considered it out-of-scope, and was going to follow the
> > current
> > > > > > > > > behavior. To summarize:
> > > > > > > > >
> > > > > > > > > GET /connector-plugins/<connector>/config will emit:
> > > > > > > > > * connector.class (already implemented)
> > > > > > > > > * connector.version (new)
> > > > > > > > > * key.converter (already implemented)
> > > > > > > > > * key.converter.version (new)
> > > > > > > > > * value.converter (already implemented)
> > > > > > > > > * value.converter.version (new)
> > > > > > > > > * header.converter (already implemented)
> > > > > > > > > * header.converter.version (new)
> > > > > > > > > But will NOT emit:
> > > > > > > > > * transforms.<alias>.type
> > > > > > > > > * transforms.<alias>.version
> > > > > > > > > * transforms.<alias>.predicate
> > > > > > > > > * predicates.<alias>.type
> > > > > > > > > * predicates.<alias>.version
> > > > > > > > > * predicates.<alias>.negate
> > > > > > > > >
> > > > > > > > > GET /connector-plugins/<converter>/config will NOT emit:
> > > > > > > > > * "" (there's not even a ".class" prefix!)
> > > > > > > > > * version
> > > > > > > > >
> > > > > > > > > GET /connector-plugins/<transform>/config will NOT emit:
> > > > > > > > > * type
> > > > > > > > > * version
> > > > > > > > > * predicate
> > > > > > > > >
> > > > > > > > > GET /connector-plugins/<predicate>/config will NOT emit:
> > > > > > > > > * type
> > > > > > > > > * version
> > > > > > > > > * negate
> > > > > > > > >
> > > > > > > > > Do you want the converter, transform, and predicate
> endpoints
> > > > > > changed?
> > > > > > > > > Do you want just "version", or do you want all of the
> > prefixed
> > > > > > configs
> > > > > > > > > including "type", "predicate" and "negate"? How would you
> > want
> > > to
> > > > > > > > > handle the converters?
> > > > > > > > > And when I say the configs are "not part of the plugin
> config
> > > > > itself"
> > > > > > > > > I mean that saying that GET
> > > > > > > > > /connector-plugins/Flatten$Key/config?version=3.8.0 has a
> > > > "version"
> > > > > > > > > config that must be "3.8.0" is a little bit nonsense, as
> the
> > > > > version
> > > > > > > > > is already specified.
> > > > > > > > >
> > > > > > > > > > IMO
> > > > > > > > > > it's worth including this information somewhere directly
> > > > > accessible
> > > > > > > > > without
> > > > > > > > > > having to provide a full connector config. FWIW I'd be
> fine
> > > > with
> > > > > > GET
> > > > > > > > > > /connector-plugins/<plugin>/versions as a first-class
> > > endpoint
> > > > > > > > >
> > > > > > > > > You don't have to provide a configuration to call GET
> > > > > > > > > /connector-plugins?connectorsOnly=false , is that endpoint
> > not
> > > > > close
> > > > > > > > > enough to what you have in mind? See also the Rejected
> > > > Alternative
> > > > > > > > > "Adding new REST API endpoints"
> > > > > > > > >
> > > > > > > > > If you're calling /connector-plugins/<transform>/config,
> you
> > > know
> > > > > the
> > > > > > > > > name of a plugin right? That either comes from out-of-band
> > > > > knowledge,
> > > > > > > > > validating a connector config, or calling GET
> > > > > > > > > /connector-plugins?connectorsOnly=false.
> > > > > > > > > * If you have out-of-band knowledge of plugin classes,
> > perhaps
> > > > you
> > > > > > > > > have out-of-band knowledge of versions too.
> > > > > > > > > * If you've just validated a connector config, there should
> > be
> > > an
> > > > > > > > > accompanying "version" field there with an accurate default
> > > value
> > > > > and
> > > > > > > > > recommenders.
> > > > > > > > > * If you've called GET
> > /connector-plugins?connectorsOnly=false,
> > > > > that
> > > > > > > > > endpoint includes version information.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Greg
> > > > > > > > >
> > > > > > > > > On Wed, May 22, 2024 at 11:05 AM Chris Egerton
> > > > > > <chrise@aiven.io.invalid
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Greg,
> > > > > > > > > >
> > > > > > > > > > Hope you had a nice weekend! Gonna try to keep things
> > > concise.
> > > > > > > > > >
> > > > > > > > > > Concluded points:
> > > > > > > > > >
> > > > > > > > > > RE version recommenders, I agree it's likely that
> > > programmatic
> > > > > UIs
> > > > > > > will
> > > > > > > > > > already be able to handle dynamic configuration
> > definitions,
> > > > and
> > > > > > the
> > > > > > > > > detail
> > > > > > > > > > about SMTs is a great point. I still anticipate some
> > > > awkwardness
> > > > > > with
> > > > > > > > > > connector versions, though: if the latest version
> supports
> > > some
> > > > > new
> > > > > > > > > > properties, then a user switches to an earlier version, a
> > UI
> > > > may
> > > > > > > > respond
> > > > > > > > > by
> > > > > > > > > > wiping values for these properties. I guess we can bite
> the
> > > > > bullet,
> > > > > > > > > though.
> > > > > > > > > >
> > > > > > > > > > RE double-dinging during preflight validation for invalid
> > > > > > versions, I
> > > > > > > > > like
> > > > > > > > > > the analogy with login credentials. I'm convinced that
> the
> > > > > proposal
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > KIP is best 👍
> > > > > > > > > >
> > > > > > > > > > Continued points:
> > > > > > > > > >
> > > > > > > > > > RE failing on worker startup, sorry, I should be clearer:
> > > there
> > > > > is
> > > > > > no
> > > > > > > > > _new_
> > > > > > > > > > justification for it that doesn't also apply to existing
> > > > > behavior.
> > > > > > We
> > > > > > > > > > shouldn't diverge from existing behavior solely for this
> > new
> > > > > case.
> > > > > > An
> > > > > > > > > > alternative could be to change existing behavior to fail
> > fast
> > > > on
> > > > > > any
> > > > > > > > > > invalid default converter configuration instead of just
> for
> > > > > invalid
> > > > > > > > > > versions, but I'd vote to just stick to existing behavior
> > and
> > > > not
> > > > > > > > > > complicate things, especially since no other part of the
> > KIP
> > > > > > requires
> > > > > > > > > this
> > > > > > > > > > change.
> > > > > > > > > >
> > > > > > > > > > RE exposing the version property in the
> > > > > > > > > /connector-plugins/<plugin>/config
> > > > > > > > > > endpoint, the behavior is inconsistent across plugin
> types.
> > > > > Hitting
> > > > > > > the
> > > > > > > > > > endpoint for the FileStreamSinkConnector on version 3.7.0
> > > > yields
> > > > > a
> > > > > > > > > response
> > > > > > > > > > that includes, among other things, the "topics",
> > > > "topics.regex",
> > > > > > and
> > > > > > > > > > "errors.tolerance" properties. I see that we don't do
> this
> > > > > > everywhere
> > > > > > > > > (the
> > > > > > > > > > examples you cite for SMT and converter properties are
> > > > accurate),
> > > > > > but
> > > > > > > > IMO
> > > > > > > > > > it's worth including this information somewhere directly
> > > > > accessible
> > > > > > > > > without
> > > > > > > > > > having to provide a full connector config. FWIW I'd be
> fine
> > > > with
> > > > > > GET
> > > > > > > > > > /connector-plugins/<plugin>/versions as a first-class
> > > endpoint
> > > > > > either
> > > > > > > > > > instead of or in addition to adding recommended values
> for
> > > all
> > > > > > plugin
> > > > > > > > > > versions.
> > > > > > > > > >
> > > > > > > > > > Thanks for your continued work on this KIP, and with the
> > > > progress
> > > > > > > we're
> > > > > > > > > > making I'm optimistic about its chances of appearing in
> > > 4.0.0.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
> > > > > > > > > > Chris
> > > > > > > > > >
> > > > > > > > > > On Wed, May 15, 2024 at 1:22 PM Greg Harris
> > > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey Chris,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your quick follow up.
> > > > > > > > > > >
> > > > > > > > > > > > But this risk is already present with
> > > > > > > > > > > > existing error cases, and I don't see anything that
> > > > justifies
> > > > > > > > > changing
> > > > > > > > > > > > existing behavior with an invalid converter class, or
> > > > > diverging
> > > > > > > > from
> > > > > > > > > it
> > > > > > > > > > > in
> > > > > > > > > > > > the case of invalid converter versions.
> > > > > > > > > > >
> > > > > > > > > > > The justification is to fail-fast, and prevent REST API
> > > users
> > > > > > from
> > > > > > > > > > > receiving errors from bad configs that they didn't
> write,
> > > or
> > > > > > maybe
> > > > > > > > > > > don't even know apply to them.
> > > > > > > > > > > Up until recently errors in these configurations
> surfaced
> > > as
> > > > > > > failures
> > > > > > > > > > > to create the connector, or failures to start, and you
> > made
> > > > > them
> > > > > > > > > > > fail-fast during validation. I think this change is in
> > the
> > > > same
> > > > > > > > > > > spirit, pulling the validation further forward and not
> > > > letting
> > > > > > > errors
> > > > > > > > > > > lie dormant.
> > > > > > > > > > >
> > > > > > > > > > > And to call back to your original concern about
> > > interrupting
> > > > > > > > > > > connectors that explicitly provide these configurations
> > and
> > > > > don't
> > > > > > > use
> > > > > > > > > > > the worker configs: I expect that operators with a
> > majority
> > > > of
> > > > > > > these
> > > > > > > > > > > sorts of clients aren't going to be setting the worker
> > > > .version
> > > > > > > > > > > properties, because it would have no effect on the
> > majority
> > > > of
> > > > > > > their
> > > > > > > > > > > connectors. They would be able to rely on
> > > > > backwards-compatibility
> > > > > > > and
> > > > > > > > > > > continue to ignore the class properties.
> > > > > > > > > > >
> > > > > > > > > > > > CLI
> > > > > > > > > > > > and programmatic UI developers will want to develop
> > their
> > > > own
> > > > > > > > tooling
> > > > > > > > > > > > layers.
> > > > > > > > > > >
> > > > > > > > > > > This is a very compelling argument. Snehashis do you
> want
> > > to
> > > > > > figure
> > > > > > > > > > > out a REST API design for this use-case?
> > > > > > > > > > >
> > > > > > > > > > > > Regarding the GET /connector-plugins/<plugin>/config
> > > > > endpoint,
> > > > > > I
> > > > > > > > was
> > > > > > > > > > > > thinking about the response for non-connector
> plugins,
> > > > e.g.,
> > > > > > > > > > > > GET /connector-plugins/RegexRouter/config. Would a
> > > > "version"
> > > > > > > > property
> > > > > > > > > > > > appear with recommended values?
> > > > > > > > > > >
> > > > > > > > > > > I intended for the ".version" property to be like other
> > > > > framework
> > > > > > > > > > > configs (".class", ".type", ".predicate", ".negate")
> > where
> > > > they
> > > > > > are
> > > > > > > > > > > inside the plugin namespace, but not part of the plugin
> > > > config
> > > > > > > > itself.
> > > > > > > > > > > Perhaps we can deviate from those configs because it
> > would
> > > > aid
> > > > > in
> > > > > > > > > > > discovering other valid `GET
> > > > > > > > > > > /connector-plugins/<plugin>/config?version=` calls,
> > without
> > > > > > calling
> > > > > > > > > > > `GET /connector-plugins?connectorsOnly=false`.
> > > > > > > > > > > I don't really feel strongly either way.
> > > > > > > > > > >
> > > > > > > > > > > > if a user changes the
> > > > > > > > > > > > connector version in, e.g., a dropdown menu, then the
> > UI
> > > > > either
> > > > > > > has
> > > > > > > > > to
> > > > > > > > > > > > re-fetch the ConfigDef for the new version, or risk
> > > > operating
> > > > > > on
> > > > > > > > > stale
> > > > > > > > > > > > information
> > > > > > > > > > >
> > > > > > > > > > > This is a really interesting situation, thanks for
> > finding
> > > > > that!
> > > > > > > This
> > > > > > > > > > > is already a footgun with transformations and
> predicates;
> > > > Once
> > > > > > you
> > > > > > > > > > > fill out a transformation class (which includes a
> > > > recommender)
> > > > > > you
> > > > > > > > > > > will then see all of the transformations
> configurations.
> > > > > > > > > > > I think this means that UI developers should have
> already
> > > > > > developed
> > > > > > > > > > > the infrastructure for handling dynamic recommenders,
> or
> > > else
> > > > > > > they've
> > > > > > > > > > > had this bug since KIP-66 in 2017. It may require some
> > > manual
> > > > > > > > > > > attention to roll-out support for the ".version"
> > properties
> > > > > > > > > > > specifically, but I don't think that should prevent us
> > from
> > > > > > > providing
> > > > > > > > > > > this information in a natural place.
> > > > > > > > > > >
> > > > > > > > > > > > but will it be possible to configure
> > > > > > > > > > > > a connector to use two instances of the same
> transform
> > or
> > > > > > > > predicate,
> > > > > > > > > but
> > > > > > > > > > > > with different versions for each?
> > > > > > > > > > >
> > > > > > > > > > > Yes, and this is included in the example in the KIP,
> > > > > > > > > > > `transforms.flatten-latest` and
> `transforms.flatten-old`.
> > > > This
> > > > > > will
> > > > > > > > > > > work in the natural way, with the two instances having
> a
> > > > > > different
> > > > > > > > > > > version if configured for that. If a user wants to pin
> > the
> > > > same
> > > > > > > > > > > version of a plugin for every instance, they will need
> to
> > > > > provide
> > > > > > > the
> > > > > > > > > > > version config multiple times and keep them in-sync
> > > > themselves.
> > > > > > > > > > >
> > > > > > > > > > > > I would have expected the error
> > > > > > > > > > > > to only be attributed to the version property, and
> for
> > > the
> > > > > > class
> > > > > > > > > property
> > > > > > > > > > > > to be reported as valid.
> > > > > > > > > > >
> > > > > > > > > > > I considered this, and then went back and changed it.
> To
> > > > quote
> > > > > > > > someone
> > > > > > > > > > > else, this is to catch "misspelling cat as dog". For
> > > example
> > > > a
> > > > > > user
> > > > > > > > > > > types DogConverter and meant to type CatConverter, and
> > then
> > > > > > types a
> > > > > > > > > > > version which is valid for CatConverter, conceptually
> the
> > > > error
> > > > > > is
> > > > > > > in
> > > > > > > > > > > the class name and not the version.
> > > > > > > > > > > That's a very contrived scenario, but I think similar
> > > > arguments
> > > > > > are
> > > > > > > > > > > used for attributing validation errors to
> > > > > > endpoints/urls/hostnames
> > > > > > > > > > > when the associated credentials are unable to log into
> > the
> > > > > remote
> > > > > > > > > > > system. Did the user provide the correct testing
> > > credentials,
> > > > > but
> > > > > > > > > > > accidentally typed the production endpoint? Even if the
> > > > > > production
> > > > > > > > > > > endpoint looks valid (it's a real hostname that is
> > > reachable)
> > > > > the
> > > > > > > > > > > conceptual error is still in the hostname and should
> have
> > > the
> > > > > > error
> > > > > > > > > > > attributed to it to draw the user's attention.
> > > > > > > > > > > If that's not convincing, I think the alternative of
> only
> > > > > > > attributing
> > > > > > > > > > > version errors to the version property is also
> > acceptable.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Greg
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 15, 2024 at 8:06 AM Chris Egerton
> > > > > > > > <chrise@aiven.io.invalid
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your responses! Continuations of existing
> > > > > > discussions:
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding crashing the worker on startup--yes, there
> is
> > > > also
> > > > > a
> > > > > > > risk
> > > > > > > > > to
> > > > > > > > > > > > allowing it to join the cluster. But this risk is
> > already
> > > > > > present
> > > > > > > > > with
> > > > > > > > > > > > existing error cases, and I don't see anything that
> > > > justifies
> > > > > > > > > changing
> > > > > > > > > > > > existing behavior with an invalid converter class, or
> > > > > diverging
> > > > > > > > from
> > > > > > > > > it
> > > > > > > > > > > in
> > > > > > > > > > > > the case of invalid converter versions. I think we
> > should
> > > > > keep
> > > > > > > this
> > > > > > > > > > > simple
> > > > > > > > > > > > and not do anything different for worker startup than
> > we
> > > do
> > > > > > > > already.
> > > > > > > > > > > >
> > > > > > > > > > > > As far as REST API vs. metrics go--I think you're
> right
> > > > that
> > > > > > the
> > > > > > > > > original
> > > > > > > > > > > > version metrics were added as a "monitoring" detail.
> > > > However,
> > > > > > > this
> > > > > > > > > was
> > > > > > > > > > > back
> > > > > > > > > > > > when plugin versions were managed solely by cluster
> > > > > > > administrators.
> > > > > > > > > With
> > > > > > > > > > > > this KIP, connector users will be able to manage
> plugin
> > > > > > versions,
> > > > > > > > > and CLI
> > > > > > > > > > > > and programmatic UI developers will want to develop
> > their
> > > > own
> > > > > > > > tooling
> > > > > > > > > > > > layers. I think focusing on the REST API as the
> primary
> > > > > > interface
> > > > > > > > for
> > > > > > > > > > > this
> > > > > > > > > > > > KIP would be best for these users.
> > > > > > > > > > > >
> > > > > > > > > > > > (All that said, I don't object to the metrics that
> are
> > > > > proposed
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > KIP;
> > > > > > > > > > > > I just think they make more sense in addition to new
> > REST
> > > > API
> > > > > > > > > > > > functionality, as opposed to instead of it.)
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding the GET /connector-plugins/<plugin>/config
> > > > > endpoint,
> > > > > > I
> > > > > > > > was
> > > > > > > > > > > > thinking about the response for non-connector
> plugins,
> > > > e.g.,
> > > > > > > > > > > > GET /connector-plugins/RegexRouter/config. Would a
> > > > "version"
> > > > > > > > property
> > > > > > > > > > > > appear with recommended values?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > And new thoughts:
> > > > > > > > > > > >
> > > > > > > > > > > > 1) Regarding the recommended values for
> > > > "connector.version",
> > > > > > this
> > > > > > > > > might
> > > > > > > > > > > be
> > > > > > > > > > > > confusing since there could be differences between
> the
> > > > > > ConfigDefs
> > > > > > > > > for the
> > > > > > > > > > > > latest version of the connector and prior versions.
> It
> > > also
> > > > > > makes
> > > > > > > > the
> > > > > > > > > > > flow
> > > > > > > > > > > > a bit awkward for programmatic UI developers: if a
> user
> > > > > changes
> > > > > > > the
> > > > > > > > > > > > connector version in, e.g., a dropdown menu, then the
> > UI
> > > > > either
> > > > > > > has
> > > > > > > > > to
> > > > > > > > > > > > re-fetch the ConfigDef for the new version, or risk
> > > > operating
> > > > > > on
> > > > > > > > > stale
> > > > > > > > > > > > information. I'm starting to doubt that exposing the
> > > range
> > > > of
> > > > > > > > > available
> > > > > > > > > > > > versions via recommended values is the best way to
> > > proceed,
> > > > > > > instead
> > > > > > > > > of a
> > > > > > > > > > > > more explicit approach like GET
> > > > > > > > > /connector-plugins/<plugin>/versions, or
> > > > > > > > > > > > the "Adding new REST API endpoints" rejected
> > alternative.
> > > > > > > > > > > >
> > > > > > > > > > > > 2) I know that this is a bit heinous, but will it be
> > > > possible
> > > > > > to
> > > > > > > > > > > configure
> > > > > > > > > > > > a connector to use two instances of the same
> transform
> > or
> > > > > > > > predicate,
> > > > > > > > > but
> > > > > > > > > > > > with different versions for each? (I don't think this
> > is
> > > > > worth
> > > > > > > > > > > significant
> > > > > > > > > > > > design/implementation effort, so if it would inflate
> > > either
> > > > > of
> > > > > > > > those,
> > > > > > > > > > > > please don't feel obligated to support it.)
> > > > > > > > > > > >
> > > > > > > > > > > > 3) Just out of curiosity, why double-ding during
> config
> > > > > > > validation
> > > > > > > > > if the
> > > > > > > > > > > > version for a plugin class can't be found? I would
> have
> > > > > > expected
> > > > > > > > the
> > > > > > > > > > > error
> > > > > > > > > > > > to only be attributed to the version property, and
> for
> > > the
> > > > > > class
> > > > > > > > > property
> > > > > > > > > > > > to be reported as valid.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > >
> > > > > > > > > > > > Chris
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, May 14, 2024 at 1:11 PM Greg Harris
> > > > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hey Chris,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your comments. I'm glad you like the
> > > > > motivations,
> > > > > > > > > Snehashis
> > > > > > > > > > > > > wrote that part!
> > > > > > > > > > > > >
> > > > > > > > > > > > > > the configuration syntax for the most basic use
> > case
> > > of
> > > > > > > > > > > > > > specifying a single desired version is pretty
> > > > > > > counterintuitive.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I agree, and the "soft" requirement scheme is
> > > something I
> > > > > > > wasn't
> > > > > > > > > > > > > explicitly looking for, but would be inherited from
> > the
> > > > > > > library.
> > > > > > > > > I'm
> > > > > > > > > > > > > fine with eliminating the soft requirement
> semantics,
> > > and
> > > > > > > having
> > > > > > > > > > > > > "1.0.0" behave the same as "[1.0.0]". I'm less
> > inclined
> > > > to
> > > > > > > > include
> > > > > > > > > > > > > "range" in the property name, or have two
> properties.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Failing startup is drastic and has the potential
> to
> > > > > disrupt
> > > > > > > the
> > > > > > > > > > > > > > availability of connectors that would otherwise
> be
> > > able
> > > > > to
> > > > > > > run
> > > > > > > > > > > healthily
> > > > > > > > > > > > > > because they were explicitly configured to use
> > valid
> > > > > > > converters
> > > > > > > > > > > instead
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > the worker defaults.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think this argument cuts both ways. If someone
> > > > > > reconfigures a
> > > > > > > > > worker
> > > > > > > > > > > > > and adds an invalid ".version" string to the worker
> > (or
> > > > > > changes
> > > > > > > > the
> > > > > > > > > > > > > plugin.path to make it invalid), it would be
> > permitted
> > > to
> > > > > > enter
> > > > > > > > the
> > > > > > > > > > > > > group, and accept work assignments. If those work
> > > > > assignments
> > > > > > > > used
> > > > > > > > > > > > > these configurations, a set of tasks could
> transition
> > > to
> > > > > > FAILED
> > > > > > > > and
> > > > > > > > > > > > > not be able to recover, because they would be
> > restarted
> > > > > again
> > > > > > > on
> > > > > > > > > the
> > > > > > > > > > > > > same worker.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Why are metrics utilized to report information
> > about
> > > > > plugin
> > > > > > > > > versions
> > > > > > > > > > > > > > utilized by connectors at runtime instead of
> > > publishing
> > > > > > this
> > > > > > > > > info in
> > > > > > > > > > > the
> > > > > > > > > > > > > > REST API
> > > > > > > > > > > > >
> > > > > > > > > > > > > I was following the existing pattern for exposing
> > > runtime
> > > > > > > > versions
> > > > > > > > > for
> > > > > > > > > > > > > connectors, and it did seem like a "monitoring"
> > > feature.
> > > > If
> > > > > > > that
> > > > > > > > > > > > > approach is flawed and should be phased out, I
> think
> > it
> > > > > would
> > > > > > > be
> > > > > > > > a
> > > > > > > > > > > > > good idea to reconsider the REST API rejected
> > > > alternative.
> > > > > > > > > > > > > We would need some additional design work to spec
> out
> > > the
> > > > > > REST
> > > > > > > > API
> > > > > > > > > > > > > interface, as I don't have anything in mind
> > currently.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I'm unclear on whether or not it'll be possible
> to
> > > see
> > > > > this
> > > > > > > > > > > information
> > > > > > > > > > > > > via the
> > > > > > > > > > > > > > GET /connector-plugins/<plugin>/config endpoint
> > > > > > > > > > > > >
> > > > > > > > > > > > > There is room in the API to add recommenders for
> > > > > > > "key.converter",
> > > > > > > > > > > > > "value.converter", and "header.converter", but not
> > for
> > > > > > > transforms
> > > > > > > > > and
> > > > > > > > > > > > > predicates, as they include aliases that depend on
> an
> > > > > actual
> > > > > > > > > > > > > configuration. We could explicitly say we're going
> to
> > > do
> > > > > > that,
> > > > > > > or
> > > > > > > > > do
> > > > > > > > > > > > > whatever is convenient during the implementation
> > phase,
> > > > or
> > > > > > > leave
> > > > > > > > it
> > > > > > > > > > > > > open to be improved later.
> > > > > > > > > > > > > There will not be any recommenders for ".version"
> > > > > properties
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > > `/config` endpoint, because those recommenders are
> > > > dynamic
> > > > > > and
> > > > > > > > > depend
> > > > > > > > > > > > > on an actual configuration.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 5) There are two relevant lines in the KIP: "If a
> > > > .version
> > > > > > > > property
> > > > > > > > > > > > > contains a hard requirement, select the latest
> > > installed
> > > > > > > version
> > > > > > > > > which
> > > > > > > > > > > > > satisfies the requirement." and "This configuration
> > is
> > > > > > > > re-evaluated
> > > > > > > > > > > > > each time the connector or task are assigned to a
> new
> > > > > > worker".
> > > > > > > I
> > > > > > > > > would
> > > > > > > > > > > > > call this "eager" upgrade behavior, rather than a
> > > > "sticky"
> > > > > or
> > > > > > > > > "lazy"
> > > > > > > > > > > > > upgrade behavior.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 6) Updated!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Greg
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, May 14, 2024 at 9:14 AM Chris Egerton
> > > > > > > > > <chrise@aiven.io.invalid
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks Greg for updating the KIP, and thanks
> > > Snehashis
> > > > > for
> > > > > > > > > starting
> > > > > > > > > > > the
> > > > > > > > > > > > > > work on this originally.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The motivation section makes a pretty convincing
> > case
> > > > for
> > > > > > > this
> > > > > > > > > kind
> > > > > > > > > > > of
> > > > > > > > > > > > > > feature. My thoughts are mostly about specific
> > > details:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1) I like the support for version ranges (the
> > example
> > > > > > > > > demonstrating
> > > > > > > > > > > how
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > avoid KAFKA-10574 with the header converter was
> > > > > > particularly
> > > > > > > > > > > > > entertaining),
> > > > > > > > > > > > > > but the configuration syntax for the most basic
> use
> > > > case
> > > > > of
> > > > > > > > > > > specifying a
> > > > > > > > > > > > > > single desired version is pretty
> counterintuitive.
> > > > People
> > > > > > may
> > > > > > > > get
> > > > > > > > > > > bitten
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > at least frustrated if they put
> > > connector.version=3.8.0
> > > > > in
> > > > > > a
> > > > > > > > > config
> > > > > > > > > > > but
> > > > > > > > > > > > > > then version 3.7.5 ends up running. I'd like it
> if
> > we
> > > > > could
> > > > > > > > > either
> > > > > > > > > > > > > > intentionally deviate from Maven ranges when a
> bare
> > > > > version
> > > > > > > is
> > > > > > > > > > > present,
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > separate things out into two properties:
> > foo.version
> > > > > would
> > > > > > be
> > > > > > > > the
> > > > > > > > > > > single
> > > > > > > > > > > > > > accepted version for the foo plugin, and
> > > > > foo.version.range
> > > > > > > > would
> > > > > > > > > use
> > > > > > > > > > > > > Maven
> > > > > > > > > > > > > > range syntax. Open to other options too, just
> > > > providing a
> > > > > > > > couple
> > > > > > > > > to
> > > > > > > > > > > get
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > ball rolling.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2) Although the current behavior for a worker
> with
> > an
> > > > > > invalid
> > > > > > > > > > > > > > key/value/header converter class specified in its
> > > > config
> > > > > > file
> > > > > > > > is
> > > > > > > > > a
> > > > > > > > > > > little
> > > > > > > > > > > > > > strange (I was surprised to learn that it
> wouldn't
> > > fail
> > > > > on
> > > > > > > > > startup),
> > > > > > > > > > > I
> > > > > > > > > > > > > > don't see a good reason to deviate from this when
> > an
> > > > > > invalid
> > > > > > > > > version
> > > > > > > > > > > is
> > > > > > > > > > > > > > specified. Failing startup is drastic and has the
> > > > > potential
> > > > > > > to
> > > > > > > > > > > disrupt
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > availability of connectors that would otherwise
> be
> > > able
> > > > > to
> > > > > > > run
> > > > > > > > > > > healthily
> > > > > > > > > > > > > > because they were explicitly configured to use
> > valid
> > > > > > > converters
> > > > > > > > > > > instead
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > the worker defaults.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3) Why are metrics utilized to report information
> > > about
> > > > > > > plugin
> > > > > > > > > > > versions
> > > > > > > > > > > > > > utilized by connectors at runtime instead of
> > > publishing
> > > > > > this
> > > > > > > > > info in
> > > > > > > > > > > the
> > > > > > > > > > > > > > REST API? I saw that this was mentioned as a
> > rejected
> > > > > > > > > alternative,
> > > > > > > > > > > but I
> > > > > > > > > > > > > > didn't get a sense of why. It seems like the REST
> > API
> > > > > would
> > > > > > > be
> > > > > > > > > > > easier to
> > > > > > > > > > > > > > access and more intuitive for most users than new
> > > > > metrics.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 4) In the "Validation" section it's stated that
> > > "Users
> > > > > can
> > > > > > > use
> > > > > > > > > these
> > > > > > > > > > > > > > recommenders to discover the valid plugin classes
> > and
> > > > > > > versions,
> > > > > > > > > > > without
> > > > > > > > > > > > > > requiring an earlier call to GET
> > > > > > > > > > > > > /connector-plugins?connectorsOnly=false."
> > > > > > > > > > > > > > I really like the creativity and simplicity of
> > > reusing
> > > > > the
> > > > > > > > > > > recommender
> > > > > > > > > > > > > > mechanism to expose available versions for
> plugins
> > > via
> > > > > the
> > > > > > > REST
> > > > > > > > > API.
> > > > > > > > > > > I'm
> > > > > > > > > > > > > > unclear on whether or not it'll be possible to
> see
> > > this
> > > > > > > > > information
> > > > > > > > > > > via
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > GET /connector-plugins/<plugin>/config endpoint,
> > > > though.
> > > > > > It'd
> > > > > > > > be
> > > > > > > > > > > great if
> > > > > > > > > > > > > > this were supported, since we learned in KIP-769
> > [1]
> > > > that
> > > > > > > > people
> > > > > > > > > > > really
> > > > > > > > > > > > > > want to be able to see configuration options for
> > > > > connectors
> > > > > > > and
> > > > > > > > > their
> > > > > > > > > > > > > > plugins via some kind of GET endpoint without
> > having
> > > to
> > > > > > > > provide a
> > > > > > > > > > > > > complete
> > > > > > > > > > > > > > connector config for validation.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 5) In the Maven version range docs, it's stated
> > that
> > > > > "Maven
> > > > > > > > > picks the
> > > > > > > > > > > > > > highest version of each project that satisfies
> all
> > > the
> > > > > hard
> > > > > > > > > > > requirements
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > the dependencies on that project." I'm guessing
> > this
> > > > > > behavior
> > > > > > > > > will be
> > > > > > > > > > > > > > retained for Connect; i.e., the highest-possible
> > > > version
> > > > > of
> > > > > > > > each
> > > > > > > > > > > plugin
> > > > > > > > > > > > > > that satisfies the user-specified version
> > constraints
> > > > > will
> > > > > > be
> > > > > > > > > run?
> > > > > > > > > > > (An
> > > > > > > > > > > > > > alternative approach could be to have some kind
> of
> > > > > "sticky"
> > > > > > > > logic
> > > > > > > > > > > that
> > > > > > > > > > > > > only
> > > > > > > > > > > > > > restarts connectors/tasks when their
> currently-used
> > > > > version
> > > > > > > > > becomes
> > > > > > > > > > > > > > incompatible with the configured constraints.)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 6) (Nit) It'd be nice to add a link to the
> > > TestPlugins
> > > > > > class
> > > > > > > or
> > > > > > > > > > > somewhere
> > > > > > > > > > > > > > in its neighborhood to the testing plan;
> unfamiliar
> > > > > readers
> > > > > > > > > probably
> > > > > > > > > > > > > won't
> > > > > > > > > > > > > > get much out of what's there right now.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1] -
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, May 13, 2024 at 2:01 PM Snehashis <
> > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > That is much appreciated. No complaints on the
> > > > > additional
> > > > > > > > > scope, I
> > > > > > > > > > > will
> > > > > > > > > > > > > > > make some time out to work on this once we have
> > > > > approval.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> > > > > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hey Snehashis,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'm glad to hear you're still interested in
> > this
> > > > KIP!
> > > > > > > > > > > > > > > > I'm happy to let you drive this, and I
> > apologize
> > > > for
> > > > > > > > > increasing
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > scope of work so drastically. To make up for
> > > that,
> > > > > I'll
> > > > > > > > > > > volunteer to
> > > > > > > > > > > > > > > > be the primary PR reviewer to help get this
> > done
> > > > > > quickly
> > > > > > > > once
> > > > > > > > > > > the KIP
> > > > > > > > > > > > > > > > is approved.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 3:51 AM Snehashis <
> > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for the follow up to my original
> KIP,
> > I
> > > am
> > > > > in
> > > > > > > > > favour of
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > additions made to expand its scope, the
> > > addition
> > > > of
> > > > > > > range
> > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > specifically make a lot of sense.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Apologies if I have not publicly worked on
> > this
> > > > KIP
> > > > > > > for a
> > > > > > > > > long
> > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > original work was done when the move to
> > service
> > > > > > loading
> > > > > > > > > was in
> > > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > > and I wanted to loop back to this only
> after
> > > that
> > > > > > work
> > > > > > > > was
> > > > > > > > > > > > > completed.
> > > > > > > > > > > > > > > > Post
> > > > > > > > > > > > > > > > > its conclusion, I have not been able to
> take
> > > this
> > > > > up
> > > > > > > due
> > > > > > > > to
> > > > > > > > > > > other
> > > > > > > > > > > > > > > > > priorities. If it's okay with you, I would
> > > still
> > > > > like
> > > > > > > to
> > > > > > > > > get
> > > > > > > > > > > this
> > > > > > > > > > > > > > > > > implemented myself, including the
> additional
> > > > scope.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks and regards
> > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 12:45 AM Greg
> Harris
> > > > > > > > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'd like to reboot the discussion on
> > KIP-891:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I've made some changes, most notably:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. Specifying versions for all plugins in
> > > > > Connector
> > > > > > > > > configs
> > > > > > > > > > > > > > > > > > (converters, header converters,
> transforms,
> > > and
> > > > > > > > > predicates)
> > > > > > > > > > > not
> > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > connectors & tasks
> > > > > > > > > > > > > > > > > > 2. Specifying a range of versions instead
> > of
> > > an
> > > > > > exact
> > > > > > > > > match
> > > > > > > > > > > > > > > > > > 3. New metrics to observe what versions
> are
> > > > > in-use
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks to Snehashis for the original KIP
> > > idea!
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg
> > Harris <
> > > > > > > > > > > > > greg.harris@aiven.io>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Snehashis,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thank you for the KIP! This is
> something
> > > I've
> > > > > > > wanted
> > > > > > > > > for a
> > > > > > > > > > > long
> > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I know the discussion has gone cold,
> are
> > > you
> > > > > > still
> > > > > > > > > > > interested
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > pursuing this feature? I'll make time
> to
> > > > review
> > > > > > the
> > > > > > > > > KIP if
> > > > > > > > > > > you
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > still accepting comments.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM
> > Snehashis
> > > <
> > > > > > > > > > > > > > > snehashisp1995@gmail.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for the points Sagar.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1) Should we update the GET
> > /connectors
> > > > > > > endpoint
> > > > > > > > to
> > > > > > > > > > > > > include the
> > > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > > the plugin that is running? It
> could
> > be
> > > > > > useful
> > > > > > > to
> > > > > > > > > > > figure
> > > > > > > > > > > > > out
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > the plugin or I am assuming it gets
> > > > > returned
> > > > > > by
> > > > > > > > the
> > > > > > > > > > > > > expand=info
> > > > > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think this is good to have and
> > possible
> > > > > > future
> > > > > > > > > > > > > enhancement. The
> > > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > info will be present in the config of
> > the
> > > > > > > connector
> > > > > > > > > if
> > > > > > > > > > > the
> > > > > > > > > > > > > user
> > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > > specified the version. Otherwise it
> is
> > > the
> > > > > > latest
> > > > > > > > > version
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > > can find out from the
> connector-plugin
> > > > > > endpoint.
> > > > > > > > The
> > > > > > > > > > > > > information
> > > > > > > > > > > > > > > > can be
> > > > > > > > > > > > > > > > > > > > introduced to the response of the GET
> > > > > > /connectors
> > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > itself,
> > > > > > > > > > > > > > > > > > however
> > > > > > > > > > > > > > > > > > > > the most ideal way of doing this
> would
> > be
> > > > to
> > > > > > get
> > > > > > > > the
> > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > > > instance of the connector and get the
> > > > version
> > > > > > > > > directly
> > > > > > > > > > > from
> > > > > > > > > > > > > > > there.
> > > > > > > > > > > > > > > > > > This is
> > > > > > > > > > > > > > > > > > > > slightly tricky as the connector
> could
> > be
> > > > > > running
> > > > > > > > in
> > > > > > > > > a
> > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > node.
> > > > > > > > > > > > > > > > > > > > One way to do this would be to
> persist
> > > the
> > > > > > > version
> > > > > > > > > > > > > information in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > status backing store during
> > instantiation
> > > > of
> > > > > > the
> > > > > > > > > > > connector.
> > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > requires
> > > > > > > > > > > > > > > > > > > > some more thought and since the
> version
> > > is
> > > > > part
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > > configs if
> > > > > > > > > > > > > > > > > > provided
> > > > > > > > > > > > > > > > > > > > and evident otherwise, I have not
> > > included
> > > > it
> > > > > > in
> > > > > > > > this
> > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2) I am not aware of this and hence
> > > > asking,
> > > > > > > can 2
> > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > versions have the same name? Does
> the
> > > > > plugin
> > > > > > > > > isolation
> > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > > could have a bearing when using the
> > > > > lifecycle
> > > > > > > > > > > endpoints for
> > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > All connectors in a cluster need to
> > have
> > > > > > uniquire
> > > > > > > > > > > connector
> > > > > > > > > > > > > names
> > > > > > > > > > > > > > > > > > > > regardless of what version of the
> > plugin
> > > > the
> > > > > > > > > connector is
> > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > > > underneath. This is something
> enforced
> > by
> > > > the
> > > > > > > > connect
> > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > itself.
> > > > > > > > > > > > > > > > > > All
> > > > > > > > > > > > > > > > > > > > connect CRUD operations are keyed on
> > the
> > > > > > > connector
> > > > > > > > > name
> > > > > > > > > > > so
> > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > be an issue.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM
> Sagar <
> > > > > > > > > > > > > sagarmeansocean@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hey Snehashsih,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for the KIP. It looks like a
> > > very
> > > > > > useful
> > > > > > > > > > > feature.
> > > > > > > > > > > > > Couple
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > small-ish points, let me know what
> > you
> > > > > think:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1) Should we update the GET
> > /connectors
> > > > > > > endpoint
> > > > > > > > to
> > > > > > > > > > > > > include the
> > > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > > the plugin that is running? It
> could
> > be
> > > > > > useful
> > > > > > > to
> > > > > > > > > > > figure
> > > > > > > > > > > > > out
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > > the plugin or I am assuming it gets
> > > > > returned
> > > > > > by
> > > > > > > > the
> > > > > > > > > > > > > expand=info
> > > > > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > > > > > 2) I am not aware of this and hence
> > > > asking,
> > > > > > > can 2
> > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > versions have the same name? Does
> the
> > > > > plugin
> > > > > > > > > isolation
> > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > > could have a bearing when using the
> > > > > lifecycle
> > > > > > > > > > > endpoints for
> > > > > > > > > > > > > > > > > > connectors like
> > > > > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > > > Sagar.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM
> > Ashwin
> > > > > > > > > > > > > > > > <apankaj@confluent.io.invalid
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > IIUC (please correct me if I am
> > > wrong
> > > > > > > here),
> > > > > > > > > what
> > > > > > > > > > > you
> > > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > a versioning scheme for a
> connector
> > > > > config
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > same
> > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > different versions of a connector
> > > > > plugin).
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Sorry for not being more precise
> in
> > > my
> > > > > > > wording
> > > > > > > > > - I
> > > > > > > > > > > meant
> > > > > > > > > > > > > > > > > > registering
> > > > > > > > > > > > > > > > > > > > > > versions of schema for connector
> > > > config.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Let's take the example of a
> > fictional
> > > > > > > connector
> > > > > > > > > which
> > > > > > > > > > > > > uses a
> > > > > > > > > > > > > > > > > > fictional
> > > > > > > > > > > > > > > > > > > > > AWS
> > > > > > > > > > > > > > > > > > > > > > service.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Fictional Connector Config schema
> > > > > > version:2.0
> > > > > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > > > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Fictional Connector config schema
> > > > > > version:3.0
> > > > > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > "iam_role": {
> > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > > > > "iam_role"
> > > > > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The connector which supports
> > > Fictional
> > > > > > config
> > > > > > > > > schema
> > > > > > > > > > > 2.0
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > validate
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > access key and secret key.
> > > > > > > > > > > > > > > > > > > > > > Whereas a connector which
> supports
> > > > config
> > > > > > > with
> > > > > > > > > schema
> > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > 3.0
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > > > > > validate the IAM role.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This is the alternative which I
> > > wanted
> > > > to
> > > > > > > > > suggest.
> > > > > > > > > > > Each
> > > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > register the schema versions of
> > > > connector
> > > > > > > > config
> > > > > > > > > > > which it
> > > > > > > > > > > > > > > > supports.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The plugin paths may be
> optionally
> > > > > > different
> > > > > > > > > i.e we
> > > > > > > > > > > > > don't
> > > > > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > > > > > mandatorily add a new plugin path
> > to
> > > > > > support
> > > > > > > a
> > > > > > > > > new
> > > > > > > > > > > schema
> > > > > > > > > > > > > > > > version.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:47 PM
> > > > > Snehashis
> > > > > > <
> > > > > > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on the
> > > > rejected
> > > > > > > > > > > alternatives ?
> > > > > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > config is versioned and has a
> > > > schema.
> > > > > > > Then
> > > > > > > > a
> > > > > > > > > > > single
> > > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > > > > dependencies have not
> changed)
> > > can
> > > > > > handle
> > > > > > > > > > > multiple
> > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > IIUC (please correct me if I am
> > > wrong
> > > > > > > here),
> > > > > > > > > what
> > > > > > > > > > > you
> > > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > a versioning scheme for a
> > connector
> > > > > > config
> > > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > connector (and
> > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > different versions of a
> connector
> > > > > > plugin).
> > > > > > > > > That is
> > > > > > > > > > > a
> > > > > > > > > > > > > > > somewhat
> > > > > > > > > > > > > > > > > > > > > tangential
> > > > > > > > > > > > > > > > > > > > > > > problem. While it is
> definitely a
> > > > > useful
> > > > > > > > > feature to
> > > > > > > > > > > > > have,
> > > > > > > > > > > > > > > > like a
> > > > > > > > > > > > > > > > > > log to
> > > > > > > > > > > > > > > > > > > > > > > check what changes were made
> over
> > > > time
> > > > > to
> > > > > > > the
> > > > > > > > > > > config
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > easier to do rollbacks, it is
> not
> > > the
> > > > > > focus
> > > > > > > > > here.
> > > > > > > > > > > Here
> > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > version we
> > > > > > > > > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > > > > > > > to say what underlying version
> of
> > > the
> > > > > > > plugin
> > > > > > > > > > > should the
> > > > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > of the connector use. Perhaps
> it
> > is
> > > > > > better
> > > > > > > to
> > > > > > > > > > > change
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > name of
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > parameter from
> connector.version
> > to
> > > > > > > > > > > > > > > connector.plugin.version
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > plugin.version if it was
> > confusing.
> > > > > wdyt?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2. Any plans to support
> > assisted
> > > > > > > migration
> > > > > > > > > e.g
> > > > > > > > > > > if a
> > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > > > >
> > connector/config?migrate=latest",
> > > > the
> > > > > > > > latest
> > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > > > > transform the existing config
> > to
> > > > the
> > > > > > > newer
> > > > > > > > > > > version.
> > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > adding a method like "boolean
> > > > > > > > migrate(Version
> > > > > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This is an enhancement we can
> > think
> > > > of
> > > > > > > doing
> > > > > > > > in
> > > > > > > > > > > future.
> > > > > > > > > > > > > > > > Users can
> > > > > > > > > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > > > > > > a PUT call with the updated
> > config
> > > > > which
> > > > > > > has
> > > > > > > > > the
> > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > > number.
> > > > > > > > > > > > > > > > > > > > > > > The assisted mode could be
> handy
> > as
> > > > the
> > > > > > > user
> > > > > > > > > does
> > > > > > > > > > > not
> > > > > > > > > > > > > need
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > know the
> > > > > > > > > > > > > > > > > > > > > > > config but beyond this it does
> > not
> > > > seem
> > > > > > to
> > > > > > > > > justify
> > > > > > > > > > > its
> > > > > > > > > > > > > > > > existence.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 10:50
> AM
> > > > Ashwin
> > > > > > > > > > > > > > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This is a really useful
> feature
> > > and
> > > > > > > thanks
> > > > > > > > > for
> > > > > > > > > > > > > initiating
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I had the following
> questions -
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on the
> > > > rejected
> > > > > > > > > > > alternatives ?
> > > > > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > config is versioned and has a
> > > > schema.
> > > > > > > Then
> > > > > > > > a
> > > > > > > > > > > single
> > > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > > > > dependencies have not
> changed)
> > > can
> > > > > > handle
> > > > > > > > > > > multiple
> > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2. Any plans to support
> > assisted
> > > > > > > migration
> > > > > > > > > e.g
> > > > > > > > > > > if a
> > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > > > >
> > connector/config?migrate=latest",
> > > > the
> > > > > > > > latest
> > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > > > > transform the existing config
> > to
> > > > the
> > > > > > > newer
> > > > > > > > > > > version.
> > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > adding a method like "boolean
> > > > > > > > migrate(Version
> > > > > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 21, 2022 at 2:27
> PM
> > > > > > > Snehashis <
> > > > > > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I'd like to start a
> > discussion
> > > > > thread
> > > > > > > on
> > > > > > > > > > > KIP-891:
> > > > > > > > > > > > > > > Running
> > > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > The KIP aims to add the
> > ability
> > > > for
> > > > > > the
> > > > > > > > > connect
> > > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > > > > versions of a connector.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Please take a look and let
> me
> > > > know
> > > > > > what
> > > > > > > > you
> > > > > > > > > > > think.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Snehashis <sn...@gmail.com>.
Hi Greg,
No issues, I have been caught up in a few things myself.
I have added the points we discussed. In addition, I have added config
providers as part of the set of plugins which will not support versioning,
for the same reason as to why it is not supported in the other plugins that
are initiated on startup.
On whether to deviate from maven versioning for hard requirements as
discussed between you and Chris.Whether to simply simply specify
connector.version=1.1.1 as a hard requirement instead of [1.1.1]. Deviating
it from the spec seems unnecessary if we document it accordingly, however
It's probably less intuitive and can lead to confusion. I would just keep
it as is but making it simpler is also fine. Also for converter versions
specified as part of the worker configs I believe we concluded that this
step need not be fatal during worker startup if the required version is not
found but LMK if otherwise.
Regards
Snehashis
On Mon, Aug 19, 2024 at 9:24 PM Greg Harris <gr...@aiven.io.invalid>
wrote:
> Hi Snehashis,
>
> Sorry for the late reply.
>
> > Heterogeneous dependencies in a multi cluster deployment is highly
> discouraged
>
> Right, this remains unchanged in this KIP.
>
> > Let's add the version information for both connector and tasks in the
> connector status itself
> > once we add these two additions to the KIP (LMK if you want me to take
> that up).
>
> Could you make these additions?
>
> I'm interested to see if we can include this in 4.0.
>
> Thanks,
> Greg
>
> On Tue, Jul 2, 2024 at 2:52 AM Snehashis <sn...@gmail.com> wrote:
>
> > Hi Greg,
> >
> > Thanks for taking a look at this, to conclude on the two points above.
> >
> > 1. I'm okay with the status quo of leaving the dependency management of
> > plugins to systems outside of the connect runtime as it is now. Given
> that
> > the dependencies are homogenous across a connect cluster, it should
> ensure
> > that task and connector versions are uniform across a deployment, after
> it
> > has stabilised post an upgrade operation. Heterogeneous dependencies in a
> > multi cluster deployment is highly discouraged and we should point out
> this
> > can lead to inconsistent behaviour with connectors and even validations.
> >
> > 2. Let's add the version information for both connector and tasks in the
> > connector status itself, I don't see another endpoint being required for
> > this.
> >
> > I don't have any further points, so if you are okay we can put this to a
> > vote, once we add these two additions to the KIP (LMK if you want me to
> > take that up).
> >
> > Regards
> > Snehashis
> >
> > On Tue, Jul 2, 2024 at 12:08 AM Greg Harris <greg.harris@aiven.io.invalid
> >
> > wrote:
> >
> > > Hey Snehashis,
> > >
> > > Sorry for the late reply, and thanks for helping close out the
> > discussion.
> > >
> > > > Note that if my assumptions are correct then
> > > > this can happen with the existing framework as well, or is there some
> > > > safeguard from this happening?
> > >
> > > Currently, if a cluster has a heterogeneous plugin installation, each
> > > worker may have a different definition of "latest". If you call to
> > > validate a configuration, it will validate against whatever latest
> > version
> > > is present locally.
> > > With this KIP, that behavior will continue when the version isn't
> > > specified, when the version is a soft requirement, or when the version
> > is a
> > > range. For users that do not want to tolerate their tasks having
> > > heterogeneous versions, the single-version hard requirements are there,
> > but
> > > come at the cost of micromanagement of upgrades.
> > > Perhaps there is some room in the future for a connector configured
> with
> > a
> > > range to "pin" the version that all of its tasks should use. Or
> > > version-aware-scheduling can schedule connectors and tasks to workers
> > that
> > > will resolve the same versions. The important primitive is being able
> to
> > > enforce which version is being used, everything else looks more like a
> > > convenience feature to me.
> > >
> > > > So far, we could have pointed to the
> > > > misconfigured cluster configuration and somewhat differ this problem
> to
> > > > something outside of connect runtime.
> > >
> > > I think this will still be the case. Even with this feature, we will
> > still
> > > recommend homogeneous clusters that have the same set of plugins
> > installed
> > > on every worker. Version-aware scheduling is a rejected alternative,
> and
> > > your example for performing validation requests on an arbitrary worker
> > is a
> > > great example of a complication for heterogeneous clusters that we're
> not
> > > ready to solve.
> > >
> > > > We can introduce
> > > > a new path under this for version
> (/connector/connector-name/version),
> > > but
> > > > perhaps adding this as part of the status is a valid alternative.
> > >
> > > I think either is fine. Adding a new field to the existing endpoints is
> > > fine, and shouldn't require special compatibility such as a query
> > > parameter.
> > >
> > > > Also, to go further I think
> > > > version information for tasks could also be available,
> > >
> > > I completely agree. The version of each connector and each task should
> be
> > > visible individually, since they may be different during a rolling
> > upgrade.
> > >
> > > Thanks!
> > > Greg
> > >
> > > On Tue, Jun 18, 2024 at 5:18 AM Snehashis <sn...@gmail.com>
> > > wrote:
> > >
> > > > Hi Greg, Chris
> > > >
> > > > Thanks for the in-depth discussion, I have a couple of discussion
> > points
> > > > and would like your thoughts on this.
> > > >
> > > > 1) One concern I have with the new addition of 'soft' and 'hard'
> > version
> > > > requirements is that there could be a mismatch in the plugin version
> > that
> > > > two different tasks are running, if a soft requirement is provided
> and
> > > the
> > > > nodes a multi cluster deployment are not in sync w.r.t the plugin
> > > versions
> > > > that they are configured with. Note that if my assumptions are
> correct
> > > then
> > > > this can happen with the existing framework as well, or is there some
> > > > safeguard from this happening? So far, we could have pointed to the
> > > > misconfigured cluster configuration and somewhat differ this problem
> to
> > > > something outside of connect runtime. With this feature in place
> > perhaps
> > > > the expectation is more on connect to not be running with such
> > > > inconsistency, especially if a connector version is specified. This
> is
> > > also
> > > > a problem with validation if different cluster have different
> > > > configurations, as IIRC validations are local to the worker which
> > > receives
> > > > the rest call for validate. So, we might be validating with a certain
> > > > version which is different from the one that will be used to create
> > > > connector and tasks. Again, this is likely how the current state is,
> > but
> > > > perhaps such inconsistencies warrant a deeper look with the addition
> of
> > > > this feature. The problems associated with them can be somewhat
> > insidious
> > > > and hard to diagnose.
> > > >
> > > > 2) There was some discussion on the need for a new REST endpoint to
> > > provide
> > > > information on the versions of running connectors, and I think adding
> > > this
> > > > information via REST is a valuable addition. The way I see it the
> > version
> > > > is an intrinsic property of an instance of a running connector and
> > hence
> > > > this should be part of the set of APIs under
> > /connector/<connector-name>
> > > > (also the /connectors API should also have this information as it is
> an
> > > > amalgamation of all the individual connector information). We can
> > > introduce
> > > > a new path under this for version
> (/connector/connector-name/version),
> > > but
> > > > perhaps adding this as part of the status is a valid alternative.
> This
> > is
> > > > mentioned as a rejected alternative right now. Also, to go further I
> > > think
> > > > version information for tasks could also be available, especially if
> we
> > > > choose to not address the pitfalls discussed in my point 1), this
> will
> > > > at-least provide admins a quick and easy way to determine if such and
> > > > inconsistent state exist in any of the connectors.
> > > >
> > > > Thanks again for reviving my original KIP and working to improve it.
> > > > Looking forward to your thoughts on the points mentioned above.
> > > > Regards
> > > > Snehashis
> > > >
> > > >
> > > > On Wed, May 29, 2024 at 9:59 PM Chris Egerton
> <chrise@aiven.io.invalid
> > >
> > > > wrote:
> > > >
> > > > > Hi Greg,
> > > > >
> > > > > First, an apology! I mistakenly assumed that each plugin appeared
> > only
> > > > once
> > > > > in the responses from GET /connector-plugins?connectorsOnly=false.
> > > Thank
> > > > > you for correcting me and pointing out that all versions of each
> > plugin
> > > > > appear in that response, which does indeed satisfy my desire for
> > users
> > > to
> > > > > discover this information in at most two REST requests (and in
> fact,
> > > does
> > > > > it in only one)!
> > > > >
> > > > > And secondly, with the revelation about recommenders, I agree that
> > it's
> > > > > best to leave the "version" property out of the lists of properties
> > > > > returned from the GET /connector-plugins/<plugin>/config endpoint.
> > > > >
> > > > > With those two points settled, I think the only unresolved item is
> > the
> > > > > small change to version parsing added to the KIP (where raw version
> > > > numbers
> > > > > are treated as an exact match, instead of a best-effort match with
> a
> > > > > fallback on the default version). If the KIP is updated with that
> > then
> > > > I'd
> > > > > be ready to vote on it.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Wed, May 29, 2024 at 12:00 PM Greg Harris
> > > > <greg.harris@aiven.io.invalid
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hey Chris,
> > > > > >
> > > > > > Thanks for your thoughts.
> > > > > >
> > > > > > > Won't it still only expose the
> > > > > > > latest version for each plugin, instead of the range of
> versions
> > > > > > available?
> > > > > >
> > > > > > Here is a snippet of the current output of the GET
> > > > > > /connector-plugins?connectorsOnly=false endpoint, after I
> installed
> > > two
> > > > > > versions of the debezium PostgresConnector:
> > > > > >
> > > > > > {
> > > > > > "class":
> "io.debezium.connector.postgresql.PostgresConnector",
> > > > > > "type": "source",
> > > > > > "version": "2.0.1.Final"
> > > > > > },
> > > > > > {
> > > > > > "class":
> "io.debezium.connector.postgresql.PostgresConnector",
> > > > > > "type": "source",
> > > > > > "version": "2.6.1.Final"
> > > > > > },
> > > > > >
> > > > > > I think this satisfies your requirement to learn about all
> plugins
> > > and
> > > > > all
> > > > > > versions in two or fewer REST calls.
> > > > > >
> > > > > > I tried to get an example of the output of `/config` by
> hardcoding
> > > the
> > > > > > Recommender, and realized that Recommenders aren't executed on
> the
> > > > > > `/config` endpoint at all: only during validation, when a
> > > configuration
> > > > > is
> > > > > > actually present.
> > > > > > And this led me to discover that the `/config` endpoint returns a
> > > > > > List<ConfigKeyInfo>, and ConfigKeyInfo does not contain a
> > > > > recommendedValues
> > > > > > field. The ConfigValue field is the object which contains
> > > > > > recommendedValues, and it is only generated during validation.
> > > > > > I think it's out of scope to start calling recommenders on empty
> > > > configs
> > > > > > that might throw exceptions, changing the existing REST entities,
> > or
> > > > > > changing the core ConfigDef implementation.
> > > > > > Someone could add this functionality later, I don't think it's
> > > > necessary
> > > > > > here.
> > > > > >
> > > > > > Then the question is: should "version" without recommenders
> appear
> > in
> > > > > > non-connector plugins? I think I'd rather be consistent with
> > > > "predicate"
> > > > > > and "negate" on release, and let a later improvement add them.
> > > > > >
> > > > > > Thanks,
> > > > > > Greg
> > > > > >
> > > > > > On Wed, May 29, 2024 at 8:06 AM Chris Egerton
> > > <chrise@aiven.io.invalid
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Greg,
> > > > > > >
> > > > > > > I'm confused about the behavior for GET
> > > > > > > /connector-plugins?connectorsOnly=false. Won't it still only
> > expose
> > > > the
> > > > > > > latest version for each plugin, instead of the range of
> versions
> > > > > > available?
> > > > > > >
> > > > > > > I'm hoping we can provide a flow where people need at most two
> > REST
> > > > > calls
> > > > > > > to discover 1) the complete set of plugins available on the
> > worker
> > > > > (which
> > > > > > > is already possible with the endpoint under discussion) and 2)
> > the
> > > > set
> > > > > of
> > > > > > > versions available for a specific plugin on the worker (which
> > > doesn't
> > > > > > > appear to be possible, at least for some plugin types). This
> > > wouldn't
> > > > > > > require any out-of-band knowledge and would be valuable for
> > > connector
> > > > > > users
> > > > > > > (who may want to, for example, know what their options are when
> > > > > > considering
> > > > > > > a plugin upgrade) and cluster administrators (who could use it
> > as a
> > > > > > sanity
> > > > > > > check for the setup of their Kafka Connect clusters without
> > having
> > > to
> > > > > > pore
> > > > > > > over log files).
> > > > > > >
> > > > > > > As far as modifying the content of the GET
> > > > > > > /connector-plugins/<plugin>/config endpoint to include a
> > "version"
> > > > > > property
> > > > > > > goes, I think your point about returning that property from
> > > requests
> > > > to
> > > > > > > that endpoint that include a version query parameter is
> salient,
> > > but
> > > > it
> > > > > > > also unfortunately applies to all types of plugin. I don't
> think
> > it
> > > > > > should
> > > > > > > completely disqualify that option. I also wasn't imagining
> adding
> > > > > > anything
> > > > > > > besides that single property to that endpoint, so no new
> > > "predicate"
> > > > > > > property for SMTs, no new "negate" property for predicates, and
> > no
> > > > new
> > > > > > > "type" property for either. I'm not necessarily opposed to
> adding
> > > > > those,
> > > > > > > but it can be done without being pulled into the scope of this
> > KIP.
> > > > > > >
> > > > > > > So to summarize how I imagine people might use the REST API
> after
> > > > this
> > > > > > KIP:
> > > > > > >
> > > > > > > 1. Discover the set of plugins on the worker via GET
> > > > > > > /connector-plugins?connectorsOnly=false
> > > > > > > 2. For any of those plugins, discover the set of available
> > versions
> > > > by
> > > > > > > examining the recommended values for the "version" property
> after
> > > > > hitting
> > > > > > > the GET GET /connector-plugins/<plugin>/config endpoint
> > > > > > >
> > > > > > > I don't think this is necessarily optimal since plucking the
> > > > "version"
> > > > > > > property out of the response from that endpoint might be a PITA
> > for
> > > > ad
> > > > > > hoc
> > > > > > > queries via, e.g., curl, but I think it'd be enough for this
> KIP.
> > > > Still
> > > > > > > open to other options though, and if I've misunderstood
> anything
> > in
> > > > the
> > > > > > > proposal, please let me know!
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Wed, May 22, 2024 at 4:05 PM Greg Harris
> > > > > <greg.harris@aiven.io.invalid
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey Chris,
> > > > > > > >
> > > > > > > > Thanks for your comments, and I'm glad that it seems like
> we're
> > > > > > > > aligning on the vision here.
> > > > > > > >
> > > > > > > > > An
> > > > > > > > > alternative could be to change existing behavior to fail
> fast
> > > on
> > > > > any
> > > > > > > > > invalid default converter configuration instead of just for
> > > > invalid
> > > > > > > > > versions
> > > > > > > >
> > > > > > > > I suppose if this is landing in 4.0, we have the opportunity
> to
> > > > break
> > > > > > > > compatibility and strictly validate the worker class configs,
> > and
> > > > > > > > could have a new consistent behavior instead of inconsistent
> > but
> > > > > > > > backwards-compatible behavior.
> > > > > > > >
> > > > > > > > > no other part of the KIP requires this change.
> > > > > > > >
> > > > > > > > This is correct, and a compelling argument. I'm fine leaving
> > the
> > > > > > > > strict worker validation off of this configuration to
> > potentially
> > > > be
> > > > > > > > added later. If a KIP was raised to perform strict validation
> > of
> > > > the
> > > > > > > > worker config, it would include the version config, and can
> > > address
> > > > > > > > backwards compatibility for both configs together.
> > > > > > > >
> > > > > > > > > RE exposing the version property in the
> > > > > > > > /connector-plugins/<plugin>/config
> > > > > > > > > endpoint, the behavior is inconsistent across plugin types.
> > > > > > > >
> > > > > > > > Yes it is, and that started in a bugfix, once people started
> > > using
> > > > > > > > this endpoint:
> > https://issues.apache.org/jira/browse/KAFKA-14843
> > > > . I
> > > > > > > > wasn't planning on converging the behavior in this KIP, as I
> > > > > > > > considered it out-of-scope, and was going to follow the
> current
> > > > > > > > behavior. To summarize:
> > > > > > > >
> > > > > > > > GET /connector-plugins/<connector>/config will emit:
> > > > > > > > * connector.class (already implemented)
> > > > > > > > * connector.version (new)
> > > > > > > > * key.converter (already implemented)
> > > > > > > > * key.converter.version (new)
> > > > > > > > * value.converter (already implemented)
> > > > > > > > * value.converter.version (new)
> > > > > > > > * header.converter (already implemented)
> > > > > > > > * header.converter.version (new)
> > > > > > > > But will NOT emit:
> > > > > > > > * transforms.<alias>.type
> > > > > > > > * transforms.<alias>.version
> > > > > > > > * transforms.<alias>.predicate
> > > > > > > > * predicates.<alias>.type
> > > > > > > > * predicates.<alias>.version
> > > > > > > > * predicates.<alias>.negate
> > > > > > > >
> > > > > > > > GET /connector-plugins/<converter>/config will NOT emit:
> > > > > > > > * "" (there's not even a ".class" prefix!)
> > > > > > > > * version
> > > > > > > >
> > > > > > > > GET /connector-plugins/<transform>/config will NOT emit:
> > > > > > > > * type
> > > > > > > > * version
> > > > > > > > * predicate
> > > > > > > >
> > > > > > > > GET /connector-plugins/<predicate>/config will NOT emit:
> > > > > > > > * type
> > > > > > > > * version
> > > > > > > > * negate
> > > > > > > >
> > > > > > > > Do you want the converter, transform, and predicate endpoints
> > > > > changed?
> > > > > > > > Do you want just "version", or do you want all of the
> prefixed
> > > > > configs
> > > > > > > > including "type", "predicate" and "negate"? How would you
> want
> > to
> > > > > > > > handle the converters?
> > > > > > > > And when I say the configs are "not part of the plugin config
> > > > itself"
> > > > > > > > I mean that saying that GET
> > > > > > > > /connector-plugins/Flatten$Key/config?version=3.8.0 has a
> > > "version"
> > > > > > > > config that must be "3.8.0" is a little bit nonsense, as the
> > > > version
> > > > > > > > is already specified.
> > > > > > > >
> > > > > > > > > IMO
> > > > > > > > > it's worth including this information somewhere directly
> > > > accessible
> > > > > > > > without
> > > > > > > > > having to provide a full connector config. FWIW I'd be fine
> > > with
> > > > > GET
> > > > > > > > > /connector-plugins/<plugin>/versions as a first-class
> > endpoint
> > > > > > > >
> > > > > > > > You don't have to provide a configuration to call GET
> > > > > > > > /connector-plugins?connectorsOnly=false , is that endpoint
> not
> > > > close
> > > > > > > > enough to what you have in mind? See also the Rejected
> > > Alternative
> > > > > > > > "Adding new REST API endpoints"
> > > > > > > >
> > > > > > > > If you're calling /connector-plugins/<transform>/config, you
> > know
> > > > the
> > > > > > > > name of a plugin right? That either comes from out-of-band
> > > > knowledge,
> > > > > > > > validating a connector config, or calling GET
> > > > > > > > /connector-plugins?connectorsOnly=false.
> > > > > > > > * If you have out-of-band knowledge of plugin classes,
> perhaps
> > > you
> > > > > > > > have out-of-band knowledge of versions too.
> > > > > > > > * If you've just validated a connector config, there should
> be
> > an
> > > > > > > > accompanying "version" field there with an accurate default
> > value
> > > > and
> > > > > > > > recommenders.
> > > > > > > > * If you've called GET
> /connector-plugins?connectorsOnly=false,
> > > > that
> > > > > > > > endpoint includes version information.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Greg
> > > > > > > >
> > > > > > > > On Wed, May 22, 2024 at 11:05 AM Chris Egerton
> > > > > <chrise@aiven.io.invalid
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Greg,
> > > > > > > > >
> > > > > > > > > Hope you had a nice weekend! Gonna try to keep things
> > concise.
> > > > > > > > >
> > > > > > > > > Concluded points:
> > > > > > > > >
> > > > > > > > > RE version recommenders, I agree it's likely that
> > programmatic
> > > > UIs
> > > > > > will
> > > > > > > > > already be able to handle dynamic configuration
> definitions,
> > > and
> > > > > the
> > > > > > > > detail
> > > > > > > > > about SMTs is a great point. I still anticipate some
> > > awkwardness
> > > > > with
> > > > > > > > > connector versions, though: if the latest version supports
> > some
> > > > new
> > > > > > > > > properties, then a user switches to an earlier version, a
> UI
> > > may
> > > > > > > respond
> > > > > > > > by
> > > > > > > > > wiping values for these properties. I guess we can bite the
> > > > bullet,
> > > > > > > > though.
> > > > > > > > >
> > > > > > > > > RE double-dinging during preflight validation for invalid
> > > > > versions, I
> > > > > > > > like
> > > > > > > > > the analogy with login credentials. I'm convinced that the
> > > > proposal
> > > > > > in
> > > > > > > > the
> > > > > > > > > KIP is best 👍
> > > > > > > > >
> > > > > > > > > Continued points:
> > > > > > > > >
> > > > > > > > > RE failing on worker startup, sorry, I should be clearer:
> > there
> > > > is
> > > > > no
> > > > > > > > _new_
> > > > > > > > > justification for it that doesn't also apply to existing
> > > > behavior.
> > > > > We
> > > > > > > > > shouldn't diverge from existing behavior solely for this
> new
> > > > case.
> > > > > An
> > > > > > > > > alternative could be to change existing behavior to fail
> fast
> > > on
> > > > > any
> > > > > > > > > invalid default converter configuration instead of just for
> > > > invalid
> > > > > > > > > versions, but I'd vote to just stick to existing behavior
> and
> > > not
> > > > > > > > > complicate things, especially since no other part of the
> KIP
> > > > > requires
> > > > > > > > this
> > > > > > > > > change.
> > > > > > > > >
> > > > > > > > > RE exposing the version property in the
> > > > > > > > /connector-plugins/<plugin>/config
> > > > > > > > > endpoint, the behavior is inconsistent across plugin types.
> > > > Hitting
> > > > > > the
> > > > > > > > > endpoint for the FileStreamSinkConnector on version 3.7.0
> > > yields
> > > > a
> > > > > > > > response
> > > > > > > > > that includes, among other things, the "topics",
> > > "topics.regex",
> > > > > and
> > > > > > > > > "errors.tolerance" properties. I see that we don't do this
> > > > > everywhere
> > > > > > > > (the
> > > > > > > > > examples you cite for SMT and converter properties are
> > > accurate),
> > > > > but
> > > > > > > IMO
> > > > > > > > > it's worth including this information somewhere directly
> > > > accessible
> > > > > > > > without
> > > > > > > > > having to provide a full connector config. FWIW I'd be fine
> > > with
> > > > > GET
> > > > > > > > > /connector-plugins/<plugin>/versions as a first-class
> > endpoint
> > > > > either
> > > > > > > > > instead of or in addition to adding recommended values for
> > all
> > > > > plugin
> > > > > > > > > versions.
> > > > > > > > >
> > > > > > > > > Thanks for your continued work on this KIP, and with the
> > > progress
> > > > > > we're
> > > > > > > > > making I'm optimistic about its chances of appearing in
> > 4.0.0.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Wed, May 15, 2024 at 1:22 PM Greg Harris
> > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hey Chris,
> > > > > > > > > >
> > > > > > > > > > Thanks for your quick follow up.
> > > > > > > > > >
> > > > > > > > > > > But this risk is already present with
> > > > > > > > > > > existing error cases, and I don't see anything that
> > > justifies
> > > > > > > > changing
> > > > > > > > > > > existing behavior with an invalid converter class, or
> > > > diverging
> > > > > > > from
> > > > > > > > it
> > > > > > > > > > in
> > > > > > > > > > > the case of invalid converter versions.
> > > > > > > > > >
> > > > > > > > > > The justification is to fail-fast, and prevent REST API
> > users
> > > > > from
> > > > > > > > > > receiving errors from bad configs that they didn't write,
> > or
> > > > > maybe
> > > > > > > > > > don't even know apply to them.
> > > > > > > > > > Up until recently errors in these configurations surfaced
> > as
> > > > > > failures
> > > > > > > > > > to create the connector, or failures to start, and you
> made
> > > > them
> > > > > > > > > > fail-fast during validation. I think this change is in
> the
> > > same
> > > > > > > > > > spirit, pulling the validation further forward and not
> > > letting
> > > > > > errors
> > > > > > > > > > lie dormant.
> > > > > > > > > >
> > > > > > > > > > And to call back to your original concern about
> > interrupting
> > > > > > > > > > connectors that explicitly provide these configurations
> and
> > > > don't
> > > > > > use
> > > > > > > > > > the worker configs: I expect that operators with a
> majority
> > > of
> > > > > > these
> > > > > > > > > > sorts of clients aren't going to be setting the worker
> > > .version
> > > > > > > > > > properties, because it would have no effect on the
> majority
> > > of
> > > > > > their
> > > > > > > > > > connectors. They would be able to rely on
> > > > backwards-compatibility
> > > > > > and
> > > > > > > > > > continue to ignore the class properties.
> > > > > > > > > >
> > > > > > > > > > > CLI
> > > > > > > > > > > and programmatic UI developers will want to develop
> their
> > > own
> > > > > > > tooling
> > > > > > > > > > > layers.
> > > > > > > > > >
> > > > > > > > > > This is a very compelling argument. Snehashis do you want
> > to
> > > > > figure
> > > > > > > > > > out a REST API design for this use-case?
> > > > > > > > > >
> > > > > > > > > > > Regarding the GET /connector-plugins/<plugin>/config
> > > > endpoint,
> > > > > I
> > > > > > > was
> > > > > > > > > > > thinking about the response for non-connector plugins,
> > > e.g.,
> > > > > > > > > > > GET /connector-plugins/RegexRouter/config. Would a
> > > "version"
> > > > > > > property
> > > > > > > > > > > appear with recommended values?
> > > > > > > > > >
> > > > > > > > > > I intended for the ".version" property to be like other
> > > > framework
> > > > > > > > > > configs (".class", ".type", ".predicate", ".negate")
> where
> > > they
> > > > > are
> > > > > > > > > > inside the plugin namespace, but not part of the plugin
> > > config
> > > > > > > itself.
> > > > > > > > > > Perhaps we can deviate from those configs because it
> would
> > > aid
> > > > in
> > > > > > > > > > discovering other valid `GET
> > > > > > > > > > /connector-plugins/<plugin>/config?version=` calls,
> without
> > > > > calling
> > > > > > > > > > `GET /connector-plugins?connectorsOnly=false`.
> > > > > > > > > > I don't really feel strongly either way.
> > > > > > > > > >
> > > > > > > > > > > if a user changes the
> > > > > > > > > > > connector version in, e.g., a dropdown menu, then the
> UI
> > > > either
> > > > > > has
> > > > > > > > to
> > > > > > > > > > > re-fetch the ConfigDef for the new version, or risk
> > > operating
> > > > > on
> > > > > > > > stale
> > > > > > > > > > > information
> > > > > > > > > >
> > > > > > > > > > This is a really interesting situation, thanks for
> finding
> > > > that!
> > > > > > This
> > > > > > > > > > is already a footgun with transformations and predicates;
> > > Once
> > > > > you
> > > > > > > > > > fill out a transformation class (which includes a
> > > recommender)
> > > > > you
> > > > > > > > > > will then see all of the transformations configurations.
> > > > > > > > > > I think this means that UI developers should have already
> > > > > developed
> > > > > > > > > > the infrastructure for handling dynamic recommenders, or
> > else
> > > > > > they've
> > > > > > > > > > had this bug since KIP-66 in 2017. It may require some
> > manual
> > > > > > > > > > attention to roll-out support for the ".version"
> properties
> > > > > > > > > > specifically, but I don't think that should prevent us
> from
> > > > > > providing
> > > > > > > > > > this information in a natural place.
> > > > > > > > > >
> > > > > > > > > > > but will it be possible to configure
> > > > > > > > > > > a connector to use two instances of the same transform
> or
> > > > > > > predicate,
> > > > > > > > but
> > > > > > > > > > > with different versions for each?
> > > > > > > > > >
> > > > > > > > > > Yes, and this is included in the example in the KIP,
> > > > > > > > > > `transforms.flatten-latest` and `transforms.flatten-old`.
> > > This
> > > > > will
> > > > > > > > > > work in the natural way, with the two instances having a
> > > > > different
> > > > > > > > > > version if configured for that. If a user wants to pin
> the
> > > same
> > > > > > > > > > version of a plugin for every instance, they will need to
> > > > provide
> > > > > > the
> > > > > > > > > > version config multiple times and keep them in-sync
> > > themselves.
> > > > > > > > > >
> > > > > > > > > > > I would have expected the error
> > > > > > > > > > > to only be attributed to the version property, and for
> > the
> > > > > class
> > > > > > > > property
> > > > > > > > > > > to be reported as valid.
> > > > > > > > > >
> > > > > > > > > > I considered this, and then went back and changed it. To
> > > quote
> > > > > > > someone
> > > > > > > > > > else, this is to catch "misspelling cat as dog". For
> > example
> > > a
> > > > > user
> > > > > > > > > > types DogConverter and meant to type CatConverter, and
> then
> > > > > types a
> > > > > > > > > > version which is valid for CatConverter, conceptually the
> > > error
> > > > > is
> > > > > > in
> > > > > > > > > > the class name and not the version.
> > > > > > > > > > That's a very contrived scenario, but I think similar
> > > arguments
> > > > > are
> > > > > > > > > > used for attributing validation errors to
> > > > > endpoints/urls/hostnames
> > > > > > > > > > when the associated credentials are unable to log into
> the
> > > > remote
> > > > > > > > > > system. Did the user provide the correct testing
> > credentials,
> > > > but
> > > > > > > > > > accidentally typed the production endpoint? Even if the
> > > > > production
> > > > > > > > > > endpoint looks valid (it's a real hostname that is
> > reachable)
> > > > the
> > > > > > > > > > conceptual error is still in the hostname and should have
> > the
> > > > > error
> > > > > > > > > > attributed to it to draw the user's attention.
> > > > > > > > > > If that's not convincing, I think the alternative of only
> > > > > > attributing
> > > > > > > > > > version errors to the version property is also
> acceptable.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Greg
> > > > > > > > > >
> > > > > > > > > > On Wed, May 15, 2024 at 8:06 AM Chris Egerton
> > > > > > > <chrise@aiven.io.invalid
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Greg,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your responses! Continuations of existing
> > > > > discussions:
> > > > > > > > > > >
> > > > > > > > > > > Regarding crashing the worker on startup--yes, there is
> > > also
> > > > a
> > > > > > risk
> > > > > > > > to
> > > > > > > > > > > allowing it to join the cluster. But this risk is
> already
> > > > > present
> > > > > > > > with
> > > > > > > > > > > existing error cases, and I don't see anything that
> > > justifies
> > > > > > > > changing
> > > > > > > > > > > existing behavior with an invalid converter class, or
> > > > diverging
> > > > > > > from
> > > > > > > > it
> > > > > > > > > > in
> > > > > > > > > > > the case of invalid converter versions. I think we
> should
> > > > keep
> > > > > > this
> > > > > > > > > > simple
> > > > > > > > > > > and not do anything different for worker startup than
> we
> > do
> > > > > > > already.
> > > > > > > > > > >
> > > > > > > > > > > As far as REST API vs. metrics go--I think you're right
> > > that
> > > > > the
> > > > > > > > original
> > > > > > > > > > > version metrics were added as a "monitoring" detail.
> > > However,
> > > > > > this
> > > > > > > > was
> > > > > > > > > > back
> > > > > > > > > > > when plugin versions were managed solely by cluster
> > > > > > administrators.
> > > > > > > > With
> > > > > > > > > > > this KIP, connector users will be able to manage plugin
> > > > > versions,
> > > > > > > > and CLI
> > > > > > > > > > > and programmatic UI developers will want to develop
> their
> > > own
> > > > > > > tooling
> > > > > > > > > > > layers. I think focusing on the REST API as the primary
> > > > > interface
> > > > > > > for
> > > > > > > > > > this
> > > > > > > > > > > KIP would be best for these users.
> > > > > > > > > > >
> > > > > > > > > > > (All that said, I don't object to the metrics that are
> > > > proposed
> > > > > > in
> > > > > > > > the
> > > > > > > > > > KIP;
> > > > > > > > > > > I just think they make more sense in addition to new
> REST
> > > API
> > > > > > > > > > > functionality, as opposed to instead of it.)
> > > > > > > > > > >
> > > > > > > > > > > Regarding the GET /connector-plugins/<plugin>/config
> > > > endpoint,
> > > > > I
> > > > > > > was
> > > > > > > > > > > thinking about the response for non-connector plugins,
> > > e.g.,
> > > > > > > > > > > GET /connector-plugins/RegexRouter/config. Would a
> > > "version"
> > > > > > > property
> > > > > > > > > > > appear with recommended values?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > And new thoughts:
> > > > > > > > > > >
> > > > > > > > > > > 1) Regarding the recommended values for
> > > "connector.version",
> > > > > this
> > > > > > > > might
> > > > > > > > > > be
> > > > > > > > > > > confusing since there could be differences between the
> > > > > ConfigDefs
> > > > > > > > for the
> > > > > > > > > > > latest version of the connector and prior versions. It
> > also
> > > > > makes
> > > > > > > the
> > > > > > > > > > flow
> > > > > > > > > > > a bit awkward for programmatic UI developers: if a user
> > > > changes
> > > > > > the
> > > > > > > > > > > connector version in, e.g., a dropdown menu, then the
> UI
> > > > either
> > > > > > has
> > > > > > > > to
> > > > > > > > > > > re-fetch the ConfigDef for the new version, or risk
> > > operating
> > > > > on
> > > > > > > > stale
> > > > > > > > > > > information. I'm starting to doubt that exposing the
> > range
> > > of
> > > > > > > > available
> > > > > > > > > > > versions via recommended values is the best way to
> > proceed,
> > > > > > instead
> > > > > > > > of a
> > > > > > > > > > > more explicit approach like GET
> > > > > > > > /connector-plugins/<plugin>/versions, or
> > > > > > > > > > > the "Adding new REST API endpoints" rejected
> alternative.
> > > > > > > > > > >
> > > > > > > > > > > 2) I know that this is a bit heinous, but will it be
> > > possible
> > > > > to
> > > > > > > > > > configure
> > > > > > > > > > > a connector to use two instances of the same transform
> or
> > > > > > > predicate,
> > > > > > > > but
> > > > > > > > > > > with different versions for each? (I don't think this
> is
> > > > worth
> > > > > > > > > > significant
> > > > > > > > > > > design/implementation effort, so if it would inflate
> > either
> > > > of
> > > > > > > those,
> > > > > > > > > > > please don't feel obligated to support it.)
> > > > > > > > > > >
> > > > > > > > > > > 3) Just out of curiosity, why double-ding during config
> > > > > > validation
> > > > > > > > if the
> > > > > > > > > > > version for a plugin class can't be found? I would have
> > > > > expected
> > > > > > > the
> > > > > > > > > > error
> > > > > > > > > > > to only be attributed to the version property, and for
> > the
> > > > > class
> > > > > > > > property
> > > > > > > > > > > to be reported as valid.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris
> > > > > > > > > > >
> > > > > > > > > > > On Tue, May 14, 2024 at 1:11 PM Greg Harris
> > > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hey Chris,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your comments. I'm glad you like the
> > > > motivations,
> > > > > > > > Snehashis
> > > > > > > > > > > > wrote that part!
> > > > > > > > > > > >
> > > > > > > > > > > > > the configuration syntax for the most basic use
> case
> > of
> > > > > > > > > > > > > specifying a single desired version is pretty
> > > > > > counterintuitive.
> > > > > > > > > > > >
> > > > > > > > > > > > I agree, and the "soft" requirement scheme is
> > something I
> > > > > > wasn't
> > > > > > > > > > > > explicitly looking for, but would be inherited from
> the
> > > > > > library.
> > > > > > > > I'm
> > > > > > > > > > > > fine with eliminating the soft requirement semantics,
> > and
> > > > > > having
> > > > > > > > > > > > "1.0.0" behave the same as "[1.0.0]". I'm less
> inclined
> > > to
> > > > > > > include
> > > > > > > > > > > > "range" in the property name, or have two properties.
> > > > > > > > > > > >
> > > > > > > > > > > > > Failing startup is drastic and has the potential to
> > > > disrupt
> > > > > > the
> > > > > > > > > > > > > availability of connectors that would otherwise be
> > able
> > > > to
> > > > > > run
> > > > > > > > > > healthily
> > > > > > > > > > > > > because they were explicitly configured to use
> valid
> > > > > > converters
> > > > > > > > > > instead
> > > > > > > > > > > > of
> > > > > > > > > > > > > the worker defaults.
> > > > > > > > > > > >
> > > > > > > > > > > > I think this argument cuts both ways. If someone
> > > > > reconfigures a
> > > > > > > > worker
> > > > > > > > > > > > and adds an invalid ".version" string to the worker
> (or
> > > > > changes
> > > > > > > the
> > > > > > > > > > > > plugin.path to make it invalid), it would be
> permitted
> > to
> > > > > enter
> > > > > > > the
> > > > > > > > > > > > group, and accept work assignments. If those work
> > > > assignments
> > > > > > > used
> > > > > > > > > > > > these configurations, a set of tasks could transition
> > to
> > > > > FAILED
> > > > > > > and
> > > > > > > > > > > > not be able to recover, because they would be
> restarted
> > > > again
> > > > > > on
> > > > > > > > the
> > > > > > > > > > > > same worker.
> > > > > > > > > > > >
> > > > > > > > > > > > > Why are metrics utilized to report information
> about
> > > > plugin
> > > > > > > > versions
> > > > > > > > > > > > > utilized by connectors at runtime instead of
> > publishing
> > > > > this
> > > > > > > > info in
> > > > > > > > > > the
> > > > > > > > > > > > > REST API
> > > > > > > > > > > >
> > > > > > > > > > > > I was following the existing pattern for exposing
> > runtime
> > > > > > > versions
> > > > > > > > for
> > > > > > > > > > > > connectors, and it did seem like a "monitoring"
> > feature.
> > > If
> > > > > > that
> > > > > > > > > > > > approach is flawed and should be phased out, I think
> it
> > > > would
> > > > > > be
> > > > > > > a
> > > > > > > > > > > > good idea to reconsider the REST API rejected
> > > alternative.
> > > > > > > > > > > > We would need some additional design work to spec out
> > the
> > > > > REST
> > > > > > > API
> > > > > > > > > > > > interface, as I don't have anything in mind
> currently.
> > > > > > > > > > > >
> > > > > > > > > > > > > I'm unclear on whether or not it'll be possible to
> > see
> > > > this
> > > > > > > > > > information
> > > > > > > > > > > > via the
> > > > > > > > > > > > > GET /connector-plugins/<plugin>/config endpoint
> > > > > > > > > > > >
> > > > > > > > > > > > There is room in the API to add recommenders for
> > > > > > "key.converter",
> > > > > > > > > > > > "value.converter", and "header.converter", but not
> for
> > > > > > transforms
> > > > > > > > and
> > > > > > > > > > > > predicates, as they include aliases that depend on an
> > > > actual
> > > > > > > > > > > > configuration. We could explicitly say we're going to
> > do
> > > > > that,
> > > > > > or
> > > > > > > > do
> > > > > > > > > > > > whatever is convenient during the implementation
> phase,
> > > or
> > > > > > leave
> > > > > > > it
> > > > > > > > > > > > open to be improved later.
> > > > > > > > > > > > There will not be any recommenders for ".version"
> > > > properties
> > > > > in
> > > > > > > the
> > > > > > > > > > > > `/config` endpoint, because those recommenders are
> > > dynamic
> > > > > and
> > > > > > > > depend
> > > > > > > > > > > > on an actual configuration.
> > > > > > > > > > > >
> > > > > > > > > > > > 5) There are two relevant lines in the KIP: "If a
> > > .version
> > > > > > > property
> > > > > > > > > > > > contains a hard requirement, select the latest
> > installed
> > > > > > version
> > > > > > > > which
> > > > > > > > > > > > satisfies the requirement." and "This configuration
> is
> > > > > > > re-evaluated
> > > > > > > > > > > > each time the connector or task are assigned to a new
> > > > > worker".
> > > > > > I
> > > > > > > > would
> > > > > > > > > > > > call this "eager" upgrade behavior, rather than a
> > > "sticky"
> > > > or
> > > > > > > > "lazy"
> > > > > > > > > > > > upgrade behavior.
> > > > > > > > > > > >
> > > > > > > > > > > > 6) Updated!
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Greg
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, May 14, 2024 at 9:14 AM Chris Egerton
> > > > > > > > <chrise@aiven.io.invalid
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks Greg for updating the KIP, and thanks
> > Snehashis
> > > > for
> > > > > > > > starting
> > > > > > > > > > the
> > > > > > > > > > > > > work on this originally.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The motivation section makes a pretty convincing
> case
> > > for
> > > > > > this
> > > > > > > > kind
> > > > > > > > > > of
> > > > > > > > > > > > > feature. My thoughts are mostly about specific
> > details:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1) I like the support for version ranges (the
> example
> > > > > > > > demonstrating
> > > > > > > > > > how
> > > > > > > > > > > > to
> > > > > > > > > > > > > avoid KAFKA-10574 with the header converter was
> > > > > particularly
> > > > > > > > > > > > entertaining),
> > > > > > > > > > > > > but the configuration syntax for the most basic use
> > > case
> > > > of
> > > > > > > > > > specifying a
> > > > > > > > > > > > > single desired version is pretty counterintuitive.
> > > People
> > > > > may
> > > > > > > get
> > > > > > > > > > bitten
> > > > > > > > > > > > or
> > > > > > > > > > > > > at least frustrated if they put
> > connector.version=3.8.0
> > > > in
> > > > > a
> > > > > > > > config
> > > > > > > > > > but
> > > > > > > > > > > > > then version 3.7.5 ends up running. I'd like it if
> we
> > > > could
> > > > > > > > either
> > > > > > > > > > > > > intentionally deviate from Maven ranges when a bare
> > > > version
> > > > > > is
> > > > > > > > > > present,
> > > > > > > > > > > > or
> > > > > > > > > > > > > separate things out into two properties:
> foo.version
> > > > would
> > > > > be
> > > > > > > the
> > > > > > > > > > single
> > > > > > > > > > > > > accepted version for the foo plugin, and
> > > > foo.version.range
> > > > > > > would
> > > > > > > > use
> > > > > > > > > > > > Maven
> > > > > > > > > > > > > range syntax. Open to other options too, just
> > > providing a
> > > > > > > couple
> > > > > > > > to
> > > > > > > > > > get
> > > > > > > > > > > > the
> > > > > > > > > > > > > ball rolling.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2) Although the current behavior for a worker with
> an
> > > > > invalid
> > > > > > > > > > > > > key/value/header converter class specified in its
> > > config
> > > > > file
> > > > > > > is
> > > > > > > > a
> > > > > > > > > > little
> > > > > > > > > > > > > strange (I was surprised to learn that it wouldn't
> > fail
> > > > on
> > > > > > > > startup),
> > > > > > > > > > I
> > > > > > > > > > > > > don't see a good reason to deviate from this when
> an
> > > > > invalid
> > > > > > > > version
> > > > > > > > > > is
> > > > > > > > > > > > > specified. Failing startup is drastic and has the
> > > > potential
> > > > > > to
> > > > > > > > > > disrupt
> > > > > > > > > > > > the
> > > > > > > > > > > > > availability of connectors that would otherwise be
> > able
> > > > to
> > > > > > run
> > > > > > > > > > healthily
> > > > > > > > > > > > > because they were explicitly configured to use
> valid
> > > > > > converters
> > > > > > > > > > instead
> > > > > > > > > > > > of
> > > > > > > > > > > > > the worker defaults.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3) Why are metrics utilized to report information
> > about
> > > > > > plugin
> > > > > > > > > > versions
> > > > > > > > > > > > > utilized by connectors at runtime instead of
> > publishing
> > > > > this
> > > > > > > > info in
> > > > > > > > > > the
> > > > > > > > > > > > > REST API? I saw that this was mentioned as a
> rejected
> > > > > > > > alternative,
> > > > > > > > > > but I
> > > > > > > > > > > > > didn't get a sense of why. It seems like the REST
> API
> > > > would
> > > > > > be
> > > > > > > > > > easier to
> > > > > > > > > > > > > access and more intuitive for most users than new
> > > > metrics.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 4) In the "Validation" section it's stated that
> > "Users
> > > > can
> > > > > > use
> > > > > > > > these
> > > > > > > > > > > > > recommenders to discover the valid plugin classes
> and
> > > > > > versions,
> > > > > > > > > > without
> > > > > > > > > > > > > requiring an earlier call to GET
> > > > > > > > > > > > /connector-plugins?connectorsOnly=false."
> > > > > > > > > > > > > I really like the creativity and simplicity of
> > reusing
> > > > the
> > > > > > > > > > recommender
> > > > > > > > > > > > > mechanism to expose available versions for plugins
> > via
> > > > the
> > > > > > REST
> > > > > > > > API.
> > > > > > > > > > I'm
> > > > > > > > > > > > > unclear on whether or not it'll be possible to see
> > this
> > > > > > > > information
> > > > > > > > > > via
> > > > > > > > > > > > the
> > > > > > > > > > > > > GET /connector-plugins/<plugin>/config endpoint,
> > > though.
> > > > > It'd
> > > > > > > be
> > > > > > > > > > great if
> > > > > > > > > > > > > this were supported, since we learned in KIP-769
> [1]
> > > that
> > > > > > > people
> > > > > > > > > > really
> > > > > > > > > > > > > want to be able to see configuration options for
> > > > connectors
> > > > > > and
> > > > > > > > their
> > > > > > > > > > > > > plugins via some kind of GET endpoint without
> having
> > to
> > > > > > > provide a
> > > > > > > > > > > > complete
> > > > > > > > > > > > > connector config for validation.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 5) In the Maven version range docs, it's stated
> that
> > > > "Maven
> > > > > > > > picks the
> > > > > > > > > > > > > highest version of each project that satisfies all
> > the
> > > > hard
> > > > > > > > > > requirements
> > > > > > > > > > > > of
> > > > > > > > > > > > > the dependencies on that project." I'm guessing
> this
> > > > > behavior
> > > > > > > > will be
> > > > > > > > > > > > > retained for Connect; i.e., the highest-possible
> > > version
> > > > of
> > > > > > > each
> > > > > > > > > > plugin
> > > > > > > > > > > > > that satisfies the user-specified version
> constraints
> > > > will
> > > > > be
> > > > > > > > run?
> > > > > > > > > > (An
> > > > > > > > > > > > > alternative approach could be to have some kind of
> > > > "sticky"
> > > > > > > logic
> > > > > > > > > > that
> > > > > > > > > > > > only
> > > > > > > > > > > > > restarts connectors/tasks when their currently-used
> > > > version
> > > > > > > > becomes
> > > > > > > > > > > > > incompatible with the configured constraints.)
> > > > > > > > > > > > >
> > > > > > > > > > > > > 6) (Nit) It'd be nice to add a link to the
> > TestPlugins
> > > > > class
> > > > > > or
> > > > > > > > > > somewhere
> > > > > > > > > > > > > in its neighborhood to the testing plan; unfamiliar
> > > > readers
> > > > > > > > probably
> > > > > > > > > > > > won't
> > > > > > > > > > > > > get much out of what's there right now.
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1] -
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Chris
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, May 13, 2024 at 2:01 PM Snehashis <
> > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > That is much appreciated. No complaints on the
> > > > additional
> > > > > > > > scope, I
> > > > > > > > > > will
> > > > > > > > > > > > > > make some time out to work on this once we have
> > > > approval.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> > > > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hey Snehashis,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm glad to hear you're still interested in
> this
> > > KIP!
> > > > > > > > > > > > > > > I'm happy to let you drive this, and I
> apologize
> > > for
> > > > > > > > increasing
> > > > > > > > > > the
> > > > > > > > > > > > > > > scope of work so drastically. To make up for
> > that,
> > > > I'll
> > > > > > > > > > volunteer to
> > > > > > > > > > > > > > > be the primary PR reviewer to help get this
> done
> > > > > quickly
> > > > > > > once
> > > > > > > > > > the KIP
> > > > > > > > > > > > > > > is approved.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, May 10, 2024 at 3:51 AM Snehashis <
> > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for the follow up to my original KIP,
> I
> > am
> > > > in
> > > > > > > > favour of
> > > > > > > > > > the
> > > > > > > > > > > > > > > > additions made to expand its scope, the
> > addition
> > > of
> > > > > > range
> > > > > > > > > > versions
> > > > > > > > > > > > > > > > specifically make a lot of sense.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Apologies if I have not publicly worked on
> this
> > > KIP
> > > > > > for a
> > > > > > > > long
> > > > > > > > > > > > time.
> > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > original work was done when the move to
> service
> > > > > loading
> > > > > > > > was in
> > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > and I wanted to loop back to this only after
> > that
> > > > > work
> > > > > > > was
> > > > > > > > > > > > completed.
> > > > > > > > > > > > > > > Post
> > > > > > > > > > > > > > > > its conclusion, I have not been able to take
> > this
> > > > up
> > > > > > due
> > > > > > > to
> > > > > > > > > > other
> > > > > > > > > > > > > > > > priorities. If it's okay with you, I would
> > still
> > > > like
> > > > > > to
> > > > > > > > get
> > > > > > > > > > this
> > > > > > > > > > > > > > > > implemented myself, including the additional
> > > scope.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks and regards
> > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> > > > > > > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'd like to reboot the discussion on
> KIP-891:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I've made some changes, most notably:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. Specifying versions for all plugins in
> > > > Connector
> > > > > > > > configs
> > > > > > > > > > > > > > > > > (converters, header converters, transforms,
> > and
> > > > > > > > predicates)
> > > > > > > > > > not
> > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > connectors & tasks
> > > > > > > > > > > > > > > > > 2. Specifying a range of versions instead
> of
> > an
> > > > > exact
> > > > > > > > match
> > > > > > > > > > > > > > > > > 3. New metrics to observe what versions are
> > > > in-use
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks to Snehashis for the original KIP
> > idea!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg
> Harris <
> > > > > > > > > > > > greg.harris@aiven.io>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Snehashis,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thank you for the KIP! This is something
> > I've
> > > > > > wanted
> > > > > > > > for a
> > > > > > > > > > long
> > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I know the discussion has gone cold, are
> > you
> > > > > still
> > > > > > > > > > interested
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > pursuing this feature? I'll make time to
> > > review
> > > > > the
> > > > > > > > KIP if
> > > > > > > > > > you
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > still accepting comments.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM
> Snehashis
> > <
> > > > > > > > > > > > > > snehashisp1995@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for the points Sagar.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1) Should we update the GET
> /connectors
> > > > > > endpoint
> > > > > > > to
> > > > > > > > > > > > include the
> > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > the plugin that is running? It could
> be
> > > > > useful
> > > > > > to
> > > > > > > > > > figure
> > > > > > > > > > > > out
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > the plugin or I am assuming it gets
> > > > returned
> > > > > by
> > > > > > > the
> > > > > > > > > > > > expand=info
> > > > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think this is good to have and
> possible
> > > > > future
> > > > > > > > > > > > enhancement. The
> > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > info will be present in the config of
> the
> > > > > > connector
> > > > > > > > if
> > > > > > > > > > the
> > > > > > > > > > > > user
> > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > specified the version. Otherwise it is
> > the
> > > > > latest
> > > > > > > > version
> > > > > > > > > > > > which
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > can find out from the connector-plugin
> > > > > endpoint.
> > > > > > > The
> > > > > > > > > > > > information
> > > > > > > > > > > > > > > can be
> > > > > > > > > > > > > > > > > > > introduced to the response of the GET
> > > > > /connectors
> > > > > > > > > > endpoint
> > > > > > > > > > > > > > itself,
> > > > > > > > > > > > > > > > > however
> > > > > > > > > > > > > > > > > > > the most ideal way of doing this would
> be
> > > to
> > > > > get
> > > > > > > the
> > > > > > > > > > > > currently
> > > > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > > instance of the connector and get the
> > > version
> > > > > > > > directly
> > > > > > > > > > from
> > > > > > > > > > > > > > there.
> > > > > > > > > > > > > > > > > This is
> > > > > > > > > > > > > > > > > > > slightly tricky as the connector could
> be
> > > > > running
> > > > > > > in
> > > > > > > > a
> > > > > > > > > > > > different
> > > > > > > > > > > > > > > node.
> > > > > > > > > > > > > > > > > > > One way to do this would be to persist
> > the
> > > > > > version
> > > > > > > > > > > > information in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > status backing store during
> instantiation
> > > of
> > > > > the
> > > > > > > > > > connector.
> > > > > > > > > > > > It
> > > > > > > > > > > > > > > requires
> > > > > > > > > > > > > > > > > > > some more thought and since the version
> > is
> > > > part
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > configs if
> > > > > > > > > > > > > > > > > provided
> > > > > > > > > > > > > > > > > > > and evident otherwise, I have not
> > included
> > > it
> > > > > in
> > > > > > > this
> > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2) I am not aware of this and hence
> > > asking,
> > > > > > can 2
> > > > > > > > > > > > connectors
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > versions have the same name? Does the
> > > > plugin
> > > > > > > > isolation
> > > > > > > > > > > > allow
> > > > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > could have a bearing when using the
> > > > lifecycle
> > > > > > > > > > endpoints for
> > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > All connectors in a cluster need to
> have
> > > > > uniquire
> > > > > > > > > > connector
> > > > > > > > > > > > names
> > > > > > > > > > > > > > > > > > > regardless of what version of the
> plugin
> > > the
> > > > > > > > connector is
> > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > > underneath. This is something enforced
> by
> > > the
> > > > > > > connect
> > > > > > > > > > runtime
> > > > > > > > > > > > > > > itself.
> > > > > > > > > > > > > > > > > All
> > > > > > > > > > > > > > > > > > > connect CRUD operations are keyed on
> the
> > > > > > connector
> > > > > > > > name
> > > > > > > > > > so
> > > > > > > > > > > > there
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > be an issue.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <
> > > > > > > > > > > > sagarmeansocean@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hey Snehashsih,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for the KIP. It looks like a
> > very
> > > > > useful
> > > > > > > > > > feature.
> > > > > > > > > > > > Couple
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > small-ish points, let me know what
> you
> > > > think:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1) Should we update the GET
> /connectors
> > > > > > endpoint
> > > > > > > to
> > > > > > > > > > > > include the
> > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > the plugin that is running? It could
> be
> > > > > useful
> > > > > > to
> > > > > > > > > > figure
> > > > > > > > > > > > out
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > the plugin or I am assuming it gets
> > > > returned
> > > > > by
> > > > > > > the
> > > > > > > > > > > > expand=info
> > > > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > > > > 2) I am not aware of this and hence
> > > asking,
> > > > > > can 2
> > > > > > > > > > > > connectors
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > versions have the same name? Does the
> > > > plugin
> > > > > > > > isolation
> > > > > > > > > > > > allow
> > > > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > could have a bearing when using the
> > > > lifecycle
> > > > > > > > > > endpoints for
> > > > > > > > > > > > > > > > > connectors like
> > > > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > > Sagar.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM
> Ashwin
> > > > > > > > > > > > > > > <apankaj@confluent.io.invalid
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > IIUC (please correct me if I am
> > wrong
> > > > > > here),
> > > > > > > > what
> > > > > > > > > > you
> > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > a versioning scheme for a connector
> > > > config
> > > > > > for
> > > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > different versions of a connector
> > > > plugin).
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Sorry for not being more precise in
> > my
> > > > > > wording
> > > > > > > > - I
> > > > > > > > > > meant
> > > > > > > > > > > > > > > > > registering
> > > > > > > > > > > > > > > > > > > > > versions of schema for connector
> > > config.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Let's take the example of a
> fictional
> > > > > > connector
> > > > > > > > which
> > > > > > > > > > > > uses a
> > > > > > > > > > > > > > > > > fictional
> > > > > > > > > > > > > > > > > > > > AWS
> > > > > > > > > > > > > > > > > > > > > service.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Fictional Connector Config schema
> > > > > version:2.0
> > > > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Fictional Connector config schema
> > > > > version:3.0
> > > > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > "iam_role": {
> > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > > > "iam_role"
> > > > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The connector which supports
> > Fictional
> > > > > config
> > > > > > > > schema
> > > > > > > > > > 2.0
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > validate
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > access key and secret key.
> > > > > > > > > > > > > > > > > > > > > Whereas a connector which supports
> > > config
> > > > > > with
> > > > > > > > schema
> > > > > > > > > > > > version
> > > > > > > > > > > > > > > 3.0
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > > > > validate the IAM role.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This is the alternative which I
> > wanted
> > > to
> > > > > > > > suggest.
> > > > > > > > > > Each
> > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > register the schema versions of
> > > connector
> > > > > > > config
> > > > > > > > > > which it
> > > > > > > > > > > > > > > supports.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The plugin paths may be optionally
> > > > > different
> > > > > > > > i.e we
> > > > > > > > > > > > don't
> > > > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > > > > mandatorily add a new plugin path
> to
> > > > > support
> > > > > > a
> > > > > > > > new
> > > > > > > > > > schema
> > > > > > > > > > > > > > > version.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:47 PM
> > > > Snehashis
> > > > > <
> > > > > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on the
> > > rejected
> > > > > > > > > > alternatives ?
> > > > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > config is versioned and has a
> > > schema.
> > > > > > Then
> > > > > > > a
> > > > > > > > > > single
> > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > > > dependencies have not changed)
> > can
> > > > > handle
> > > > > > > > > > multiple
> > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > IIUC (please correct me if I am
> > wrong
> > > > > > here),
> > > > > > > > what
> > > > > > > > > > you
> > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > a versioning scheme for a
> connector
> > > > > config
> > > > > > > for
> > > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > > > > > > > > connector (and
> > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > different versions of a connector
> > > > > plugin).
> > > > > > > > That is
> > > > > > > > > > a
> > > > > > > > > > > > > > somewhat
> > > > > > > > > > > > > > > > > > > > tangential
> > > > > > > > > > > > > > > > > > > > > > problem. While it is definitely a
> > > > useful
> > > > > > > > feature to
> > > > > > > > > > > > have,
> > > > > > > > > > > > > > > like a
> > > > > > > > > > > > > > > > > log to
> > > > > > > > > > > > > > > > > > > > > > check what changes were made over
> > > time
> > > > to
> > > > > > the
> > > > > > > > > > config
> > > > > > > > > > > > which
> > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > easier to do rollbacks, it is not
> > the
> > > > > focus
> > > > > > > > here.
> > > > > > > > > > Here
> > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > version we
> > > > > > > > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > > > > > > to say what underlying version of
> > the
> > > > > > plugin
> > > > > > > > > > should the
> > > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > of the connector use. Perhaps it
> is
> > > > > better
> > > > > > to
> > > > > > > > > > change
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > name of
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > parameter from connector.version
> to
> > > > > > > > > > > > > > connector.plugin.version
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > plugin.version if it was
> confusing.
> > > > wdyt?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2. Any plans to support
> assisted
> > > > > > migration
> > > > > > > > e.g
> > > > > > > > > > if a
> > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > > >
> connector/config?migrate=latest",
> > > the
> > > > > > > latest
> > > > > > > > > > version
> > > > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > > > transform the existing config
> to
> > > the
> > > > > > newer
> > > > > > > > > > version.
> > > > > > > > > > > > This
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > adding a method like "boolean
> > > > > > > migrate(Version
> > > > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This is an enhancement we can
> think
> > > of
> > > > > > doing
> > > > > > > in
> > > > > > > > > > future.
> > > > > > > > > > > > > > > Users can
> > > > > > > > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > > > > > a PUT call with the updated
> config
> > > > which
> > > > > > has
> > > > > > > > the
> > > > > > > > > > > > updated
> > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > number.
> > > > > > > > > > > > > > > > > > > > > > The assisted mode could be handy
> as
> > > the
> > > > > > user
> > > > > > > > does
> > > > > > > > > > not
> > > > > > > > > > > > need
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > know the
> > > > > > > > > > > > > > > > > > > > > > config but beyond this it does
> not
> > > seem
> > > > > to
> > > > > > > > justify
> > > > > > > > > > its
> > > > > > > > > > > > > > > existence.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 10:50 AM
> > > Ashwin
> > > > > > > > > > > > > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This is a really useful feature
> > and
> > > > > > thanks
> > > > > > > > for
> > > > > > > > > > > > initiating
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I had the following questions -
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on the
> > > rejected
> > > > > > > > > > alternatives ?
> > > > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > config is versioned and has a
> > > schema.
> > > > > > Then
> > > > > > > a
> > > > > > > > > > single
> > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > > > dependencies have not changed)
> > can
> > > > > handle
> > > > > > > > > > multiple
> > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2. Any plans to support
> assisted
> > > > > > migration
> > > > > > > > e.g
> > > > > > > > > > if a
> > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > > >
> connector/config?migrate=latest",
> > > the
> > > > > > > latest
> > > > > > > > > > version
> > > > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > > > transform the existing config
> to
> > > the
> > > > > > newer
> > > > > > > > > > version.
> > > > > > > > > > > > This
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > adding a method like "boolean
> > > > > > > migrate(Version
> > > > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM
> > > > > > Snehashis <
> > > > > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I'd like to start a
> discussion
> > > > thread
> > > > > > on
> > > > > > > > > > KIP-891:
> > > > > > > > > > > > > > Running
> > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The KIP aims to add the
> ability
> > > for
> > > > > the
> > > > > > > > connect
> > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > > > versions of a connector.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Please take a look and let me
> > > know
> > > > > what
> > > > > > > you
> > > > > > > > > > think.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Hi Snehashis,
Sorry for the late reply.
> Heterogeneous dependencies in a multi cluster deployment is highly
discouraged
Right, this remains unchanged in this KIP.
> Let's add the version information for both connector and tasks in the
connector status itself
> once we add these two additions to the KIP (LMK if you want me to take
that up).
Could you make these additions?
I'm interested to see if we can include this in 4.0.
Thanks,
Greg
On Tue, Jul 2, 2024 at 2:52 AM Snehashis <sn...@gmail.com> wrote:
> Hi Greg,
>
> Thanks for taking a look at this, to conclude on the two points above.
>
> 1. I'm okay with the status quo of leaving the dependency management of
> plugins to systems outside of the connect runtime as it is now. Given that
> the dependencies are homogenous across a connect cluster, it should ensure
> that task and connector versions are uniform across a deployment, after it
> has stabilised post an upgrade operation. Heterogeneous dependencies in a
> multi cluster deployment is highly discouraged and we should point out this
> can lead to inconsistent behaviour with connectors and even validations.
>
> 2. Let's add the version information for both connector and tasks in the
> connector status itself, I don't see another endpoint being required for
> this.
>
> I don't have any further points, so if you are okay we can put this to a
> vote, once we add these two additions to the KIP (LMK if you want me to
> take that up).
>
> Regards
> Snehashis
>
> On Tue, Jul 2, 2024 at 12:08 AM Greg Harris <gr...@aiven.io.invalid>
> wrote:
>
> > Hey Snehashis,
> >
> > Sorry for the late reply, and thanks for helping close out the
> discussion.
> >
> > > Note that if my assumptions are correct then
> > > this can happen with the existing framework as well, or is there some
> > > safeguard from this happening?
> >
> > Currently, if a cluster has a heterogeneous plugin installation, each
> > worker may have a different definition of "latest". If you call to
> > validate a configuration, it will validate against whatever latest
> version
> > is present locally.
> > With this KIP, that behavior will continue when the version isn't
> > specified, when the version is a soft requirement, or when the version
> is a
> > range. For users that do not want to tolerate their tasks having
> > heterogeneous versions, the single-version hard requirements are there,
> but
> > come at the cost of micromanagement of upgrades.
> > Perhaps there is some room in the future for a connector configured with
> a
> > range to "pin" the version that all of its tasks should use. Or
> > version-aware-scheduling can schedule connectors and tasks to workers
> that
> > will resolve the same versions. The important primitive is being able to
> > enforce which version is being used, everything else looks more like a
> > convenience feature to me.
> >
> > > So far, we could have pointed to the
> > > misconfigured cluster configuration and somewhat differ this problem to
> > > something outside of connect runtime.
> >
> > I think this will still be the case. Even with this feature, we will
> still
> > recommend homogeneous clusters that have the same set of plugins
> installed
> > on every worker. Version-aware scheduling is a rejected alternative, and
> > your example for performing validation requests on an arbitrary worker
> is a
> > great example of a complication for heterogeneous clusters that we're not
> > ready to solve.
> >
> > > We can introduce
> > > a new path under this for version (/connector/connector-name/version),
> > but
> > > perhaps adding this as part of the status is a valid alternative.
> >
> > I think either is fine. Adding a new field to the existing endpoints is
> > fine, and shouldn't require special compatibility such as a query
> > parameter.
> >
> > > Also, to go further I think
> > > version information for tasks could also be available,
> >
> > I completely agree. The version of each connector and each task should be
> > visible individually, since they may be different during a rolling
> upgrade.
> >
> > Thanks!
> > Greg
> >
> > On Tue, Jun 18, 2024 at 5:18 AM Snehashis <sn...@gmail.com>
> > wrote:
> >
> > > Hi Greg, Chris
> > >
> > > Thanks for the in-depth discussion, I have a couple of discussion
> points
> > > and would like your thoughts on this.
> > >
> > > 1) One concern I have with the new addition of 'soft' and 'hard'
> version
> > > requirements is that there could be a mismatch in the plugin version
> that
> > > two different tasks are running, if a soft requirement is provided and
> > the
> > > nodes a multi cluster deployment are not in sync w.r.t the plugin
> > versions
> > > that they are configured with. Note that if my assumptions are correct
> > then
> > > this can happen with the existing framework as well, or is there some
> > > safeguard from this happening? So far, we could have pointed to the
> > > misconfigured cluster configuration and somewhat differ this problem to
> > > something outside of connect runtime. With this feature in place
> perhaps
> > > the expectation is more on connect to not be running with such
> > > inconsistency, especially if a connector version is specified. This is
> > also
> > > a problem with validation if different cluster have different
> > > configurations, as IIRC validations are local to the worker which
> > receives
> > > the rest call for validate. So, we might be validating with a certain
> > > version which is different from the one that will be used to create
> > > connector and tasks. Again, this is likely how the current state is,
> but
> > > perhaps such inconsistencies warrant a deeper look with the addition of
> > > this feature. The problems associated with them can be somewhat
> insidious
> > > and hard to diagnose.
> > >
> > > 2) There was some discussion on the need for a new REST endpoint to
> > provide
> > > information on the versions of running connectors, and I think adding
> > this
> > > information via REST is a valuable addition. The way I see it the
> version
> > > is an intrinsic property of an instance of a running connector and
> hence
> > > this should be part of the set of APIs under
> /connector/<connector-name>
> > > (also the /connectors API should also have this information as it is an
> > > amalgamation of all the individual connector information). We can
> > introduce
> > > a new path under this for version (/connector/connector-name/version),
> > but
> > > perhaps adding this as part of the status is a valid alternative. This
> is
> > > mentioned as a rejected alternative right now. Also, to go further I
> > think
> > > version information for tasks could also be available, especially if we
> > > choose to not address the pitfalls discussed in my point 1), this will
> > > at-least provide admins a quick and easy way to determine if such and
> > > inconsistent state exist in any of the connectors.
> > >
> > > Thanks again for reviving my original KIP and working to improve it.
> > > Looking forward to your thoughts on the points mentioned above.
> > > Regards
> > > Snehashis
> > >
> > >
> > > On Wed, May 29, 2024 at 9:59 PM Chris Egerton <chrise@aiven.io.invalid
> >
> > > wrote:
> > >
> > > > Hi Greg,
> > > >
> > > > First, an apology! I mistakenly assumed that each plugin appeared
> only
> > > once
> > > > in the responses from GET /connector-plugins?connectorsOnly=false.
> > Thank
> > > > you for correcting me and pointing out that all versions of each
> plugin
> > > > appear in that response, which does indeed satisfy my desire for
> users
> > to
> > > > discover this information in at most two REST requests (and in fact,
> > does
> > > > it in only one)!
> > > >
> > > > And secondly, with the revelation about recommenders, I agree that
> it's
> > > > best to leave the "version" property out of the lists of properties
> > > > returned from the GET /connector-plugins/<plugin>/config endpoint.
> > > >
> > > > With those two points settled, I think the only unresolved item is
> the
> > > > small change to version parsing added to the KIP (where raw version
> > > numbers
> > > > are treated as an exact match, instead of a best-effort match with a
> > > > fallback on the default version). If the KIP is updated with that
> then
> > > I'd
> > > > be ready to vote on it.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Wed, May 29, 2024 at 12:00 PM Greg Harris
> > > <greg.harris@aiven.io.invalid
> > > > >
> > > > wrote:
> > > >
> > > > > Hey Chris,
> > > > >
> > > > > Thanks for your thoughts.
> > > > >
> > > > > > Won't it still only expose the
> > > > > > latest version for each plugin, instead of the range of versions
> > > > > available?
> > > > >
> > > > > Here is a snippet of the current output of the GET
> > > > > /connector-plugins?connectorsOnly=false endpoint, after I installed
> > two
> > > > > versions of the debezium PostgresConnector:
> > > > >
> > > > > {
> > > > > "class": "io.debezium.connector.postgresql.PostgresConnector",
> > > > > "type": "source",
> > > > > "version": "2.0.1.Final"
> > > > > },
> > > > > {
> > > > > "class": "io.debezium.connector.postgresql.PostgresConnector",
> > > > > "type": "source",
> > > > > "version": "2.6.1.Final"
> > > > > },
> > > > >
> > > > > I think this satisfies your requirement to learn about all plugins
> > and
> > > > all
> > > > > versions in two or fewer REST calls.
> > > > >
> > > > > I tried to get an example of the output of `/config` by hardcoding
> > the
> > > > > Recommender, and realized that Recommenders aren't executed on the
> > > > > `/config` endpoint at all: only during validation, when a
> > configuration
> > > > is
> > > > > actually present.
> > > > > And this led me to discover that the `/config` endpoint returns a
> > > > > List<ConfigKeyInfo>, and ConfigKeyInfo does not contain a
> > > > recommendedValues
> > > > > field. The ConfigValue field is the object which contains
> > > > > recommendedValues, and it is only generated during validation.
> > > > > I think it's out of scope to start calling recommenders on empty
> > > configs
> > > > > that might throw exceptions, changing the existing REST entities,
> or
> > > > > changing the core ConfigDef implementation.
> > > > > Someone could add this functionality later, I don't think it's
> > > necessary
> > > > > here.
> > > > >
> > > > > Then the question is: should "version" without recommenders appear
> in
> > > > > non-connector plugins? I think I'd rather be consistent with
> > > "predicate"
> > > > > and "negate" on release, and let a later improvement add them.
> > > > >
> > > > > Thanks,
> > > > > Greg
> > > > >
> > > > > On Wed, May 29, 2024 at 8:06 AM Chris Egerton
> > <chrise@aiven.io.invalid
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Greg,
> > > > > >
> > > > > > I'm confused about the behavior for GET
> > > > > > /connector-plugins?connectorsOnly=false. Won't it still only
> expose
> > > the
> > > > > > latest version for each plugin, instead of the range of versions
> > > > > available?
> > > > > >
> > > > > > I'm hoping we can provide a flow where people need at most two
> REST
> > > > calls
> > > > > > to discover 1) the complete set of plugins available on the
> worker
> > > > (which
> > > > > > is already possible with the endpoint under discussion) and 2)
> the
> > > set
> > > > of
> > > > > > versions available for a specific plugin on the worker (which
> > doesn't
> > > > > > appear to be possible, at least for some plugin types). This
> > wouldn't
> > > > > > require any out-of-band knowledge and would be valuable for
> > connector
> > > > > users
> > > > > > (who may want to, for example, know what their options are when
> > > > > considering
> > > > > > a plugin upgrade) and cluster administrators (who could use it
> as a
> > > > > sanity
> > > > > > check for the setup of their Kafka Connect clusters without
> having
> > to
> > > > > pore
> > > > > > over log files).
> > > > > >
> > > > > > As far as modifying the content of the GET
> > > > > > /connector-plugins/<plugin>/config endpoint to include a
> "version"
> > > > > property
> > > > > > goes, I think your point about returning that property from
> > requests
> > > to
> > > > > > that endpoint that include a version query parameter is salient,
> > but
> > > it
> > > > > > also unfortunately applies to all types of plugin. I don't think
> it
> > > > > should
> > > > > > completely disqualify that option. I also wasn't imagining adding
> > > > > anything
> > > > > > besides that single property to that endpoint, so no new
> > "predicate"
> > > > > > property for SMTs, no new "negate" property for predicates, and
> no
> > > new
> > > > > > "type" property for either. I'm not necessarily opposed to adding
> > > > those,
> > > > > > but it can be done without being pulled into the scope of this
> KIP.
> > > > > >
> > > > > > So to summarize how I imagine people might use the REST API after
> > > this
> > > > > KIP:
> > > > > >
> > > > > > 1. Discover the set of plugins on the worker via GET
> > > > > > /connector-plugins?connectorsOnly=false
> > > > > > 2. For any of those plugins, discover the set of available
> versions
> > > by
> > > > > > examining the recommended values for the "version" property after
> > > > hitting
> > > > > > the GET GET /connector-plugins/<plugin>/config endpoint
> > > > > >
> > > > > > I don't think this is necessarily optimal since plucking the
> > > "version"
> > > > > > property out of the response from that endpoint might be a PITA
> for
> > > ad
> > > > > hoc
> > > > > > queries via, e.g., curl, but I think it'd be enough for this KIP.
> > > Still
> > > > > > open to other options though, and if I've misunderstood anything
> in
> > > the
> > > > > > proposal, please let me know!
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Wed, May 22, 2024 at 4:05 PM Greg Harris
> > > > <greg.harris@aiven.io.invalid
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hey Chris,
> > > > > > >
> > > > > > > Thanks for your comments, and I'm glad that it seems like we're
> > > > > > > aligning on the vision here.
> > > > > > >
> > > > > > > > An
> > > > > > > > alternative could be to change existing behavior to fail fast
> > on
> > > > any
> > > > > > > > invalid default converter configuration instead of just for
> > > invalid
> > > > > > > > versions
> > > > > > >
> > > > > > > I suppose if this is landing in 4.0, we have the opportunity to
> > > break
> > > > > > > compatibility and strictly validate the worker class configs,
> and
> > > > > > > could have a new consistent behavior instead of inconsistent
> but
> > > > > > > backwards-compatible behavior.
> > > > > > >
> > > > > > > > no other part of the KIP requires this change.
> > > > > > >
> > > > > > > This is correct, and a compelling argument. I'm fine leaving
> the
> > > > > > > strict worker validation off of this configuration to
> potentially
> > > be
> > > > > > > added later. If a KIP was raised to perform strict validation
> of
> > > the
> > > > > > > worker config, it would include the version config, and can
> > address
> > > > > > > backwards compatibility for both configs together.
> > > > > > >
> > > > > > > > RE exposing the version property in the
> > > > > > > /connector-plugins/<plugin>/config
> > > > > > > > endpoint, the behavior is inconsistent across plugin types.
> > > > > > >
> > > > > > > Yes it is, and that started in a bugfix, once people started
> > using
> > > > > > > this endpoint:
> https://issues.apache.org/jira/browse/KAFKA-14843
> > > . I
> > > > > > > wasn't planning on converging the behavior in this KIP, as I
> > > > > > > considered it out-of-scope, and was going to follow the current
> > > > > > > behavior. To summarize:
> > > > > > >
> > > > > > > GET /connector-plugins/<connector>/config will emit:
> > > > > > > * connector.class (already implemented)
> > > > > > > * connector.version (new)
> > > > > > > * key.converter (already implemented)
> > > > > > > * key.converter.version (new)
> > > > > > > * value.converter (already implemented)
> > > > > > > * value.converter.version (new)
> > > > > > > * header.converter (already implemented)
> > > > > > > * header.converter.version (new)
> > > > > > > But will NOT emit:
> > > > > > > * transforms.<alias>.type
> > > > > > > * transforms.<alias>.version
> > > > > > > * transforms.<alias>.predicate
> > > > > > > * predicates.<alias>.type
> > > > > > > * predicates.<alias>.version
> > > > > > > * predicates.<alias>.negate
> > > > > > >
> > > > > > > GET /connector-plugins/<converter>/config will NOT emit:
> > > > > > > * "" (there's not even a ".class" prefix!)
> > > > > > > * version
> > > > > > >
> > > > > > > GET /connector-plugins/<transform>/config will NOT emit:
> > > > > > > * type
> > > > > > > * version
> > > > > > > * predicate
> > > > > > >
> > > > > > > GET /connector-plugins/<predicate>/config will NOT emit:
> > > > > > > * type
> > > > > > > * version
> > > > > > > * negate
> > > > > > >
> > > > > > > Do you want the converter, transform, and predicate endpoints
> > > > changed?
> > > > > > > Do you want just "version", or do you want all of the prefixed
> > > > configs
> > > > > > > including "type", "predicate" and "negate"? How would you want
> to
> > > > > > > handle the converters?
> > > > > > > And when I say the configs are "not part of the plugin config
> > > itself"
> > > > > > > I mean that saying that GET
> > > > > > > /connector-plugins/Flatten$Key/config?version=3.8.0 has a
> > "version"
> > > > > > > config that must be "3.8.0" is a little bit nonsense, as the
> > > version
> > > > > > > is already specified.
> > > > > > >
> > > > > > > > IMO
> > > > > > > > it's worth including this information somewhere directly
> > > accessible
> > > > > > > without
> > > > > > > > having to provide a full connector config. FWIW I'd be fine
> > with
> > > > GET
> > > > > > > > /connector-plugins/<plugin>/versions as a first-class
> endpoint
> > > > > > >
> > > > > > > You don't have to provide a configuration to call GET
> > > > > > > /connector-plugins?connectorsOnly=false , is that endpoint not
> > > close
> > > > > > > enough to what you have in mind? See also the Rejected
> > Alternative
> > > > > > > "Adding new REST API endpoints"
> > > > > > >
> > > > > > > If you're calling /connector-plugins/<transform>/config, you
> know
> > > the
> > > > > > > name of a plugin right? That either comes from out-of-band
> > > knowledge,
> > > > > > > validating a connector config, or calling GET
> > > > > > > /connector-plugins?connectorsOnly=false.
> > > > > > > * If you have out-of-band knowledge of plugin classes, perhaps
> > you
> > > > > > > have out-of-band knowledge of versions too.
> > > > > > > * If you've just validated a connector config, there should be
> an
> > > > > > > accompanying "version" field there with an accurate default
> value
> > > and
> > > > > > > recommenders.
> > > > > > > * If you've called GET /connector-plugins?connectorsOnly=false,
> > > that
> > > > > > > endpoint includes version information.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Greg
> > > > > > >
> > > > > > > On Wed, May 22, 2024 at 11:05 AM Chris Egerton
> > > > <chrise@aiven.io.invalid
> > > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Greg,
> > > > > > > >
> > > > > > > > Hope you had a nice weekend! Gonna try to keep things
> concise.
> > > > > > > >
> > > > > > > > Concluded points:
> > > > > > > >
> > > > > > > > RE version recommenders, I agree it's likely that
> programmatic
> > > UIs
> > > > > will
> > > > > > > > already be able to handle dynamic configuration definitions,
> > and
> > > > the
> > > > > > > detail
> > > > > > > > about SMTs is a great point. I still anticipate some
> > awkwardness
> > > > with
> > > > > > > > connector versions, though: if the latest version supports
> some
> > > new
> > > > > > > > properties, then a user switches to an earlier version, a UI
> > may
> > > > > > respond
> > > > > > > by
> > > > > > > > wiping values for these properties. I guess we can bite the
> > > bullet,
> > > > > > > though.
> > > > > > > >
> > > > > > > > RE double-dinging during preflight validation for invalid
> > > > versions, I
> > > > > > > like
> > > > > > > > the analogy with login credentials. I'm convinced that the
> > > proposal
> > > > > in
> > > > > > > the
> > > > > > > > KIP is best 👍
> > > > > > > >
> > > > > > > > Continued points:
> > > > > > > >
> > > > > > > > RE failing on worker startup, sorry, I should be clearer:
> there
> > > is
> > > > no
> > > > > > > _new_
> > > > > > > > justification for it that doesn't also apply to existing
> > > behavior.
> > > > We
> > > > > > > > shouldn't diverge from existing behavior solely for this new
> > > case.
> > > > An
> > > > > > > > alternative could be to change existing behavior to fail fast
> > on
> > > > any
> > > > > > > > invalid default converter configuration instead of just for
> > > invalid
> > > > > > > > versions, but I'd vote to just stick to existing behavior and
> > not
> > > > > > > > complicate things, especially since no other part of the KIP
> > > > requires
> > > > > > > this
> > > > > > > > change.
> > > > > > > >
> > > > > > > > RE exposing the version property in the
> > > > > > > /connector-plugins/<plugin>/config
> > > > > > > > endpoint, the behavior is inconsistent across plugin types.
> > > Hitting
> > > > > the
> > > > > > > > endpoint for the FileStreamSinkConnector on version 3.7.0
> > yields
> > > a
> > > > > > > response
> > > > > > > > that includes, among other things, the "topics",
> > "topics.regex",
> > > > and
> > > > > > > > "errors.tolerance" properties. I see that we don't do this
> > > > everywhere
> > > > > > > (the
> > > > > > > > examples you cite for SMT and converter properties are
> > accurate),
> > > > but
> > > > > > IMO
> > > > > > > > it's worth including this information somewhere directly
> > > accessible
> > > > > > > without
> > > > > > > > having to provide a full connector config. FWIW I'd be fine
> > with
> > > > GET
> > > > > > > > /connector-plugins/<plugin>/versions as a first-class
> endpoint
> > > > either
> > > > > > > > instead of or in addition to adding recommended values for
> all
> > > > plugin
> > > > > > > > versions.
> > > > > > > >
> > > > > > > > Thanks for your continued work on this KIP, and with the
> > progress
> > > > > we're
> > > > > > > > making I'm optimistic about its chances of appearing in
> 4.0.0.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Wed, May 15, 2024 at 1:22 PM Greg Harris
> > > > > > <greg.harris@aiven.io.invalid
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Chris,
> > > > > > > > >
> > > > > > > > > Thanks for your quick follow up.
> > > > > > > > >
> > > > > > > > > > But this risk is already present with
> > > > > > > > > > existing error cases, and I don't see anything that
> > justifies
> > > > > > > changing
> > > > > > > > > > existing behavior with an invalid converter class, or
> > > diverging
> > > > > > from
> > > > > > > it
> > > > > > > > > in
> > > > > > > > > > the case of invalid converter versions.
> > > > > > > > >
> > > > > > > > > The justification is to fail-fast, and prevent REST API
> users
> > > > from
> > > > > > > > > receiving errors from bad configs that they didn't write,
> or
> > > > maybe
> > > > > > > > > don't even know apply to them.
> > > > > > > > > Up until recently errors in these configurations surfaced
> as
> > > > > failures
> > > > > > > > > to create the connector, or failures to start, and you made
> > > them
> > > > > > > > > fail-fast during validation. I think this change is in the
> > same
> > > > > > > > > spirit, pulling the validation further forward and not
> > letting
> > > > > errors
> > > > > > > > > lie dormant.
> > > > > > > > >
> > > > > > > > > And to call back to your original concern about
> interrupting
> > > > > > > > > connectors that explicitly provide these configurations and
> > > don't
> > > > > use
> > > > > > > > > the worker configs: I expect that operators with a majority
> > of
> > > > > these
> > > > > > > > > sorts of clients aren't going to be setting the worker
> > .version
> > > > > > > > > properties, because it would have no effect on the majority
> > of
> > > > > their
> > > > > > > > > connectors. They would be able to rely on
> > > backwards-compatibility
> > > > > and
> > > > > > > > > continue to ignore the class properties.
> > > > > > > > >
> > > > > > > > > > CLI
> > > > > > > > > > and programmatic UI developers will want to develop their
> > own
> > > > > > tooling
> > > > > > > > > > layers.
> > > > > > > > >
> > > > > > > > > This is a very compelling argument. Snehashis do you want
> to
> > > > figure
> > > > > > > > > out a REST API design for this use-case?
> > > > > > > > >
> > > > > > > > > > Regarding the GET /connector-plugins/<plugin>/config
> > > endpoint,
> > > > I
> > > > > > was
> > > > > > > > > > thinking about the response for non-connector plugins,
> > e.g.,
> > > > > > > > > > GET /connector-plugins/RegexRouter/config. Would a
> > "version"
> > > > > > property
> > > > > > > > > > appear with recommended values?
> > > > > > > > >
> > > > > > > > > I intended for the ".version" property to be like other
> > > framework
> > > > > > > > > configs (".class", ".type", ".predicate", ".negate") where
> > they
> > > > are
> > > > > > > > > inside the plugin namespace, but not part of the plugin
> > config
> > > > > > itself.
> > > > > > > > > Perhaps we can deviate from those configs because it would
> > aid
> > > in
> > > > > > > > > discovering other valid `GET
> > > > > > > > > /connector-plugins/<plugin>/config?version=` calls, without
> > > > calling
> > > > > > > > > `GET /connector-plugins?connectorsOnly=false`.
> > > > > > > > > I don't really feel strongly either way.
> > > > > > > > >
> > > > > > > > > > if a user changes the
> > > > > > > > > > connector version in, e.g., a dropdown menu, then the UI
> > > either
> > > > > has
> > > > > > > to
> > > > > > > > > > re-fetch the ConfigDef for the new version, or risk
> > operating
> > > > on
> > > > > > > stale
> > > > > > > > > > information
> > > > > > > > >
> > > > > > > > > This is a really interesting situation, thanks for finding
> > > that!
> > > > > This
> > > > > > > > > is already a footgun with transformations and predicates;
> > Once
> > > > you
> > > > > > > > > fill out a transformation class (which includes a
> > recommender)
> > > > you
> > > > > > > > > will then see all of the transformations configurations.
> > > > > > > > > I think this means that UI developers should have already
> > > > developed
> > > > > > > > > the infrastructure for handling dynamic recommenders, or
> else
> > > > > they've
> > > > > > > > > had this bug since KIP-66 in 2017. It may require some
> manual
> > > > > > > > > attention to roll-out support for the ".version" properties
> > > > > > > > > specifically, but I don't think that should prevent us from
> > > > > providing
> > > > > > > > > this information in a natural place.
> > > > > > > > >
> > > > > > > > > > but will it be possible to configure
> > > > > > > > > > a connector to use two instances of the same transform or
> > > > > > predicate,
> > > > > > > but
> > > > > > > > > > with different versions for each?
> > > > > > > > >
> > > > > > > > > Yes, and this is included in the example in the KIP,
> > > > > > > > > `transforms.flatten-latest` and `transforms.flatten-old`.
> > This
> > > > will
> > > > > > > > > work in the natural way, with the two instances having a
> > > > different
> > > > > > > > > version if configured for that. If a user wants to pin the
> > same
> > > > > > > > > version of a plugin for every instance, they will need to
> > > provide
> > > > > the
> > > > > > > > > version config multiple times and keep them in-sync
> > themselves.
> > > > > > > > >
> > > > > > > > > > I would have expected the error
> > > > > > > > > > to only be attributed to the version property, and for
> the
> > > > class
> > > > > > > property
> > > > > > > > > > to be reported as valid.
> > > > > > > > >
> > > > > > > > > I considered this, and then went back and changed it. To
> > quote
> > > > > > someone
> > > > > > > > > else, this is to catch "misspelling cat as dog". For
> example
> > a
> > > > user
> > > > > > > > > types DogConverter and meant to type CatConverter, and then
> > > > types a
> > > > > > > > > version which is valid for CatConverter, conceptually the
> > error
> > > > is
> > > > > in
> > > > > > > > > the class name and not the version.
> > > > > > > > > That's a very contrived scenario, but I think similar
> > arguments
> > > > are
> > > > > > > > > used for attributing validation errors to
> > > > endpoints/urls/hostnames
> > > > > > > > > when the associated credentials are unable to log into the
> > > remote
> > > > > > > > > system. Did the user provide the correct testing
> credentials,
> > > but
> > > > > > > > > accidentally typed the production endpoint? Even if the
> > > > production
> > > > > > > > > endpoint looks valid (it's a real hostname that is
> reachable)
> > > the
> > > > > > > > > conceptual error is still in the hostname and should have
> the
> > > > error
> > > > > > > > > attributed to it to draw the user's attention.
> > > > > > > > > If that's not convincing, I think the alternative of only
> > > > > attributing
> > > > > > > > > version errors to the version property is also acceptable.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Greg
> > > > > > > > >
> > > > > > > > > On Wed, May 15, 2024 at 8:06 AM Chris Egerton
> > > > > > <chrise@aiven.io.invalid
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Greg,
> > > > > > > > > >
> > > > > > > > > > Thanks for your responses! Continuations of existing
> > > > discussions:
> > > > > > > > > >
> > > > > > > > > > Regarding crashing the worker on startup--yes, there is
> > also
> > > a
> > > > > risk
> > > > > > > to
> > > > > > > > > > allowing it to join the cluster. But this risk is already
> > > > present
> > > > > > > with
> > > > > > > > > > existing error cases, and I don't see anything that
> > justifies
> > > > > > > changing
> > > > > > > > > > existing behavior with an invalid converter class, or
> > > diverging
> > > > > > from
> > > > > > > it
> > > > > > > > > in
> > > > > > > > > > the case of invalid converter versions. I think we should
> > > keep
> > > > > this
> > > > > > > > > simple
> > > > > > > > > > and not do anything different for worker startup than we
> do
> > > > > > already.
> > > > > > > > > >
> > > > > > > > > > As far as REST API vs. metrics go--I think you're right
> > that
> > > > the
> > > > > > > original
> > > > > > > > > > version metrics were added as a "monitoring" detail.
> > However,
> > > > > this
> > > > > > > was
> > > > > > > > > back
> > > > > > > > > > when plugin versions were managed solely by cluster
> > > > > administrators.
> > > > > > > With
> > > > > > > > > > this KIP, connector users will be able to manage plugin
> > > > versions,
> > > > > > > and CLI
> > > > > > > > > > and programmatic UI developers will want to develop their
> > own
> > > > > > tooling
> > > > > > > > > > layers. I think focusing on the REST API as the primary
> > > > interface
> > > > > > for
> > > > > > > > > this
> > > > > > > > > > KIP would be best for these users.
> > > > > > > > > >
> > > > > > > > > > (All that said, I don't object to the metrics that are
> > > proposed
> > > > > in
> > > > > > > the
> > > > > > > > > KIP;
> > > > > > > > > > I just think they make more sense in addition to new REST
> > API
> > > > > > > > > > functionality, as opposed to instead of it.)
> > > > > > > > > >
> > > > > > > > > > Regarding the GET /connector-plugins/<plugin>/config
> > > endpoint,
> > > > I
> > > > > > was
> > > > > > > > > > thinking about the response for non-connector plugins,
> > e.g.,
> > > > > > > > > > GET /connector-plugins/RegexRouter/config. Would a
> > "version"
> > > > > > property
> > > > > > > > > > appear with recommended values?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > And new thoughts:
> > > > > > > > > >
> > > > > > > > > > 1) Regarding the recommended values for
> > "connector.version",
> > > > this
> > > > > > > might
> > > > > > > > > be
> > > > > > > > > > confusing since there could be differences between the
> > > > ConfigDefs
> > > > > > > for the
> > > > > > > > > > latest version of the connector and prior versions. It
> also
> > > > makes
> > > > > > the
> > > > > > > > > flow
> > > > > > > > > > a bit awkward for programmatic UI developers: if a user
> > > changes
> > > > > the
> > > > > > > > > > connector version in, e.g., a dropdown menu, then the UI
> > > either
> > > > > has
> > > > > > > to
> > > > > > > > > > re-fetch the ConfigDef for the new version, or risk
> > operating
> > > > on
> > > > > > > stale
> > > > > > > > > > information. I'm starting to doubt that exposing the
> range
> > of
> > > > > > > available
> > > > > > > > > > versions via recommended values is the best way to
> proceed,
> > > > > instead
> > > > > > > of a
> > > > > > > > > > more explicit approach like GET
> > > > > > > /connector-plugins/<plugin>/versions, or
> > > > > > > > > > the "Adding new REST API endpoints" rejected alternative.
> > > > > > > > > >
> > > > > > > > > > 2) I know that this is a bit heinous, but will it be
> > possible
> > > > to
> > > > > > > > > configure
> > > > > > > > > > a connector to use two instances of the same transform or
> > > > > > predicate,
> > > > > > > but
> > > > > > > > > > with different versions for each? (I don't think this is
> > > worth
> > > > > > > > > significant
> > > > > > > > > > design/implementation effort, so if it would inflate
> either
> > > of
> > > > > > those,
> > > > > > > > > > please don't feel obligated to support it.)
> > > > > > > > > >
> > > > > > > > > > 3) Just out of curiosity, why double-ding during config
> > > > > validation
> > > > > > > if the
> > > > > > > > > > version for a plugin class can't be found? I would have
> > > > expected
> > > > > > the
> > > > > > > > > error
> > > > > > > > > > to only be attributed to the version property, and for
> the
> > > > class
> > > > > > > property
> > > > > > > > > > to be reported as valid.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
> > > > > > > > > > Chris
> > > > > > > > > >
> > > > > > > > > > On Tue, May 14, 2024 at 1:11 PM Greg Harris
> > > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey Chris,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your comments. I'm glad you like the
> > > motivations,
> > > > > > > Snehashis
> > > > > > > > > > > wrote that part!
> > > > > > > > > > >
> > > > > > > > > > > > the configuration syntax for the most basic use case
> of
> > > > > > > > > > > > specifying a single desired version is pretty
> > > > > counterintuitive.
> > > > > > > > > > >
> > > > > > > > > > > I agree, and the "soft" requirement scheme is
> something I
> > > > > wasn't
> > > > > > > > > > > explicitly looking for, but would be inherited from the
> > > > > library.
> > > > > > > I'm
> > > > > > > > > > > fine with eliminating the soft requirement semantics,
> and
> > > > > having
> > > > > > > > > > > "1.0.0" behave the same as "[1.0.0]". I'm less inclined
> > to
> > > > > > include
> > > > > > > > > > > "range" in the property name, or have two properties.
> > > > > > > > > > >
> > > > > > > > > > > > Failing startup is drastic and has the potential to
> > > disrupt
> > > > > the
> > > > > > > > > > > > availability of connectors that would otherwise be
> able
> > > to
> > > > > run
> > > > > > > > > healthily
> > > > > > > > > > > > because they were explicitly configured to use valid
> > > > > converters
> > > > > > > > > instead
> > > > > > > > > > > of
> > > > > > > > > > > > the worker defaults.
> > > > > > > > > > >
> > > > > > > > > > > I think this argument cuts both ways. If someone
> > > > reconfigures a
> > > > > > > worker
> > > > > > > > > > > and adds an invalid ".version" string to the worker (or
> > > > changes
> > > > > > the
> > > > > > > > > > > plugin.path to make it invalid), it would be permitted
> to
> > > > enter
> > > > > > the
> > > > > > > > > > > group, and accept work assignments. If those work
> > > assignments
> > > > > > used
> > > > > > > > > > > these configurations, a set of tasks could transition
> to
> > > > FAILED
> > > > > > and
> > > > > > > > > > > not be able to recover, because they would be restarted
> > > again
> > > > > on
> > > > > > > the
> > > > > > > > > > > same worker.
> > > > > > > > > > >
> > > > > > > > > > > > Why are metrics utilized to report information about
> > > plugin
> > > > > > > versions
> > > > > > > > > > > > utilized by connectors at runtime instead of
> publishing
> > > > this
> > > > > > > info in
> > > > > > > > > the
> > > > > > > > > > > > REST API
> > > > > > > > > > >
> > > > > > > > > > > I was following the existing pattern for exposing
> runtime
> > > > > > versions
> > > > > > > for
> > > > > > > > > > > connectors, and it did seem like a "monitoring"
> feature.
> > If
> > > > > that
> > > > > > > > > > > approach is flawed and should be phased out, I think it
> > > would
> > > > > be
> > > > > > a
> > > > > > > > > > > good idea to reconsider the REST API rejected
> > alternative.
> > > > > > > > > > > We would need some additional design work to spec out
> the
> > > > REST
> > > > > > API
> > > > > > > > > > > interface, as I don't have anything in mind currently.
> > > > > > > > > > >
> > > > > > > > > > > > I'm unclear on whether or not it'll be possible to
> see
> > > this
> > > > > > > > > information
> > > > > > > > > > > via the
> > > > > > > > > > > > GET /connector-plugins/<plugin>/config endpoint
> > > > > > > > > > >
> > > > > > > > > > > There is room in the API to add recommenders for
> > > > > "key.converter",
> > > > > > > > > > > "value.converter", and "header.converter", but not for
> > > > > transforms
> > > > > > > and
> > > > > > > > > > > predicates, as they include aliases that depend on an
> > > actual
> > > > > > > > > > > configuration. We could explicitly say we're going to
> do
> > > > that,
> > > > > or
> > > > > > > do
> > > > > > > > > > > whatever is convenient during the implementation phase,
> > or
> > > > > leave
> > > > > > it
> > > > > > > > > > > open to be improved later.
> > > > > > > > > > > There will not be any recommenders for ".version"
> > > properties
> > > > in
> > > > > > the
> > > > > > > > > > > `/config` endpoint, because those recommenders are
> > dynamic
> > > > and
> > > > > > > depend
> > > > > > > > > > > on an actual configuration.
> > > > > > > > > > >
> > > > > > > > > > > 5) There are two relevant lines in the KIP: "If a
> > .version
> > > > > > property
> > > > > > > > > > > contains a hard requirement, select the latest
> installed
> > > > > version
> > > > > > > which
> > > > > > > > > > > satisfies the requirement." and "This configuration is
> > > > > > re-evaluated
> > > > > > > > > > > each time the connector or task are assigned to a new
> > > > worker".
> > > > > I
> > > > > > > would
> > > > > > > > > > > call this "eager" upgrade behavior, rather than a
> > "sticky"
> > > or
> > > > > > > "lazy"
> > > > > > > > > > > upgrade behavior.
> > > > > > > > > > >
> > > > > > > > > > > 6) Updated!
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Greg
> > > > > > > > > > >
> > > > > > > > > > > On Tue, May 14, 2024 at 9:14 AM Chris Egerton
> > > > > > > <chrise@aiven.io.invalid
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks Greg for updating the KIP, and thanks
> Snehashis
> > > for
> > > > > > > starting
> > > > > > > > > the
> > > > > > > > > > > > work on this originally.
> > > > > > > > > > > >
> > > > > > > > > > > > The motivation section makes a pretty convincing case
> > for
> > > > > this
> > > > > > > kind
> > > > > > > > > of
> > > > > > > > > > > > feature. My thoughts are mostly about specific
> details:
> > > > > > > > > > > >
> > > > > > > > > > > > 1) I like the support for version ranges (the example
> > > > > > > demonstrating
> > > > > > > > > how
> > > > > > > > > > > to
> > > > > > > > > > > > avoid KAFKA-10574 with the header converter was
> > > > particularly
> > > > > > > > > > > entertaining),
> > > > > > > > > > > > but the configuration syntax for the most basic use
> > case
> > > of
> > > > > > > > > specifying a
> > > > > > > > > > > > single desired version is pretty counterintuitive.
> > People
> > > > may
> > > > > > get
> > > > > > > > > bitten
> > > > > > > > > > > or
> > > > > > > > > > > > at least frustrated if they put
> connector.version=3.8.0
> > > in
> > > > a
> > > > > > > config
> > > > > > > > > but
> > > > > > > > > > > > then version 3.7.5 ends up running. I'd like it if we
> > > could
> > > > > > > either
> > > > > > > > > > > > intentionally deviate from Maven ranges when a bare
> > > version
> > > > > is
> > > > > > > > > present,
> > > > > > > > > > > or
> > > > > > > > > > > > separate things out into two properties: foo.version
> > > would
> > > > be
> > > > > > the
> > > > > > > > > single
> > > > > > > > > > > > accepted version for the foo plugin, and
> > > foo.version.range
> > > > > > would
> > > > > > > use
> > > > > > > > > > > Maven
> > > > > > > > > > > > range syntax. Open to other options too, just
> > providing a
> > > > > > couple
> > > > > > > to
> > > > > > > > > get
> > > > > > > > > > > the
> > > > > > > > > > > > ball rolling.
> > > > > > > > > > > >
> > > > > > > > > > > > 2) Although the current behavior for a worker with an
> > > > invalid
> > > > > > > > > > > > key/value/header converter class specified in its
> > config
> > > > file
> > > > > > is
> > > > > > > a
> > > > > > > > > little
> > > > > > > > > > > > strange (I was surprised to learn that it wouldn't
> fail
> > > on
> > > > > > > startup),
> > > > > > > > > I
> > > > > > > > > > > > don't see a good reason to deviate from this when an
> > > > invalid
> > > > > > > version
> > > > > > > > > is
> > > > > > > > > > > > specified. Failing startup is drastic and has the
> > > potential
> > > > > to
> > > > > > > > > disrupt
> > > > > > > > > > > the
> > > > > > > > > > > > availability of connectors that would otherwise be
> able
> > > to
> > > > > run
> > > > > > > > > healthily
> > > > > > > > > > > > because they were explicitly configured to use valid
> > > > > converters
> > > > > > > > > instead
> > > > > > > > > > > of
> > > > > > > > > > > > the worker defaults.
> > > > > > > > > > > >
> > > > > > > > > > > > 3) Why are metrics utilized to report information
> about
> > > > > plugin
> > > > > > > > > versions
> > > > > > > > > > > > utilized by connectors at runtime instead of
> publishing
> > > > this
> > > > > > > info in
> > > > > > > > > the
> > > > > > > > > > > > REST API? I saw that this was mentioned as a rejected
> > > > > > > alternative,
> > > > > > > > > but I
> > > > > > > > > > > > didn't get a sense of why. It seems like the REST API
> > > would
> > > > > be
> > > > > > > > > easier to
> > > > > > > > > > > > access and more intuitive for most users than new
> > > metrics.
> > > > > > > > > > > >
> > > > > > > > > > > > 4) In the "Validation" section it's stated that
> "Users
> > > can
> > > > > use
> > > > > > > these
> > > > > > > > > > > > recommenders to discover the valid plugin classes and
> > > > > versions,
> > > > > > > > > without
> > > > > > > > > > > > requiring an earlier call to GET
> > > > > > > > > > > /connector-plugins?connectorsOnly=false."
> > > > > > > > > > > > I really like the creativity and simplicity of
> reusing
> > > the
> > > > > > > > > recommender
> > > > > > > > > > > > mechanism to expose available versions for plugins
> via
> > > the
> > > > > REST
> > > > > > > API.
> > > > > > > > > I'm
> > > > > > > > > > > > unclear on whether or not it'll be possible to see
> this
> > > > > > > information
> > > > > > > > > via
> > > > > > > > > > > the
> > > > > > > > > > > > GET /connector-plugins/<plugin>/config endpoint,
> > though.
> > > > It'd
> > > > > > be
> > > > > > > > > great if
> > > > > > > > > > > > this were supported, since we learned in KIP-769 [1]
> > that
> > > > > > people
> > > > > > > > > really
> > > > > > > > > > > > want to be able to see configuration options for
> > > connectors
> > > > > and
> > > > > > > their
> > > > > > > > > > > > plugins via some kind of GET endpoint without having
> to
> > > > > > provide a
> > > > > > > > > > > complete
> > > > > > > > > > > > connector config for validation.
> > > > > > > > > > > >
> > > > > > > > > > > > 5) In the Maven version range docs, it's stated that
> > > "Maven
> > > > > > > picks the
> > > > > > > > > > > > highest version of each project that satisfies all
> the
> > > hard
> > > > > > > > > requirements
> > > > > > > > > > > of
> > > > > > > > > > > > the dependencies on that project." I'm guessing this
> > > > behavior
> > > > > > > will be
> > > > > > > > > > > > retained for Connect; i.e., the highest-possible
> > version
> > > of
> > > > > > each
> > > > > > > > > plugin
> > > > > > > > > > > > that satisfies the user-specified version constraints
> > > will
> > > > be
> > > > > > > run?
> > > > > > > > > (An
> > > > > > > > > > > > alternative approach could be to have some kind of
> > > "sticky"
> > > > > > logic
> > > > > > > > > that
> > > > > > > > > > > only
> > > > > > > > > > > > restarts connectors/tasks when their currently-used
> > > version
> > > > > > > becomes
> > > > > > > > > > > > incompatible with the configured constraints.)
> > > > > > > > > > > >
> > > > > > > > > > > > 6) (Nit) It'd be nice to add a link to the
> TestPlugins
> > > > class
> > > > > or
> > > > > > > > > somewhere
> > > > > > > > > > > > in its neighborhood to the testing plan; unfamiliar
> > > readers
> > > > > > > probably
> > > > > > > > > > > won't
> > > > > > > > > > > > get much out of what's there right now.
> > > > > > > > > > > >
> > > > > > > > > > > > [1] -
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > >
> > > > > > > > > > > > Chris
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, May 13, 2024 at 2:01 PM Snehashis <
> > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > >
> > > > > > > > > > > > > That is much appreciated. No complaints on the
> > > additional
> > > > > > > scope, I
> > > > > > > > > will
> > > > > > > > > > > > > make some time out to work on this once we have
> > > approval.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> > > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hey Snehashis,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'm glad to hear you're still interested in this
> > KIP!
> > > > > > > > > > > > > > I'm happy to let you drive this, and I apologize
> > for
> > > > > > > increasing
> > > > > > > > > the
> > > > > > > > > > > > > > scope of work so drastically. To make up for
> that,
> > > I'll
> > > > > > > > > volunteer to
> > > > > > > > > > > > > > be the primary PR reviewer to help get this done
> > > > quickly
> > > > > > once
> > > > > > > > > the KIP
> > > > > > > > > > > > > > is approved.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, May 10, 2024 at 3:51 AM Snehashis <
> > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the follow up to my original KIP, I
> am
> > > in
> > > > > > > favour of
> > > > > > > > > the
> > > > > > > > > > > > > > > additions made to expand its scope, the
> addition
> > of
> > > > > range
> > > > > > > > > versions
> > > > > > > > > > > > > > > specifically make a lot of sense.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Apologies if I have not publicly worked on this
> > KIP
> > > > > for a
> > > > > > > long
> > > > > > > > > > > time.
> > > > > > > > > > > > > The
> > > > > > > > > > > > > > > original work was done when the move to service
> > > > loading
> > > > > > > was in
> > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > and I wanted to loop back to this only after
> that
> > > > work
> > > > > > was
> > > > > > > > > > > completed.
> > > > > > > > > > > > > > Post
> > > > > > > > > > > > > > > its conclusion, I have not been able to take
> this
> > > up
> > > > > due
> > > > > > to
> > > > > > > > > other
> > > > > > > > > > > > > > > priorities. If it's okay with you, I would
> still
> > > like
> > > > > to
> > > > > > > get
> > > > > > > > > this
> > > > > > > > > > > > > > > implemented myself, including the additional
> > scope.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks and regards
> > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> > > > > > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'd like to reboot the discussion on KIP-891:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I've made some changes, most notably:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. Specifying versions for all plugins in
> > > Connector
> > > > > > > configs
> > > > > > > > > > > > > > > > (converters, header converters, transforms,
> and
> > > > > > > predicates)
> > > > > > > > > not
> > > > > > > > > > > just
> > > > > > > > > > > > > > > > connectors & tasks
> > > > > > > > > > > > > > > > 2. Specifying a range of versions instead of
> an
> > > > exact
> > > > > > > match
> > > > > > > > > > > > > > > > 3. New metrics to observe what versions are
> > > in-use
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks to Snehashis for the original KIP
> idea!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <
> > > > > > > > > > > greg.harris@aiven.io>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Snehashis,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thank you for the KIP! This is something
> I've
> > > > > wanted
> > > > > > > for a
> > > > > > > > > long
> > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I know the discussion has gone cold, are
> you
> > > > still
> > > > > > > > > interested
> > > > > > > > > > > in
> > > > > > > > > > > > > > > > > pursuing this feature? I'll make time to
> > review
> > > > the
> > > > > > > KIP if
> > > > > > > > > you
> > > > > > > > > > > are
> > > > > > > > > > > > > > > > > still accepting comments.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis
> <
> > > > > > > > > > > > > snehashisp1995@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for the points Sagar.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1) Should we update the GET /connectors
> > > > > endpoint
> > > > > > to
> > > > > > > > > > > include the
> > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > the plugin that is running? It could be
> > > > useful
> > > > > to
> > > > > > > > > figure
> > > > > > > > > > > out
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > the plugin or I am assuming it gets
> > > returned
> > > > by
> > > > > > the
> > > > > > > > > > > expand=info
> > > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think this is good to have and possible
> > > > future
> > > > > > > > > > > enhancement. The
> > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > info will be present in the config of the
> > > > > connector
> > > > > > > if
> > > > > > > > > the
> > > > > > > > > > > user
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > specified the version. Otherwise it is
> the
> > > > latest
> > > > > > > version
> > > > > > > > > > > which
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > can find out from the connector-plugin
> > > > endpoint.
> > > > > > The
> > > > > > > > > > > information
> > > > > > > > > > > > > > can be
> > > > > > > > > > > > > > > > > > introduced to the response of the GET
> > > > /connectors
> > > > > > > > > endpoint
> > > > > > > > > > > > > itself,
> > > > > > > > > > > > > > > > however
> > > > > > > > > > > > > > > > > > the most ideal way of doing this would be
> > to
> > > > get
> > > > > > the
> > > > > > > > > > > currently
> > > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > instance of the connector and get the
> > version
> > > > > > > directly
> > > > > > > > > from
> > > > > > > > > > > > > there.
> > > > > > > > > > > > > > > > This is
> > > > > > > > > > > > > > > > > > slightly tricky as the connector could be
> > > > running
> > > > > > in
> > > > > > > a
> > > > > > > > > > > different
> > > > > > > > > > > > > > node.
> > > > > > > > > > > > > > > > > > One way to do this would be to persist
> the
> > > > > version
> > > > > > > > > > > information in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > status backing store during instantiation
> > of
> > > > the
> > > > > > > > > connector.
> > > > > > > > > > > It
> > > > > > > > > > > > > > requires
> > > > > > > > > > > > > > > > > > some more thought and since the version
> is
> > > part
> > > > > of
> > > > > > > the
> > > > > > > > > > > configs if
> > > > > > > > > > > > > > > > provided
> > > > > > > > > > > > > > > > > > and evident otherwise, I have not
> included
> > it
> > > > in
> > > > > > this
> > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2) I am not aware of this and hence
> > asking,
> > > > > can 2
> > > > > > > > > > > connectors
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > versions have the same name? Does the
> > > plugin
> > > > > > > isolation
> > > > > > > > > > > allow
> > > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > could have a bearing when using the
> > > lifecycle
> > > > > > > > > endpoints for
> > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > All connectors in a cluster need to have
> > > > uniquire
> > > > > > > > > connector
> > > > > > > > > > > names
> > > > > > > > > > > > > > > > > > regardless of what version of the plugin
> > the
> > > > > > > connector is
> > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > underneath. This is something enforced by
> > the
> > > > > > connect
> > > > > > > > > runtime
> > > > > > > > > > > > > > itself.
> > > > > > > > > > > > > > > > All
> > > > > > > > > > > > > > > > > > connect CRUD operations are keyed on the
> > > > > connector
> > > > > > > name
> > > > > > > > > so
> > > > > > > > > > > there
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > be an issue.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <
> > > > > > > > > > > sagarmeansocean@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hey Snehashsih,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for the KIP. It looks like a
> very
> > > > useful
> > > > > > > > > feature.
> > > > > > > > > > > Couple
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > small-ish points, let me know what you
> > > think:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1) Should we update the GET /connectors
> > > > > endpoint
> > > > > > to
> > > > > > > > > > > include the
> > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > the plugin that is running? It could be
> > > > useful
> > > > > to
> > > > > > > > > figure
> > > > > > > > > > > out
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > the plugin or I am assuming it gets
> > > returned
> > > > by
> > > > > > the
> > > > > > > > > > > expand=info
> > > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > > > 2) I am not aware of this and hence
> > asking,
> > > > > can 2
> > > > > > > > > > > connectors
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > versions have the same name? Does the
> > > plugin
> > > > > > > isolation
> > > > > > > > > > > allow
> > > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > could have a bearing when using the
> > > lifecycle
> > > > > > > > > endpoints for
> > > > > > > > > > > > > > > > connectors like
> > > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > Sagar.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin
> > > > > > > > > > > > > > <apankaj@confluent.io.invalid
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > IIUC (please correct me if I am
> wrong
> > > > > here),
> > > > > > > what
> > > > > > > > > you
> > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > a versioning scheme for a connector
> > > config
> > > > > for
> > > > > > > the
> > > > > > > > > same
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > different versions of a connector
> > > plugin).
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Sorry for not being more precise in
> my
> > > > > wording
> > > > > > > - I
> > > > > > > > > meant
> > > > > > > > > > > > > > > > registering
> > > > > > > > > > > > > > > > > > > > versions of schema for connector
> > config.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Let's take the example of a fictional
> > > > > connector
> > > > > > > which
> > > > > > > > > > > uses a
> > > > > > > > > > > > > > > > fictional
> > > > > > > > > > > > > > > > > > > AWS
> > > > > > > > > > > > > > > > > > > > service.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Fictional Connector Config schema
> > > > version:2.0
> > > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Fictional Connector config schema
> > > > version:3.0
> > > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > "iam_role": {
> > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > > "iam_role"
> > > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The connector which supports
> Fictional
> > > > config
> > > > > > > schema
> > > > > > > > > 2.0
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > validate
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > access key and secret key.
> > > > > > > > > > > > > > > > > > > > Whereas a connector which supports
> > config
> > > > > with
> > > > > > > schema
> > > > > > > > > > > version
> > > > > > > > > > > > > > 3.0
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > > > validate the IAM role.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This is the alternative which I
> wanted
> > to
> > > > > > > suggest.
> > > > > > > > > Each
> > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > register the schema versions of
> > connector
> > > > > > config
> > > > > > > > > which it
> > > > > > > > > > > > > > supports.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The plugin paths may be optionally
> > > > different
> > > > > > > i.e we
> > > > > > > > > > > don't
> > > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > > > mandatorily add a new plugin path to
> > > > support
> > > > > a
> > > > > > > new
> > > > > > > > > schema
> > > > > > > > > > > > > > version.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:47 PM
> > > Snehashis
> > > > <
> > > > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on the
> > rejected
> > > > > > > > > alternatives ?
> > > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > config is versioned and has a
> > schema.
> > > > > Then
> > > > > > a
> > > > > > > > > single
> > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > > dependencies have not changed)
> can
> > > > handle
> > > > > > > > > multiple
> > > > > > > > > > > config
> > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > IIUC (please correct me if I am
> wrong
> > > > > here),
> > > > > > > what
> > > > > > > > > you
> > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > a versioning scheme for a connector
> > > > config
> > > > > > for
> > > > > > > the
> > > > > > > > > same
> > > > > > > > > > > > > > > > connector (and
> > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > different versions of a connector
> > > > plugin).
> > > > > > > That is
> > > > > > > > > a
> > > > > > > > > > > > > somewhat
> > > > > > > > > > > > > > > > > > > tangential
> > > > > > > > > > > > > > > > > > > > > problem. While it is definitely a
> > > useful
> > > > > > > feature to
> > > > > > > > > > > have,
> > > > > > > > > > > > > > like a
> > > > > > > > > > > > > > > > log to
> > > > > > > > > > > > > > > > > > > > > check what changes were made over
> > time
> > > to
> > > > > the
> > > > > > > > > config
> > > > > > > > > > > which
> > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > easier to do rollbacks, it is not
> the
> > > > focus
> > > > > > > here.
> > > > > > > > > Here
> > > > > > > > > > > by
> > > > > > > > > > > > > > > > version we
> > > > > > > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > > > > > to say what underlying version of
> the
> > > > > plugin
> > > > > > > > > should the
> > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > of the connector use. Perhaps it is
> > > > better
> > > > > to
> > > > > > > > > change
> > > > > > > > > > > the
> > > > > > > > > > > > > > name of
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > parameter from connector.version to
> > > > > > > > > > > > > connector.plugin.version
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > plugin.version if it was confusing.
> > > wdyt?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2. Any plans to support assisted
> > > > > migration
> > > > > > > e.g
> > > > > > > > > if a
> > > > > > > > > > > user
> > > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > > connector/config?migrate=latest",
> > the
> > > > > > latest
> > > > > > > > > version
> > > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > > transform the existing config to
> > the
> > > > > newer
> > > > > > > > > version.
> > > > > > > > > > > This
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > adding a method like "boolean
> > > > > > migrate(Version
> > > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This is an enhancement we can think
> > of
> > > > > doing
> > > > > > in
> > > > > > > > > future.
> > > > > > > > > > > > > > Users can
> > > > > > > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > > > > a PUT call with the updated config
> > > which
> > > > > has
> > > > > > > the
> > > > > > > > > > > updated
> > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > number.
> > > > > > > > > > > > > > > > > > > > > The assisted mode could be handy as
> > the
> > > > > user
> > > > > > > does
> > > > > > > > > not
> > > > > > > > > > > need
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > know the
> > > > > > > > > > > > > > > > > > > > > config but beyond this it does not
> > seem
> > > > to
> > > > > > > justify
> > > > > > > > > its
> > > > > > > > > > > > > > existence.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 10:50 AM
> > Ashwin
> > > > > > > > > > > > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This is a really useful feature
> and
> > > > > thanks
> > > > > > > for
> > > > > > > > > > > initiating
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I had the following questions -
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on the
> > rejected
> > > > > > > > > alternatives ?
> > > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > config is versioned and has a
> > schema.
> > > > > Then
> > > > > > a
> > > > > > > > > single
> > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > > dependencies have not changed)
> can
> > > > handle
> > > > > > > > > multiple
> > > > > > > > > > > config
> > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2. Any plans to support assisted
> > > > > migration
> > > > > > > e.g
> > > > > > > > > if a
> > > > > > > > > > > user
> > > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > > connector/config?migrate=latest",
> > the
> > > > > > latest
> > > > > > > > > version
> > > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > > transform the existing config to
> > the
> > > > > newer
> > > > > > > > > version.
> > > > > > > > > > > This
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > adding a method like "boolean
> > > > > > migrate(Version
> > > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM
> > > > > Snehashis <
> > > > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I'd like to start a discussion
> > > thread
> > > > > on
> > > > > > > > > KIP-891:
> > > > > > > > > > > > > Running
> > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The KIP aims to add the ability
> > for
> > > > the
> > > > > > > connect
> > > > > > > > > > > runtime
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > > versions of a connector.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Please take a look and let me
> > know
> > > > what
> > > > > > you
> > > > > > > > > think.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Snehashis <sn...@gmail.com>.
Hi Greg,
Thanks for taking a look at this, to conclude on the two points above.
1. I'm okay with the status quo of leaving the dependency management of
plugins to systems outside of the connect runtime as it is now. Given that
the dependencies are homogenous across a connect cluster, it should ensure
that task and connector versions are uniform across a deployment, after it
has stabilised post an upgrade operation. Heterogeneous dependencies in a
multi cluster deployment is highly discouraged and we should point out this
can lead to inconsistent behaviour with connectors and even validations.
2. Let's add the version information for both connector and tasks in the
connector status itself, I don't see another endpoint being required for
this.
I don't have any further points, so if you are okay we can put this to a
vote, once we add these two additions to the KIP (LMK if you want me to
take that up).
Regards
Snehashis
On Tue, Jul 2, 2024 at 12:08 AM Greg Harris <gr...@aiven.io.invalid>
wrote:
> Hey Snehashis,
>
> Sorry for the late reply, and thanks for helping close out the discussion.
>
> > Note that if my assumptions are correct then
> > this can happen with the existing framework as well, or is there some
> > safeguard from this happening?
>
> Currently, if a cluster has a heterogeneous plugin installation, each
> worker may have a different definition of "latest". If you call to
> validate a configuration, it will validate against whatever latest version
> is present locally.
> With this KIP, that behavior will continue when the version isn't
> specified, when the version is a soft requirement, or when the version is a
> range. For users that do not want to tolerate their tasks having
> heterogeneous versions, the single-version hard requirements are there, but
> come at the cost of micromanagement of upgrades.
> Perhaps there is some room in the future for a connector configured with a
> range to "pin" the version that all of its tasks should use. Or
> version-aware-scheduling can schedule connectors and tasks to workers that
> will resolve the same versions. The important primitive is being able to
> enforce which version is being used, everything else looks more like a
> convenience feature to me.
>
> > So far, we could have pointed to the
> > misconfigured cluster configuration and somewhat differ this problem to
> > something outside of connect runtime.
>
> I think this will still be the case. Even with this feature, we will still
> recommend homogeneous clusters that have the same set of plugins installed
> on every worker. Version-aware scheduling is a rejected alternative, and
> your example for performing validation requests on an arbitrary worker is a
> great example of a complication for heterogeneous clusters that we're not
> ready to solve.
>
> > We can introduce
> > a new path under this for version (/connector/connector-name/version),
> but
> > perhaps adding this as part of the status is a valid alternative.
>
> I think either is fine. Adding a new field to the existing endpoints is
> fine, and shouldn't require special compatibility such as a query
> parameter.
>
> > Also, to go further I think
> > version information for tasks could also be available,
>
> I completely agree. The version of each connector and each task should be
> visible individually, since they may be different during a rolling upgrade.
>
> Thanks!
> Greg
>
> On Tue, Jun 18, 2024 at 5:18 AM Snehashis <sn...@gmail.com>
> wrote:
>
> > Hi Greg, Chris
> >
> > Thanks for the in-depth discussion, I have a couple of discussion points
> > and would like your thoughts on this.
> >
> > 1) One concern I have with the new addition of 'soft' and 'hard' version
> > requirements is that there could be a mismatch in the plugin version that
> > two different tasks are running, if a soft requirement is provided and
> the
> > nodes a multi cluster deployment are not in sync w.r.t the plugin
> versions
> > that they are configured with. Note that if my assumptions are correct
> then
> > this can happen with the existing framework as well, or is there some
> > safeguard from this happening? So far, we could have pointed to the
> > misconfigured cluster configuration and somewhat differ this problem to
> > something outside of connect runtime. With this feature in place perhaps
> > the expectation is more on connect to not be running with such
> > inconsistency, especially if a connector version is specified. This is
> also
> > a problem with validation if different cluster have different
> > configurations, as IIRC validations are local to the worker which
> receives
> > the rest call for validate. So, we might be validating with a certain
> > version which is different from the one that will be used to create
> > connector and tasks. Again, this is likely how the current state is, but
> > perhaps such inconsistencies warrant a deeper look with the addition of
> > this feature. The problems associated with them can be somewhat insidious
> > and hard to diagnose.
> >
> > 2) There was some discussion on the need for a new REST endpoint to
> provide
> > information on the versions of running connectors, and I think adding
> this
> > information via REST is a valuable addition. The way I see it the version
> > is an intrinsic property of an instance of a running connector and hence
> > this should be part of the set of APIs under /connector/<connector-name>
> > (also the /connectors API should also have this information as it is an
> > amalgamation of all the individual connector information). We can
> introduce
> > a new path under this for version (/connector/connector-name/version),
> but
> > perhaps adding this as part of the status is a valid alternative. This is
> > mentioned as a rejected alternative right now. Also, to go further I
> think
> > version information for tasks could also be available, especially if we
> > choose to not address the pitfalls discussed in my point 1), this will
> > at-least provide admins a quick and easy way to determine if such and
> > inconsistent state exist in any of the connectors.
> >
> > Thanks again for reviving my original KIP and working to improve it.
> > Looking forward to your thoughts on the points mentioned above.
> > Regards
> > Snehashis
> >
> >
> > On Wed, May 29, 2024 at 9:59 PM Chris Egerton <ch...@aiven.io.invalid>
> > wrote:
> >
> > > Hi Greg,
> > >
> > > First, an apology! I mistakenly assumed that each plugin appeared only
> > once
> > > in the responses from GET /connector-plugins?connectorsOnly=false.
> Thank
> > > you for correcting me and pointing out that all versions of each plugin
> > > appear in that response, which does indeed satisfy my desire for users
> to
> > > discover this information in at most two REST requests (and in fact,
> does
> > > it in only one)!
> > >
> > > And secondly, with the revelation about recommenders, I agree that it's
> > > best to leave the "version" property out of the lists of properties
> > > returned from the GET /connector-plugins/<plugin>/config endpoint.
> > >
> > > With those two points settled, I think the only unresolved item is the
> > > small change to version parsing added to the KIP (where raw version
> > numbers
> > > are treated as an exact match, instead of a best-effort match with a
> > > fallback on the default version). If the KIP is updated with that then
> > I'd
> > > be ready to vote on it.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Wed, May 29, 2024 at 12:00 PM Greg Harris
> > <greg.harris@aiven.io.invalid
> > > >
> > > wrote:
> > >
> > > > Hey Chris,
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > > Won't it still only expose the
> > > > > latest version for each plugin, instead of the range of versions
> > > > available?
> > > >
> > > > Here is a snippet of the current output of the GET
> > > > /connector-plugins?connectorsOnly=false endpoint, after I installed
> two
> > > > versions of the debezium PostgresConnector:
> > > >
> > > > {
> > > > "class": "io.debezium.connector.postgresql.PostgresConnector",
> > > > "type": "source",
> > > > "version": "2.0.1.Final"
> > > > },
> > > > {
> > > > "class": "io.debezium.connector.postgresql.PostgresConnector",
> > > > "type": "source",
> > > > "version": "2.6.1.Final"
> > > > },
> > > >
> > > > I think this satisfies your requirement to learn about all plugins
> and
> > > all
> > > > versions in two or fewer REST calls.
> > > >
> > > > I tried to get an example of the output of `/config` by hardcoding
> the
> > > > Recommender, and realized that Recommenders aren't executed on the
> > > > `/config` endpoint at all: only during validation, when a
> configuration
> > > is
> > > > actually present.
> > > > And this led me to discover that the `/config` endpoint returns a
> > > > List<ConfigKeyInfo>, and ConfigKeyInfo does not contain a
> > > recommendedValues
> > > > field. The ConfigValue field is the object which contains
> > > > recommendedValues, and it is only generated during validation.
> > > > I think it's out of scope to start calling recommenders on empty
> > configs
> > > > that might throw exceptions, changing the existing REST entities, or
> > > > changing the core ConfigDef implementation.
> > > > Someone could add this functionality later, I don't think it's
> > necessary
> > > > here.
> > > >
> > > > Then the question is: should "version" without recommenders appear in
> > > > non-connector plugins? I think I'd rather be consistent with
> > "predicate"
> > > > and "negate" on release, and let a later improvement add them.
> > > >
> > > > Thanks,
> > > > Greg
> > > >
> > > > On Wed, May 29, 2024 at 8:06 AM Chris Egerton
> <chrise@aiven.io.invalid
> > >
> > > > wrote:
> > > >
> > > > > Hi Greg,
> > > > >
> > > > > I'm confused about the behavior for GET
> > > > > /connector-plugins?connectorsOnly=false. Won't it still only expose
> > the
> > > > > latest version for each plugin, instead of the range of versions
> > > > available?
> > > > >
> > > > > I'm hoping we can provide a flow where people need at most two REST
> > > calls
> > > > > to discover 1) the complete set of plugins available on the worker
> > > (which
> > > > > is already possible with the endpoint under discussion) and 2) the
> > set
> > > of
> > > > > versions available for a specific plugin on the worker (which
> doesn't
> > > > > appear to be possible, at least for some plugin types). This
> wouldn't
> > > > > require any out-of-band knowledge and would be valuable for
> connector
> > > > users
> > > > > (who may want to, for example, know what their options are when
> > > > considering
> > > > > a plugin upgrade) and cluster administrators (who could use it as a
> > > > sanity
> > > > > check for the setup of their Kafka Connect clusters without having
> to
> > > > pore
> > > > > over log files).
> > > > >
> > > > > As far as modifying the content of the GET
> > > > > /connector-plugins/<plugin>/config endpoint to include a "version"
> > > > property
> > > > > goes, I think your point about returning that property from
> requests
> > to
> > > > > that endpoint that include a version query parameter is salient,
> but
> > it
> > > > > also unfortunately applies to all types of plugin. I don't think it
> > > > should
> > > > > completely disqualify that option. I also wasn't imagining adding
> > > > anything
> > > > > besides that single property to that endpoint, so no new
> "predicate"
> > > > > property for SMTs, no new "negate" property for predicates, and no
> > new
> > > > > "type" property for either. I'm not necessarily opposed to adding
> > > those,
> > > > > but it can be done without being pulled into the scope of this KIP.
> > > > >
> > > > > So to summarize how I imagine people might use the REST API after
> > this
> > > > KIP:
> > > > >
> > > > > 1. Discover the set of plugins on the worker via GET
> > > > > /connector-plugins?connectorsOnly=false
> > > > > 2. For any of those plugins, discover the set of available versions
> > by
> > > > > examining the recommended values for the "version" property after
> > > hitting
> > > > > the GET GET /connector-plugins/<plugin>/config endpoint
> > > > >
> > > > > I don't think this is necessarily optimal since plucking the
> > "version"
> > > > > property out of the response from that endpoint might be a PITA for
> > ad
> > > > hoc
> > > > > queries via, e.g., curl, but I think it'd be enough for this KIP.
> > Still
> > > > > open to other options though, and if I've misunderstood anything in
> > the
> > > > > proposal, please let me know!
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Wed, May 22, 2024 at 4:05 PM Greg Harris
> > > <greg.harris@aiven.io.invalid
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hey Chris,
> > > > > >
> > > > > > Thanks for your comments, and I'm glad that it seems like we're
> > > > > > aligning on the vision here.
> > > > > >
> > > > > > > An
> > > > > > > alternative could be to change existing behavior to fail fast
> on
> > > any
> > > > > > > invalid default converter configuration instead of just for
> > invalid
> > > > > > > versions
> > > > > >
> > > > > > I suppose if this is landing in 4.0, we have the opportunity to
> > break
> > > > > > compatibility and strictly validate the worker class configs, and
> > > > > > could have a new consistent behavior instead of inconsistent but
> > > > > > backwards-compatible behavior.
> > > > > >
> > > > > > > no other part of the KIP requires this change.
> > > > > >
> > > > > > This is correct, and a compelling argument. I'm fine leaving the
> > > > > > strict worker validation off of this configuration to potentially
> > be
> > > > > > added later. If a KIP was raised to perform strict validation of
> > the
> > > > > > worker config, it would include the version config, and can
> address
> > > > > > backwards compatibility for both configs together.
> > > > > >
> > > > > > > RE exposing the version property in the
> > > > > > /connector-plugins/<plugin>/config
> > > > > > > endpoint, the behavior is inconsistent across plugin types.
> > > > > >
> > > > > > Yes it is, and that started in a bugfix, once people started
> using
> > > > > > this endpoint: https://issues.apache.org/jira/browse/KAFKA-14843
> > . I
> > > > > > wasn't planning on converging the behavior in this KIP, as I
> > > > > > considered it out-of-scope, and was going to follow the current
> > > > > > behavior. To summarize:
> > > > > >
> > > > > > GET /connector-plugins/<connector>/config will emit:
> > > > > > * connector.class (already implemented)
> > > > > > * connector.version (new)
> > > > > > * key.converter (already implemented)
> > > > > > * key.converter.version (new)
> > > > > > * value.converter (already implemented)
> > > > > > * value.converter.version (new)
> > > > > > * header.converter (already implemented)
> > > > > > * header.converter.version (new)
> > > > > > But will NOT emit:
> > > > > > * transforms.<alias>.type
> > > > > > * transforms.<alias>.version
> > > > > > * transforms.<alias>.predicate
> > > > > > * predicates.<alias>.type
> > > > > > * predicates.<alias>.version
> > > > > > * predicates.<alias>.negate
> > > > > >
> > > > > > GET /connector-plugins/<converter>/config will NOT emit:
> > > > > > * "" (there's not even a ".class" prefix!)
> > > > > > * version
> > > > > >
> > > > > > GET /connector-plugins/<transform>/config will NOT emit:
> > > > > > * type
> > > > > > * version
> > > > > > * predicate
> > > > > >
> > > > > > GET /connector-plugins/<predicate>/config will NOT emit:
> > > > > > * type
> > > > > > * version
> > > > > > * negate
> > > > > >
> > > > > > Do you want the converter, transform, and predicate endpoints
> > > changed?
> > > > > > Do you want just "version", or do you want all of the prefixed
> > > configs
> > > > > > including "type", "predicate" and "negate"? How would you want to
> > > > > > handle the converters?
> > > > > > And when I say the configs are "not part of the plugin config
> > itself"
> > > > > > I mean that saying that GET
> > > > > > /connector-plugins/Flatten$Key/config?version=3.8.0 has a
> "version"
> > > > > > config that must be "3.8.0" is a little bit nonsense, as the
> > version
> > > > > > is already specified.
> > > > > >
> > > > > > > IMO
> > > > > > > it's worth including this information somewhere directly
> > accessible
> > > > > > without
> > > > > > > having to provide a full connector config. FWIW I'd be fine
> with
> > > GET
> > > > > > > /connector-plugins/<plugin>/versions as a first-class endpoint
> > > > > >
> > > > > > You don't have to provide a configuration to call GET
> > > > > > /connector-plugins?connectorsOnly=false , is that endpoint not
> > close
> > > > > > enough to what you have in mind? See also the Rejected
> Alternative
> > > > > > "Adding new REST API endpoints"
> > > > > >
> > > > > > If you're calling /connector-plugins/<transform>/config, you know
> > the
> > > > > > name of a plugin right? That either comes from out-of-band
> > knowledge,
> > > > > > validating a connector config, or calling GET
> > > > > > /connector-plugins?connectorsOnly=false.
> > > > > > * If you have out-of-band knowledge of plugin classes, perhaps
> you
> > > > > > have out-of-band knowledge of versions too.
> > > > > > * If you've just validated a connector config, there should be an
> > > > > > accompanying "version" field there with an accurate default value
> > and
> > > > > > recommenders.
> > > > > > * If you've called GET /connector-plugins?connectorsOnly=false,
> > that
> > > > > > endpoint includes version information.
> > > > > >
> > > > > > Thanks,
> > > > > > Greg
> > > > > >
> > > > > > On Wed, May 22, 2024 at 11:05 AM Chris Egerton
> > > <chrise@aiven.io.invalid
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Greg,
> > > > > > >
> > > > > > > Hope you had a nice weekend! Gonna try to keep things concise.
> > > > > > >
> > > > > > > Concluded points:
> > > > > > >
> > > > > > > RE version recommenders, I agree it's likely that programmatic
> > UIs
> > > > will
> > > > > > > already be able to handle dynamic configuration definitions,
> and
> > > the
> > > > > > detail
> > > > > > > about SMTs is a great point. I still anticipate some
> awkwardness
> > > with
> > > > > > > connector versions, though: if the latest version supports some
> > new
> > > > > > > properties, then a user switches to an earlier version, a UI
> may
> > > > > respond
> > > > > > by
> > > > > > > wiping values for these properties. I guess we can bite the
> > bullet,
> > > > > > though.
> > > > > > >
> > > > > > > RE double-dinging during preflight validation for invalid
> > > versions, I
> > > > > > like
> > > > > > > the analogy with login credentials. I'm convinced that the
> > proposal
> > > > in
> > > > > > the
> > > > > > > KIP is best 👍
> > > > > > >
> > > > > > > Continued points:
> > > > > > >
> > > > > > > RE failing on worker startup, sorry, I should be clearer: there
> > is
> > > no
> > > > > > _new_
> > > > > > > justification for it that doesn't also apply to existing
> > behavior.
> > > We
> > > > > > > shouldn't diverge from existing behavior solely for this new
> > case.
> > > An
> > > > > > > alternative could be to change existing behavior to fail fast
> on
> > > any
> > > > > > > invalid default converter configuration instead of just for
> > invalid
> > > > > > > versions, but I'd vote to just stick to existing behavior and
> not
> > > > > > > complicate things, especially since no other part of the KIP
> > > requires
> > > > > > this
> > > > > > > change.
> > > > > > >
> > > > > > > RE exposing the version property in the
> > > > > > /connector-plugins/<plugin>/config
> > > > > > > endpoint, the behavior is inconsistent across plugin types.
> > Hitting
> > > > the
> > > > > > > endpoint for the FileStreamSinkConnector on version 3.7.0
> yields
> > a
> > > > > > response
> > > > > > > that includes, among other things, the "topics",
> "topics.regex",
> > > and
> > > > > > > "errors.tolerance" properties. I see that we don't do this
> > > everywhere
> > > > > > (the
> > > > > > > examples you cite for SMT and converter properties are
> accurate),
> > > but
> > > > > IMO
> > > > > > > it's worth including this information somewhere directly
> > accessible
> > > > > > without
> > > > > > > having to provide a full connector config. FWIW I'd be fine
> with
> > > GET
> > > > > > > /connector-plugins/<plugin>/versions as a first-class endpoint
> > > either
> > > > > > > instead of or in addition to adding recommended values for all
> > > plugin
> > > > > > > versions.
> > > > > > >
> > > > > > > Thanks for your continued work on this KIP, and with the
> progress
> > > > we're
> > > > > > > making I'm optimistic about its chances of appearing in 4.0.0.
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Wed, May 15, 2024 at 1:22 PM Greg Harris
> > > > > <greg.harris@aiven.io.invalid
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey Chris,
> > > > > > > >
> > > > > > > > Thanks for your quick follow up.
> > > > > > > >
> > > > > > > > > But this risk is already present with
> > > > > > > > > existing error cases, and I don't see anything that
> justifies
> > > > > > changing
> > > > > > > > > existing behavior with an invalid converter class, or
> > diverging
> > > > > from
> > > > > > it
> > > > > > > > in
> > > > > > > > > the case of invalid converter versions.
> > > > > > > >
> > > > > > > > The justification is to fail-fast, and prevent REST API users
> > > from
> > > > > > > > receiving errors from bad configs that they didn't write, or
> > > maybe
> > > > > > > > don't even know apply to them.
> > > > > > > > Up until recently errors in these configurations surfaced as
> > > > failures
> > > > > > > > to create the connector, or failures to start, and you made
> > them
> > > > > > > > fail-fast during validation. I think this change is in the
> same
> > > > > > > > spirit, pulling the validation further forward and not
> letting
> > > > errors
> > > > > > > > lie dormant.
> > > > > > > >
> > > > > > > > And to call back to your original concern about interrupting
> > > > > > > > connectors that explicitly provide these configurations and
> > don't
> > > > use
> > > > > > > > the worker configs: I expect that operators with a majority
> of
> > > > these
> > > > > > > > sorts of clients aren't going to be setting the worker
> .version
> > > > > > > > properties, because it would have no effect on the majority
> of
> > > > their
> > > > > > > > connectors. They would be able to rely on
> > backwards-compatibility
> > > > and
> > > > > > > > continue to ignore the class properties.
> > > > > > > >
> > > > > > > > > CLI
> > > > > > > > > and programmatic UI developers will want to develop their
> own
> > > > > tooling
> > > > > > > > > layers.
> > > > > > > >
> > > > > > > > This is a very compelling argument. Snehashis do you want to
> > > figure
> > > > > > > > out a REST API design for this use-case?
> > > > > > > >
> > > > > > > > > Regarding the GET /connector-plugins/<plugin>/config
> > endpoint,
> > > I
> > > > > was
> > > > > > > > > thinking about the response for non-connector plugins,
> e.g.,
> > > > > > > > > GET /connector-plugins/RegexRouter/config. Would a
> "version"
> > > > > property
> > > > > > > > > appear with recommended values?
> > > > > > > >
> > > > > > > > I intended for the ".version" property to be like other
> > framework
> > > > > > > > configs (".class", ".type", ".predicate", ".negate") where
> they
> > > are
> > > > > > > > inside the plugin namespace, but not part of the plugin
> config
> > > > > itself.
> > > > > > > > Perhaps we can deviate from those configs because it would
> aid
> > in
> > > > > > > > discovering other valid `GET
> > > > > > > > /connector-plugins/<plugin>/config?version=` calls, without
> > > calling
> > > > > > > > `GET /connector-plugins?connectorsOnly=false`.
> > > > > > > > I don't really feel strongly either way.
> > > > > > > >
> > > > > > > > > if a user changes the
> > > > > > > > > connector version in, e.g., a dropdown menu, then the UI
> > either
> > > > has
> > > > > > to
> > > > > > > > > re-fetch the ConfigDef for the new version, or risk
> operating
> > > on
> > > > > > stale
> > > > > > > > > information
> > > > > > > >
> > > > > > > > This is a really interesting situation, thanks for finding
> > that!
> > > > This
> > > > > > > > is already a footgun with transformations and predicates;
> Once
> > > you
> > > > > > > > fill out a transformation class (which includes a
> recommender)
> > > you
> > > > > > > > will then see all of the transformations configurations.
> > > > > > > > I think this means that UI developers should have already
> > > developed
> > > > > > > > the infrastructure for handling dynamic recommenders, or else
> > > > they've
> > > > > > > > had this bug since KIP-66 in 2017. It may require some manual
> > > > > > > > attention to roll-out support for the ".version" properties
> > > > > > > > specifically, but I don't think that should prevent us from
> > > > providing
> > > > > > > > this information in a natural place.
> > > > > > > >
> > > > > > > > > but will it be possible to configure
> > > > > > > > > a connector to use two instances of the same transform or
> > > > > predicate,
> > > > > > but
> > > > > > > > > with different versions for each?
> > > > > > > >
> > > > > > > > Yes, and this is included in the example in the KIP,
> > > > > > > > `transforms.flatten-latest` and `transforms.flatten-old`.
> This
> > > will
> > > > > > > > work in the natural way, with the two instances having a
> > > different
> > > > > > > > version if configured for that. If a user wants to pin the
> same
> > > > > > > > version of a plugin for every instance, they will need to
> > provide
> > > > the
> > > > > > > > version config multiple times and keep them in-sync
> themselves.
> > > > > > > >
> > > > > > > > > I would have expected the error
> > > > > > > > > to only be attributed to the version property, and for the
> > > class
> > > > > > property
> > > > > > > > > to be reported as valid.
> > > > > > > >
> > > > > > > > I considered this, and then went back and changed it. To
> quote
> > > > > someone
> > > > > > > > else, this is to catch "misspelling cat as dog". For example
> a
> > > user
> > > > > > > > types DogConverter and meant to type CatConverter, and then
> > > types a
> > > > > > > > version which is valid for CatConverter, conceptually the
> error
> > > is
> > > > in
> > > > > > > > the class name and not the version.
> > > > > > > > That's a very contrived scenario, but I think similar
> arguments
> > > are
> > > > > > > > used for attributing validation errors to
> > > endpoints/urls/hostnames
> > > > > > > > when the associated credentials are unable to log into the
> > remote
> > > > > > > > system. Did the user provide the correct testing credentials,
> > but
> > > > > > > > accidentally typed the production endpoint? Even if the
> > > production
> > > > > > > > endpoint looks valid (it's a real hostname that is reachable)
> > the
> > > > > > > > conceptual error is still in the hostname and should have the
> > > error
> > > > > > > > attributed to it to draw the user's attention.
> > > > > > > > If that's not convincing, I think the alternative of only
> > > > attributing
> > > > > > > > version errors to the version property is also acceptable.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Greg
> > > > > > > >
> > > > > > > > On Wed, May 15, 2024 at 8:06 AM Chris Egerton
> > > > > <chrise@aiven.io.invalid
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Greg,
> > > > > > > > >
> > > > > > > > > Thanks for your responses! Continuations of existing
> > > discussions:
> > > > > > > > >
> > > > > > > > > Regarding crashing the worker on startup--yes, there is
> also
> > a
> > > > risk
> > > > > > to
> > > > > > > > > allowing it to join the cluster. But this risk is already
> > > present
> > > > > > with
> > > > > > > > > existing error cases, and I don't see anything that
> justifies
> > > > > > changing
> > > > > > > > > existing behavior with an invalid converter class, or
> > diverging
> > > > > from
> > > > > > it
> > > > > > > > in
> > > > > > > > > the case of invalid converter versions. I think we should
> > keep
> > > > this
> > > > > > > > simple
> > > > > > > > > and not do anything different for worker startup than we do
> > > > > already.
> > > > > > > > >
> > > > > > > > > As far as REST API vs. metrics go--I think you're right
> that
> > > the
> > > > > > original
> > > > > > > > > version metrics were added as a "monitoring" detail.
> However,
> > > > this
> > > > > > was
> > > > > > > > back
> > > > > > > > > when plugin versions were managed solely by cluster
> > > > administrators.
> > > > > > With
> > > > > > > > > this KIP, connector users will be able to manage plugin
> > > versions,
> > > > > > and CLI
> > > > > > > > > and programmatic UI developers will want to develop their
> own
> > > > > tooling
> > > > > > > > > layers. I think focusing on the REST API as the primary
> > > interface
> > > > > for
> > > > > > > > this
> > > > > > > > > KIP would be best for these users.
> > > > > > > > >
> > > > > > > > > (All that said, I don't object to the metrics that are
> > proposed
> > > > in
> > > > > > the
> > > > > > > > KIP;
> > > > > > > > > I just think they make more sense in addition to new REST
> API
> > > > > > > > > functionality, as opposed to instead of it.)
> > > > > > > > >
> > > > > > > > > Regarding the GET /connector-plugins/<plugin>/config
> > endpoint,
> > > I
> > > > > was
> > > > > > > > > thinking about the response for non-connector plugins,
> e.g.,
> > > > > > > > > GET /connector-plugins/RegexRouter/config. Would a
> "version"
> > > > > property
> > > > > > > > > appear with recommended values?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > And new thoughts:
> > > > > > > > >
> > > > > > > > > 1) Regarding the recommended values for
> "connector.version",
> > > this
> > > > > > might
> > > > > > > > be
> > > > > > > > > confusing since there could be differences between the
> > > ConfigDefs
> > > > > > for the
> > > > > > > > > latest version of the connector and prior versions. It also
> > > makes
> > > > > the
> > > > > > > > flow
> > > > > > > > > a bit awkward for programmatic UI developers: if a user
> > changes
> > > > the
> > > > > > > > > connector version in, e.g., a dropdown menu, then the UI
> > either
> > > > has
> > > > > > to
> > > > > > > > > re-fetch the ConfigDef for the new version, or risk
> operating
> > > on
> > > > > > stale
> > > > > > > > > information. I'm starting to doubt that exposing the range
> of
> > > > > > available
> > > > > > > > > versions via recommended values is the best way to proceed,
> > > > instead
> > > > > > of a
> > > > > > > > > more explicit approach like GET
> > > > > > /connector-plugins/<plugin>/versions, or
> > > > > > > > > the "Adding new REST API endpoints" rejected alternative.
> > > > > > > > >
> > > > > > > > > 2) I know that this is a bit heinous, but will it be
> possible
> > > to
> > > > > > > > configure
> > > > > > > > > a connector to use two instances of the same transform or
> > > > > predicate,
> > > > > > but
> > > > > > > > > with different versions for each? (I don't think this is
> > worth
> > > > > > > > significant
> > > > > > > > > design/implementation effort, so if it would inflate either
> > of
> > > > > those,
> > > > > > > > > please don't feel obligated to support it.)
> > > > > > > > >
> > > > > > > > > 3) Just out of curiosity, why double-ding during config
> > > > validation
> > > > > > if the
> > > > > > > > > version for a plugin class can't be found? I would have
> > > expected
> > > > > the
> > > > > > > > error
> > > > > > > > > to only be attributed to the version property, and for the
> > > class
> > > > > > property
> > > > > > > > > to be reported as valid.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Tue, May 14, 2024 at 1:11 PM Greg Harris
> > > > > > <greg.harris@aiven.io.invalid
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hey Chris,
> > > > > > > > > >
> > > > > > > > > > Thanks for your comments. I'm glad you like the
> > motivations,
> > > > > > Snehashis
> > > > > > > > > > wrote that part!
> > > > > > > > > >
> > > > > > > > > > > the configuration syntax for the most basic use case of
> > > > > > > > > > > specifying a single desired version is pretty
> > > > counterintuitive.
> > > > > > > > > >
> > > > > > > > > > I agree, and the "soft" requirement scheme is something I
> > > > wasn't
> > > > > > > > > > explicitly looking for, but would be inherited from the
> > > > library.
> > > > > > I'm
> > > > > > > > > > fine with eliminating the soft requirement semantics, and
> > > > having
> > > > > > > > > > "1.0.0" behave the same as "[1.0.0]". I'm less inclined
> to
> > > > > include
> > > > > > > > > > "range" in the property name, or have two properties.
> > > > > > > > > >
> > > > > > > > > > > Failing startup is drastic and has the potential to
> > disrupt
> > > > the
> > > > > > > > > > > availability of connectors that would otherwise be able
> > to
> > > > run
> > > > > > > > healthily
> > > > > > > > > > > because they were explicitly configured to use valid
> > > > converters
> > > > > > > > instead
> > > > > > > > > > of
> > > > > > > > > > > the worker defaults.
> > > > > > > > > >
> > > > > > > > > > I think this argument cuts both ways. If someone
> > > reconfigures a
> > > > > > worker
> > > > > > > > > > and adds an invalid ".version" string to the worker (or
> > > changes
> > > > > the
> > > > > > > > > > plugin.path to make it invalid), it would be permitted to
> > > enter
> > > > > the
> > > > > > > > > > group, and accept work assignments. If those work
> > assignments
> > > > > used
> > > > > > > > > > these configurations, a set of tasks could transition to
> > > FAILED
> > > > > and
> > > > > > > > > > not be able to recover, because they would be restarted
> > again
> > > > on
> > > > > > the
> > > > > > > > > > same worker.
> > > > > > > > > >
> > > > > > > > > > > Why are metrics utilized to report information about
> > plugin
> > > > > > versions
> > > > > > > > > > > utilized by connectors at runtime instead of publishing
> > > this
> > > > > > info in
> > > > > > > > the
> > > > > > > > > > > REST API
> > > > > > > > > >
> > > > > > > > > > I was following the existing pattern for exposing runtime
> > > > > versions
> > > > > > for
> > > > > > > > > > connectors, and it did seem like a "monitoring" feature.
> If
> > > > that
> > > > > > > > > > approach is flawed and should be phased out, I think it
> > would
> > > > be
> > > > > a
> > > > > > > > > > good idea to reconsider the REST API rejected
> alternative.
> > > > > > > > > > We would need some additional design work to spec out the
> > > REST
> > > > > API
> > > > > > > > > > interface, as I don't have anything in mind currently.
> > > > > > > > > >
> > > > > > > > > > > I'm unclear on whether or not it'll be possible to see
> > this
> > > > > > > > information
> > > > > > > > > > via the
> > > > > > > > > > > GET /connector-plugins/<plugin>/config endpoint
> > > > > > > > > >
> > > > > > > > > > There is room in the API to add recommenders for
> > > > "key.converter",
> > > > > > > > > > "value.converter", and "header.converter", but not for
> > > > transforms
> > > > > > and
> > > > > > > > > > predicates, as they include aliases that depend on an
> > actual
> > > > > > > > > > configuration. We could explicitly say we're going to do
> > > that,
> > > > or
> > > > > > do
> > > > > > > > > > whatever is convenient during the implementation phase,
> or
> > > > leave
> > > > > it
> > > > > > > > > > open to be improved later.
> > > > > > > > > > There will not be any recommenders for ".version"
> > properties
> > > in
> > > > > the
> > > > > > > > > > `/config` endpoint, because those recommenders are
> dynamic
> > > and
> > > > > > depend
> > > > > > > > > > on an actual configuration.
> > > > > > > > > >
> > > > > > > > > > 5) There are two relevant lines in the KIP: "If a
> .version
> > > > > property
> > > > > > > > > > contains a hard requirement, select the latest installed
> > > > version
> > > > > > which
> > > > > > > > > > satisfies the requirement." and "This configuration is
> > > > > re-evaluated
> > > > > > > > > > each time the connector or task are assigned to a new
> > > worker".
> > > > I
> > > > > > would
> > > > > > > > > > call this "eager" upgrade behavior, rather than a
> "sticky"
> > or
> > > > > > "lazy"
> > > > > > > > > > upgrade behavior.
> > > > > > > > > >
> > > > > > > > > > 6) Updated!
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Greg
> > > > > > > > > >
> > > > > > > > > > On Tue, May 14, 2024 at 9:14 AM Chris Egerton
> > > > > > <chrise@aiven.io.invalid
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > Thanks Greg for updating the KIP, and thanks Snehashis
> > for
> > > > > > starting
> > > > > > > > the
> > > > > > > > > > > work on this originally.
> > > > > > > > > > >
> > > > > > > > > > > The motivation section makes a pretty convincing case
> for
> > > > this
> > > > > > kind
> > > > > > > > of
> > > > > > > > > > > feature. My thoughts are mostly about specific details:
> > > > > > > > > > >
> > > > > > > > > > > 1) I like the support for version ranges (the example
> > > > > > demonstrating
> > > > > > > > how
> > > > > > > > > > to
> > > > > > > > > > > avoid KAFKA-10574 with the header converter was
> > > particularly
> > > > > > > > > > entertaining),
> > > > > > > > > > > but the configuration syntax for the most basic use
> case
> > of
> > > > > > > > specifying a
> > > > > > > > > > > single desired version is pretty counterintuitive.
> People
> > > may
> > > > > get
> > > > > > > > bitten
> > > > > > > > > > or
> > > > > > > > > > > at least frustrated if they put connector.version=3.8.0
> > in
> > > a
> > > > > > config
> > > > > > > > but
> > > > > > > > > > > then version 3.7.5 ends up running. I'd like it if we
> > could
> > > > > > either
> > > > > > > > > > > intentionally deviate from Maven ranges when a bare
> > version
> > > > is
> > > > > > > > present,
> > > > > > > > > > or
> > > > > > > > > > > separate things out into two properties: foo.version
> > would
> > > be
> > > > > the
> > > > > > > > single
> > > > > > > > > > > accepted version for the foo plugin, and
> > foo.version.range
> > > > > would
> > > > > > use
> > > > > > > > > > Maven
> > > > > > > > > > > range syntax. Open to other options too, just
> providing a
> > > > > couple
> > > > > > to
> > > > > > > > get
> > > > > > > > > > the
> > > > > > > > > > > ball rolling.
> > > > > > > > > > >
> > > > > > > > > > > 2) Although the current behavior for a worker with an
> > > invalid
> > > > > > > > > > > key/value/header converter class specified in its
> config
> > > file
> > > > > is
> > > > > > a
> > > > > > > > little
> > > > > > > > > > > strange (I was surprised to learn that it wouldn't fail
> > on
> > > > > > startup),
> > > > > > > > I
> > > > > > > > > > > don't see a good reason to deviate from this when an
> > > invalid
> > > > > > version
> > > > > > > > is
> > > > > > > > > > > specified. Failing startup is drastic and has the
> > potential
> > > > to
> > > > > > > > disrupt
> > > > > > > > > > the
> > > > > > > > > > > availability of connectors that would otherwise be able
> > to
> > > > run
> > > > > > > > healthily
> > > > > > > > > > > because they were explicitly configured to use valid
> > > > converters
> > > > > > > > instead
> > > > > > > > > > of
> > > > > > > > > > > the worker defaults.
> > > > > > > > > > >
> > > > > > > > > > > 3) Why are metrics utilized to report information about
> > > > plugin
> > > > > > > > versions
> > > > > > > > > > > utilized by connectors at runtime instead of publishing
> > > this
> > > > > > info in
> > > > > > > > the
> > > > > > > > > > > REST API? I saw that this was mentioned as a rejected
> > > > > > alternative,
> > > > > > > > but I
> > > > > > > > > > > didn't get a sense of why. It seems like the REST API
> > would
> > > > be
> > > > > > > > easier to
> > > > > > > > > > > access and more intuitive for most users than new
> > metrics.
> > > > > > > > > > >
> > > > > > > > > > > 4) In the "Validation" section it's stated that "Users
> > can
> > > > use
> > > > > > these
> > > > > > > > > > > recommenders to discover the valid plugin classes and
> > > > versions,
> > > > > > > > without
> > > > > > > > > > > requiring an earlier call to GET
> > > > > > > > > > /connector-plugins?connectorsOnly=false."
> > > > > > > > > > > I really like the creativity and simplicity of reusing
> > the
> > > > > > > > recommender
> > > > > > > > > > > mechanism to expose available versions for plugins via
> > the
> > > > REST
> > > > > > API.
> > > > > > > > I'm
> > > > > > > > > > > unclear on whether or not it'll be possible to see this
> > > > > > information
> > > > > > > > via
> > > > > > > > > > the
> > > > > > > > > > > GET /connector-plugins/<plugin>/config endpoint,
> though.
> > > It'd
> > > > > be
> > > > > > > > great if
> > > > > > > > > > > this were supported, since we learned in KIP-769 [1]
> that
> > > > > people
> > > > > > > > really
> > > > > > > > > > > want to be able to see configuration options for
> > connectors
> > > > and
> > > > > > their
> > > > > > > > > > > plugins via some kind of GET endpoint without having to
> > > > > provide a
> > > > > > > > > > complete
> > > > > > > > > > > connector config for validation.
> > > > > > > > > > >
> > > > > > > > > > > 5) In the Maven version range docs, it's stated that
> > "Maven
> > > > > > picks the
> > > > > > > > > > > highest version of each project that satisfies all the
> > hard
> > > > > > > > requirements
> > > > > > > > > > of
> > > > > > > > > > > the dependencies on that project." I'm guessing this
> > > behavior
> > > > > > will be
> > > > > > > > > > > retained for Connect; i.e., the highest-possible
> version
> > of
> > > > > each
> > > > > > > > plugin
> > > > > > > > > > > that satisfies the user-specified version constraints
> > will
> > > be
> > > > > > run?
> > > > > > > > (An
> > > > > > > > > > > alternative approach could be to have some kind of
> > "sticky"
> > > > > logic
> > > > > > > > that
> > > > > > > > > > only
> > > > > > > > > > > restarts connectors/tasks when their currently-used
> > version
> > > > > > becomes
> > > > > > > > > > > incompatible with the configured constraints.)
> > > > > > > > > > >
> > > > > > > > > > > 6) (Nit) It'd be nice to add a link to the TestPlugins
> > > class
> > > > or
> > > > > > > > somewhere
> > > > > > > > > > > in its neighborhood to the testing plan; unfamiliar
> > readers
> > > > > > probably
> > > > > > > > > > won't
> > > > > > > > > > > get much out of what's there right now.
> > > > > > > > > > >
> > > > > > > > > > > [1] -
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris
> > > > > > > > > > >
> > > > > > > > > > > On Mon, May 13, 2024 at 2:01 PM Snehashis <
> > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > >
> > > > > > > > > > > > That is much appreciated. No complaints on the
> > additional
> > > > > > scope, I
> > > > > > > > will
> > > > > > > > > > > > make some time out to work on this once we have
> > approval.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > > Snehashis
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hey Snehashis,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm glad to hear you're still interested in this
> KIP!
> > > > > > > > > > > > > I'm happy to let you drive this, and I apologize
> for
> > > > > > increasing
> > > > > > > > the
> > > > > > > > > > > > > scope of work so drastically. To make up for that,
> > I'll
> > > > > > > > volunteer to
> > > > > > > > > > > > > be the primary PR reviewer to help get this done
> > > quickly
> > > > > once
> > > > > > > > the KIP
> > > > > > > > > > > > > is approved.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Greg
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, May 10, 2024 at 3:51 AM Snehashis <
> > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the follow up to my original KIP, I am
> > in
> > > > > > favour of
> > > > > > > > the
> > > > > > > > > > > > > > additions made to expand its scope, the addition
> of
> > > > range
> > > > > > > > versions
> > > > > > > > > > > > > > specifically make a lot of sense.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Apologies if I have not publicly worked on this
> KIP
> > > > for a
> > > > > > long
> > > > > > > > > > time.
> > > > > > > > > > > > The
> > > > > > > > > > > > > > original work was done when the move to service
> > > loading
> > > > > > was in
> > > > > > > > > > > > discussion
> > > > > > > > > > > > > > and I wanted to loop back to this only after that
> > > work
> > > > > was
> > > > > > > > > > completed.
> > > > > > > > > > > > > Post
> > > > > > > > > > > > > > its conclusion, I have not been able to take this
> > up
> > > > due
> > > > > to
> > > > > > > > other
> > > > > > > > > > > > > > priorities. If it's okay with you, I would still
> > like
> > > > to
> > > > > > get
> > > > > > > > this
> > > > > > > > > > > > > > implemented myself, including the additional
> scope.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks and regards
> > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> > > > > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'd like to reboot the discussion on KIP-891:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I've made some changes, most notably:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. Specifying versions for all plugins in
> > Connector
> > > > > > configs
> > > > > > > > > > > > > > > (converters, header converters, transforms, and
> > > > > > predicates)
> > > > > > > > not
> > > > > > > > > > just
> > > > > > > > > > > > > > > connectors & tasks
> > > > > > > > > > > > > > > 2. Specifying a range of versions instead of an
> > > exact
> > > > > > match
> > > > > > > > > > > > > > > 3. New metrics to observe what versions are
> > in-use
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks to Snehashis for the original KIP idea!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <
> > > > > > > > > > greg.harris@aiven.io>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Snehashis,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thank you for the KIP! This is something I've
> > > > wanted
> > > > > > for a
> > > > > > > > long
> > > > > > > > > > > > time.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I know the discussion has gone cold, are you
> > > still
> > > > > > > > interested
> > > > > > > > > > in
> > > > > > > > > > > > > > > > pursuing this feature? I'll make time to
> review
> > > the
> > > > > > KIP if
> > > > > > > > you
> > > > > > > > > > are
> > > > > > > > > > > > > > > > still accepting comments.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis <
> > > > > > > > > > > > snehashisp1995@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for the points Sagar.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1) Should we update the GET /connectors
> > > > endpoint
> > > > > to
> > > > > > > > > > include the
> > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > the plugin that is running? It could be
> > > useful
> > > > to
> > > > > > > > figure
> > > > > > > > > > out
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > the plugin or I am assuming it gets
> > returned
> > > by
> > > > > the
> > > > > > > > > > expand=info
> > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think this is good to have and possible
> > > future
> > > > > > > > > > enhancement. The
> > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > info will be present in the config of the
> > > > connector
> > > > > > if
> > > > > > > > the
> > > > > > > > > > user
> > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > specified the version. Otherwise it is the
> > > latest
> > > > > > version
> > > > > > > > > > which
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > can find out from the connector-plugin
> > > endpoint.
> > > > > The
> > > > > > > > > > information
> > > > > > > > > > > > > can be
> > > > > > > > > > > > > > > > > introduced to the response of the GET
> > > /connectors
> > > > > > > > endpoint
> > > > > > > > > > > > itself,
> > > > > > > > > > > > > > > however
> > > > > > > > > > > > > > > > > the most ideal way of doing this would be
> to
> > > get
> > > > > the
> > > > > > > > > > currently
> > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > instance of the connector and get the
> version
> > > > > > directly
> > > > > > > > from
> > > > > > > > > > > > there.
> > > > > > > > > > > > > > > This is
> > > > > > > > > > > > > > > > > slightly tricky as the connector could be
> > > running
> > > > > in
> > > > > > a
> > > > > > > > > > different
> > > > > > > > > > > > > node.
> > > > > > > > > > > > > > > > > One way to do this would be to persist the
> > > > version
> > > > > > > > > > information in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > status backing store during instantiation
> of
> > > the
> > > > > > > > connector.
> > > > > > > > > > It
> > > > > > > > > > > > > requires
> > > > > > > > > > > > > > > > > some more thought and since the version is
> > part
> > > > of
> > > > > > the
> > > > > > > > > > configs if
> > > > > > > > > > > > > > > provided
> > > > > > > > > > > > > > > > > and evident otherwise, I have not included
> it
> > > in
> > > > > this
> > > > > > > > KIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2) I am not aware of this and hence
> asking,
> > > > can 2
> > > > > > > > > > connectors
> > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > versions have the same name? Does the
> > plugin
> > > > > > isolation
> > > > > > > > > > allow
> > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > could have a bearing when using the
> > lifecycle
> > > > > > > > endpoints for
> > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > All connectors in a cluster need to have
> > > uniquire
> > > > > > > > connector
> > > > > > > > > > names
> > > > > > > > > > > > > > > > > regardless of what version of the plugin
> the
> > > > > > connector is
> > > > > > > > > > running
> > > > > > > > > > > > > > > > > underneath. This is something enforced by
> the
> > > > > connect
> > > > > > > > runtime
> > > > > > > > > > > > > itself.
> > > > > > > > > > > > > > > All
> > > > > > > > > > > > > > > > > connect CRUD operations are keyed on the
> > > > connector
> > > > > > name
> > > > > > > > so
> > > > > > > > > > there
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > be an issue.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <
> > > > > > > > > > sagarmeansocean@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hey Snehashsih,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for the KIP. It looks like a very
> > > useful
> > > > > > > > feature.
> > > > > > > > > > Couple
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > small-ish points, let me know what you
> > think:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1) Should we update the GET /connectors
> > > > endpoint
> > > > > to
> > > > > > > > > > include the
> > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > the plugin that is running? It could be
> > > useful
> > > > to
> > > > > > > > figure
> > > > > > > > > > out
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > the plugin or I am assuming it gets
> > returned
> > > by
> > > > > the
> > > > > > > > > > expand=info
> > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > > 2) I am not aware of this and hence
> asking,
> > > > can 2
> > > > > > > > > > connectors
> > > > > > > > > > > > with
> > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > versions have the same name? Does the
> > plugin
> > > > > > isolation
> > > > > > > > > > allow
> > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > could have a bearing when using the
> > lifecycle
> > > > > > > > endpoints for
> > > > > > > > > > > > > > > connectors like
> > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > Sagar.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin
> > > > > > > > > > > > > <apankaj@confluent.io.invalid
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > IIUC (please correct me if I am wrong
> > > > here),
> > > > > > what
> > > > > > > > you
> > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > a versioning scheme for a connector
> > config
> > > > for
> > > > > > the
> > > > > > > > same
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > different versions of a connector
> > plugin).
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Sorry for not being more precise in my
> > > > wording
> > > > > > - I
> > > > > > > > meant
> > > > > > > > > > > > > > > registering
> > > > > > > > > > > > > > > > > > > versions of schema for connector
> config.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Let's take the example of a fictional
> > > > connector
> > > > > > which
> > > > > > > > > > uses a
> > > > > > > > > > > > > > > fictional
> > > > > > > > > > > > > > > > > > AWS
> > > > > > > > > > > > > > > > > > > service.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Fictional Connector Config schema
> > > version:2.0
> > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Fictional Connector config schema
> > > version:3.0
> > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > "iam_role": {
> > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > "iam_role"
> > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The connector which supports Fictional
> > > config
> > > > > > schema
> > > > > > > > 2.0
> > > > > > > > > > > > will
> > > > > > > > > > > > > > > validate
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > access key and secret key.
> > > > > > > > > > > > > > > > > > > Whereas a connector which supports
> config
> > > > with
> > > > > > schema
> > > > > > > > > > version
> > > > > > > > > > > > > 3.0
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > > validate the IAM role.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This is the alternative which I wanted
> to
> > > > > > suggest.
> > > > > > > > Each
> > > > > > > > > > > > plugin
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > register the schema versions of
> connector
> > > > > config
> > > > > > > > which it
> > > > > > > > > > > > > supports.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The plugin paths may be optionally
> > > different
> > > > > > i.e we
> > > > > > > > > > don't
> > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > > mandatorily add a new plugin path to
> > > support
> > > > a
> > > > > > new
> > > > > > > > schema
> > > > > > > > > > > > > version.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:47 PM
> > Snehashis
> > > <
> > > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on the
> rejected
> > > > > > > > alternatives ?
> > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > config is versioned and has a
> schema.
> > > > Then
> > > > > a
> > > > > > > > single
> > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > dependencies have not changed) can
> > > handle
> > > > > > > > multiple
> > > > > > > > > > config
> > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > IIUC (please correct me if I am wrong
> > > > here),
> > > > > > what
> > > > > > > > you
> > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > a versioning scheme for a connector
> > > config
> > > > > for
> > > > > > the
> > > > > > > > same
> > > > > > > > > > > > > > > connector (and
> > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > different versions of a connector
> > > plugin).
> > > > > > That is
> > > > > > > > a
> > > > > > > > > > > > somewhat
> > > > > > > > > > > > > > > > > > tangential
> > > > > > > > > > > > > > > > > > > > problem. While it is definitely a
> > useful
> > > > > > feature to
> > > > > > > > > > have,
> > > > > > > > > > > > > like a
> > > > > > > > > > > > > > > log to
> > > > > > > > > > > > > > > > > > > > check what changes were made over
> time
> > to
> > > > the
> > > > > > > > config
> > > > > > > > > > which
> > > > > > > > > > > > > might
> > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > easier to do rollbacks, it is not the
> > > focus
> > > > > > here.
> > > > > > > > Here
> > > > > > > > > > by
> > > > > > > > > > > > > > > version we
> > > > > > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > > > > to say what underlying version of the
> > > > plugin
> > > > > > > > should the
> > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > of the connector use. Perhaps it is
> > > better
> > > > to
> > > > > > > > change
> > > > > > > > > > the
> > > > > > > > > > > > > name of
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > parameter from connector.version to
> > > > > > > > > > > > connector.plugin.version
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > plugin.version if it was confusing.
> > wdyt?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2. Any plans to support assisted
> > > > migration
> > > > > > e.g
> > > > > > > > if a
> > > > > > > > > > user
> > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > connector/config?migrate=latest",
> the
> > > > > latest
> > > > > > > > version
> > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > transform the existing config to
> the
> > > > newer
> > > > > > > > version.
> > > > > > > > > > This
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > adding a method like "boolean
> > > > > migrate(Version
> > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This is an enhancement we can think
> of
> > > > doing
> > > > > in
> > > > > > > > future.
> > > > > > > > > > > > > Users can
> > > > > > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > > > a PUT call with the updated config
> > which
> > > > has
> > > > > > the
> > > > > > > > > > updated
> > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > number.
> > > > > > > > > > > > > > > > > > > > The assisted mode could be handy as
> the
> > > > user
> > > > > > does
> > > > > > > > not
> > > > > > > > > > need
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > know the
> > > > > > > > > > > > > > > > > > > > config but beyond this it does not
> seem
> > > to
> > > > > > justify
> > > > > > > > its
> > > > > > > > > > > > > existence.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 10:50 AM
> Ashwin
> > > > > > > > > > > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This is a really useful feature and
> > > > thanks
> > > > > > for
> > > > > > > > > > initiating
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I had the following questions -
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on the
> rejected
> > > > > > > > alternatives ?
> > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > config is versioned and has a
> schema.
> > > > Then
> > > > > a
> > > > > > > > single
> > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > dependencies have not changed) can
> > > handle
> > > > > > > > multiple
> > > > > > > > > > config
> > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2. Any plans to support assisted
> > > > migration
> > > > > > e.g
> > > > > > > > if a
> > > > > > > > > > user
> > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > connector/config?migrate=latest",
> the
> > > > > latest
> > > > > > > > version
> > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > transform the existing config to
> the
> > > > newer
> > > > > > > > version.
> > > > > > > > > > This
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > adding a method like "boolean
> > > > > migrate(Version
> > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM
> > > > Snehashis <
> > > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I'd like to start a discussion
> > thread
> > > > on
> > > > > > > > KIP-891:
> > > > > > > > > > > > Running
> > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The KIP aims to add the ability
> for
> > > the
> > > > > > connect
> > > > > > > > > > runtime
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > versions of a connector.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Please take a look and let me
> know
> > > what
> > > > > you
> > > > > > > > think.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Hey Snehashis,
Sorry for the late reply, and thanks for helping close out the discussion.
> Note that if my assumptions are correct then
> this can happen with the existing framework as well, or is there some
> safeguard from this happening?
Currently, if a cluster has a heterogeneous plugin installation, each
worker may have a different definition of "latest". If you call to
validate a configuration, it will validate against whatever latest version
is present locally.
With this KIP, that behavior will continue when the version isn't
specified, when the version is a soft requirement, or when the version is a
range. For users that do not want to tolerate their tasks having
heterogeneous versions, the single-version hard requirements are there, but
come at the cost of micromanagement of upgrades.
Perhaps there is some room in the future for a connector configured with a
range to "pin" the version that all of its tasks should use. Or
version-aware-scheduling can schedule connectors and tasks to workers that
will resolve the same versions. The important primitive is being able to
enforce which version is being used, everything else looks more like a
convenience feature to me.
> So far, we could have pointed to the
> misconfigured cluster configuration and somewhat differ this problem to
> something outside of connect runtime.
I think this will still be the case. Even with this feature, we will still
recommend homogeneous clusters that have the same set of plugins installed
on every worker. Version-aware scheduling is a rejected alternative, and
your example for performing validation requests on an arbitrary worker is a
great example of a complication for heterogeneous clusters that we're not
ready to solve.
> We can introduce
> a new path under this for version (/connector/connector-name/version), but
> perhaps adding this as part of the status is a valid alternative.
I think either is fine. Adding a new field to the existing endpoints is
fine, and shouldn't require special compatibility such as a query parameter.
> Also, to go further I think
> version information for tasks could also be available,
I completely agree. The version of each connector and each task should be
visible individually, since they may be different during a rolling upgrade.
Thanks!
Greg
On Tue, Jun 18, 2024 at 5:18 AM Snehashis <sn...@gmail.com> wrote:
> Hi Greg, Chris
>
> Thanks for the in-depth discussion, I have a couple of discussion points
> and would like your thoughts on this.
>
> 1) One concern I have with the new addition of 'soft' and 'hard' version
> requirements is that there could be a mismatch in the plugin version that
> two different tasks are running, if a soft requirement is provided and the
> nodes a multi cluster deployment are not in sync w.r.t the plugin versions
> that they are configured with. Note that if my assumptions are correct then
> this can happen with the existing framework as well, or is there some
> safeguard from this happening? So far, we could have pointed to the
> misconfigured cluster configuration and somewhat differ this problem to
> something outside of connect runtime. With this feature in place perhaps
> the expectation is more on connect to not be running with such
> inconsistency, especially if a connector version is specified. This is also
> a problem with validation if different cluster have different
> configurations, as IIRC validations are local to the worker which receives
> the rest call for validate. So, we might be validating with a certain
> version which is different from the one that will be used to create
> connector and tasks. Again, this is likely how the current state is, but
> perhaps such inconsistencies warrant a deeper look with the addition of
> this feature. The problems associated with them can be somewhat insidious
> and hard to diagnose.
>
> 2) There was some discussion on the need for a new REST endpoint to provide
> information on the versions of running connectors, and I think adding this
> information via REST is a valuable addition. The way I see it the version
> is an intrinsic property of an instance of a running connector and hence
> this should be part of the set of APIs under /connector/<connector-name>
> (also the /connectors API should also have this information as it is an
> amalgamation of all the individual connector information). We can introduce
> a new path under this for version (/connector/connector-name/version), but
> perhaps adding this as part of the status is a valid alternative. This is
> mentioned as a rejected alternative right now. Also, to go further I think
> version information for tasks could also be available, especially if we
> choose to not address the pitfalls discussed in my point 1), this will
> at-least provide admins a quick and easy way to determine if such and
> inconsistent state exist in any of the connectors.
>
> Thanks again for reviving my original KIP and working to improve it.
> Looking forward to your thoughts on the points mentioned above.
> Regards
> Snehashis
>
>
> On Wed, May 29, 2024 at 9:59 PM Chris Egerton <ch...@aiven.io.invalid>
> wrote:
>
> > Hi Greg,
> >
> > First, an apology! I mistakenly assumed that each plugin appeared only
> once
> > in the responses from GET /connector-plugins?connectorsOnly=false. Thank
> > you for correcting me and pointing out that all versions of each plugin
> > appear in that response, which does indeed satisfy my desire for users to
> > discover this information in at most two REST requests (and in fact, does
> > it in only one)!
> >
> > And secondly, with the revelation about recommenders, I agree that it's
> > best to leave the "version" property out of the lists of properties
> > returned from the GET /connector-plugins/<plugin>/config endpoint.
> >
> > With those two points settled, I think the only unresolved item is the
> > small change to version parsing added to the KIP (where raw version
> numbers
> > are treated as an exact match, instead of a best-effort match with a
> > fallback on the default version). If the KIP is updated with that then
> I'd
> > be ready to vote on it.
> >
> > Cheers,
> >
> > Chris
> >
> > On Wed, May 29, 2024 at 12:00 PM Greg Harris
> <greg.harris@aiven.io.invalid
> > >
> > wrote:
> >
> > > Hey Chris,
> > >
> > > Thanks for your thoughts.
> > >
> > > > Won't it still only expose the
> > > > latest version for each plugin, instead of the range of versions
> > > available?
> > >
> > > Here is a snippet of the current output of the GET
> > > /connector-plugins?connectorsOnly=false endpoint, after I installed two
> > > versions of the debezium PostgresConnector:
> > >
> > > {
> > > "class": "io.debezium.connector.postgresql.PostgresConnector",
> > > "type": "source",
> > > "version": "2.0.1.Final"
> > > },
> > > {
> > > "class": "io.debezium.connector.postgresql.PostgresConnector",
> > > "type": "source",
> > > "version": "2.6.1.Final"
> > > },
> > >
> > > I think this satisfies your requirement to learn about all plugins and
> > all
> > > versions in two or fewer REST calls.
> > >
> > > I tried to get an example of the output of `/config` by hardcoding the
> > > Recommender, and realized that Recommenders aren't executed on the
> > > `/config` endpoint at all: only during validation, when a configuration
> > is
> > > actually present.
> > > And this led me to discover that the `/config` endpoint returns a
> > > List<ConfigKeyInfo>, and ConfigKeyInfo does not contain a
> > recommendedValues
> > > field. The ConfigValue field is the object which contains
> > > recommendedValues, and it is only generated during validation.
> > > I think it's out of scope to start calling recommenders on empty
> configs
> > > that might throw exceptions, changing the existing REST entities, or
> > > changing the core ConfigDef implementation.
> > > Someone could add this functionality later, I don't think it's
> necessary
> > > here.
> > >
> > > Then the question is: should "version" without recommenders appear in
> > > non-connector plugins? I think I'd rather be consistent with
> "predicate"
> > > and "negate" on release, and let a later improvement add them.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Wed, May 29, 2024 at 8:06 AM Chris Egerton <chrise@aiven.io.invalid
> >
> > > wrote:
> > >
> > > > Hi Greg,
> > > >
> > > > I'm confused about the behavior for GET
> > > > /connector-plugins?connectorsOnly=false. Won't it still only expose
> the
> > > > latest version for each plugin, instead of the range of versions
> > > available?
> > > >
> > > > I'm hoping we can provide a flow where people need at most two REST
> > calls
> > > > to discover 1) the complete set of plugins available on the worker
> > (which
> > > > is already possible with the endpoint under discussion) and 2) the
> set
> > of
> > > > versions available for a specific plugin on the worker (which doesn't
> > > > appear to be possible, at least for some plugin types). This wouldn't
> > > > require any out-of-band knowledge and would be valuable for connector
> > > users
> > > > (who may want to, for example, know what their options are when
> > > considering
> > > > a plugin upgrade) and cluster administrators (who could use it as a
> > > sanity
> > > > check for the setup of their Kafka Connect clusters without having to
> > > pore
> > > > over log files).
> > > >
> > > > As far as modifying the content of the GET
> > > > /connector-plugins/<plugin>/config endpoint to include a "version"
> > > property
> > > > goes, I think your point about returning that property from requests
> to
> > > > that endpoint that include a version query parameter is salient, but
> it
> > > > also unfortunately applies to all types of plugin. I don't think it
> > > should
> > > > completely disqualify that option. I also wasn't imagining adding
> > > anything
> > > > besides that single property to that endpoint, so no new "predicate"
> > > > property for SMTs, no new "negate" property for predicates, and no
> new
> > > > "type" property for either. I'm not necessarily opposed to adding
> > those,
> > > > but it can be done without being pulled into the scope of this KIP.
> > > >
> > > > So to summarize how I imagine people might use the REST API after
> this
> > > KIP:
> > > >
> > > > 1. Discover the set of plugins on the worker via GET
> > > > /connector-plugins?connectorsOnly=false
> > > > 2. For any of those plugins, discover the set of available versions
> by
> > > > examining the recommended values for the "version" property after
> > hitting
> > > > the GET GET /connector-plugins/<plugin>/config endpoint
> > > >
> > > > I don't think this is necessarily optimal since plucking the
> "version"
> > > > property out of the response from that endpoint might be a PITA for
> ad
> > > hoc
> > > > queries via, e.g., curl, but I think it'd be enough for this KIP.
> Still
> > > > open to other options though, and if I've misunderstood anything in
> the
> > > > proposal, please let me know!
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Wed, May 22, 2024 at 4:05 PM Greg Harris
> > <greg.harris@aiven.io.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hey Chris,
> > > > >
> > > > > Thanks for your comments, and I'm glad that it seems like we're
> > > > > aligning on the vision here.
> > > > >
> > > > > > An
> > > > > > alternative could be to change existing behavior to fail fast on
> > any
> > > > > > invalid default converter configuration instead of just for
> invalid
> > > > > > versions
> > > > >
> > > > > I suppose if this is landing in 4.0, we have the opportunity to
> break
> > > > > compatibility and strictly validate the worker class configs, and
> > > > > could have a new consistent behavior instead of inconsistent but
> > > > > backwards-compatible behavior.
> > > > >
> > > > > > no other part of the KIP requires this change.
> > > > >
> > > > > This is correct, and a compelling argument. I'm fine leaving the
> > > > > strict worker validation off of this configuration to potentially
> be
> > > > > added later. If a KIP was raised to perform strict validation of
> the
> > > > > worker config, it would include the version config, and can address
> > > > > backwards compatibility for both configs together.
> > > > >
> > > > > > RE exposing the version property in the
> > > > > /connector-plugins/<plugin>/config
> > > > > > endpoint, the behavior is inconsistent across plugin types.
> > > > >
> > > > > Yes it is, and that started in a bugfix, once people started using
> > > > > this endpoint: https://issues.apache.org/jira/browse/KAFKA-14843
> . I
> > > > > wasn't planning on converging the behavior in this KIP, as I
> > > > > considered it out-of-scope, and was going to follow the current
> > > > > behavior. To summarize:
> > > > >
> > > > > GET /connector-plugins/<connector>/config will emit:
> > > > > * connector.class (already implemented)
> > > > > * connector.version (new)
> > > > > * key.converter (already implemented)
> > > > > * key.converter.version (new)
> > > > > * value.converter (already implemented)
> > > > > * value.converter.version (new)
> > > > > * header.converter (already implemented)
> > > > > * header.converter.version (new)
> > > > > But will NOT emit:
> > > > > * transforms.<alias>.type
> > > > > * transforms.<alias>.version
> > > > > * transforms.<alias>.predicate
> > > > > * predicates.<alias>.type
> > > > > * predicates.<alias>.version
> > > > > * predicates.<alias>.negate
> > > > >
> > > > > GET /connector-plugins/<converter>/config will NOT emit:
> > > > > * "" (there's not even a ".class" prefix!)
> > > > > * version
> > > > >
> > > > > GET /connector-plugins/<transform>/config will NOT emit:
> > > > > * type
> > > > > * version
> > > > > * predicate
> > > > >
> > > > > GET /connector-plugins/<predicate>/config will NOT emit:
> > > > > * type
> > > > > * version
> > > > > * negate
> > > > >
> > > > > Do you want the converter, transform, and predicate endpoints
> > changed?
> > > > > Do you want just "version", or do you want all of the prefixed
> > configs
> > > > > including "type", "predicate" and "negate"? How would you want to
> > > > > handle the converters?
> > > > > And when I say the configs are "not part of the plugin config
> itself"
> > > > > I mean that saying that GET
> > > > > /connector-plugins/Flatten$Key/config?version=3.8.0 has a "version"
> > > > > config that must be "3.8.0" is a little bit nonsense, as the
> version
> > > > > is already specified.
> > > > >
> > > > > > IMO
> > > > > > it's worth including this information somewhere directly
> accessible
> > > > > without
> > > > > > having to provide a full connector config. FWIW I'd be fine with
> > GET
> > > > > > /connector-plugins/<plugin>/versions as a first-class endpoint
> > > > >
> > > > > You don't have to provide a configuration to call GET
> > > > > /connector-plugins?connectorsOnly=false , is that endpoint not
> close
> > > > > enough to what you have in mind? See also the Rejected Alternative
> > > > > "Adding new REST API endpoints"
> > > > >
> > > > > If you're calling /connector-plugins/<transform>/config, you know
> the
> > > > > name of a plugin right? That either comes from out-of-band
> knowledge,
> > > > > validating a connector config, or calling GET
> > > > > /connector-plugins?connectorsOnly=false.
> > > > > * If you have out-of-band knowledge of plugin classes, perhaps you
> > > > > have out-of-band knowledge of versions too.
> > > > > * If you've just validated a connector config, there should be an
> > > > > accompanying "version" field there with an accurate default value
> and
> > > > > recommenders.
> > > > > * If you've called GET /connector-plugins?connectorsOnly=false,
> that
> > > > > endpoint includes version information.
> > > > >
> > > > > Thanks,
> > > > > Greg
> > > > >
> > > > > On Wed, May 22, 2024 at 11:05 AM Chris Egerton
> > <chrise@aiven.io.invalid
> > > >
> > > > > wrote:
> > > > > >
> > > > > > Hi Greg,
> > > > > >
> > > > > > Hope you had a nice weekend! Gonna try to keep things concise.
> > > > > >
> > > > > > Concluded points:
> > > > > >
> > > > > > RE version recommenders, I agree it's likely that programmatic
> UIs
> > > will
> > > > > > already be able to handle dynamic configuration definitions, and
> > the
> > > > > detail
> > > > > > about SMTs is a great point. I still anticipate some awkwardness
> > with
> > > > > > connector versions, though: if the latest version supports some
> new
> > > > > > properties, then a user switches to an earlier version, a UI may
> > > > respond
> > > > > by
> > > > > > wiping values for these properties. I guess we can bite the
> bullet,
> > > > > though.
> > > > > >
> > > > > > RE double-dinging during preflight validation for invalid
> > versions, I
> > > > > like
> > > > > > the analogy with login credentials. I'm convinced that the
> proposal
> > > in
> > > > > the
> > > > > > KIP is best 👍
> > > > > >
> > > > > > Continued points:
> > > > > >
> > > > > > RE failing on worker startup, sorry, I should be clearer: there
> is
> > no
> > > > > _new_
> > > > > > justification for it that doesn't also apply to existing
> behavior.
> > We
> > > > > > shouldn't diverge from existing behavior solely for this new
> case.
> > An
> > > > > > alternative could be to change existing behavior to fail fast on
> > any
> > > > > > invalid default converter configuration instead of just for
> invalid
> > > > > > versions, but I'd vote to just stick to existing behavior and not
> > > > > > complicate things, especially since no other part of the KIP
> > requires
> > > > > this
> > > > > > change.
> > > > > >
> > > > > > RE exposing the version property in the
> > > > > /connector-plugins/<plugin>/config
> > > > > > endpoint, the behavior is inconsistent across plugin types.
> Hitting
> > > the
> > > > > > endpoint for the FileStreamSinkConnector on version 3.7.0 yields
> a
> > > > > response
> > > > > > that includes, among other things, the "topics", "topics.regex",
> > and
> > > > > > "errors.tolerance" properties. I see that we don't do this
> > everywhere
> > > > > (the
> > > > > > examples you cite for SMT and converter properties are accurate),
> > but
> > > > IMO
> > > > > > it's worth including this information somewhere directly
> accessible
> > > > > without
> > > > > > having to provide a full connector config. FWIW I'd be fine with
> > GET
> > > > > > /connector-plugins/<plugin>/versions as a first-class endpoint
> > either
> > > > > > instead of or in addition to adding recommended values for all
> > plugin
> > > > > > versions.
> > > > > >
> > > > > > Thanks for your continued work on this KIP, and with the progress
> > > we're
> > > > > > making I'm optimistic about its chances of appearing in 4.0.0.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Wed, May 15, 2024 at 1:22 PM Greg Harris
> > > > <greg.harris@aiven.io.invalid
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hey Chris,
> > > > > > >
> > > > > > > Thanks for your quick follow up.
> > > > > > >
> > > > > > > > But this risk is already present with
> > > > > > > > existing error cases, and I don't see anything that justifies
> > > > > changing
> > > > > > > > existing behavior with an invalid converter class, or
> diverging
> > > > from
> > > > > it
> > > > > > > in
> > > > > > > > the case of invalid converter versions.
> > > > > > >
> > > > > > > The justification is to fail-fast, and prevent REST API users
> > from
> > > > > > > receiving errors from bad configs that they didn't write, or
> > maybe
> > > > > > > don't even know apply to them.
> > > > > > > Up until recently errors in these configurations surfaced as
> > > failures
> > > > > > > to create the connector, or failures to start, and you made
> them
> > > > > > > fail-fast during validation. I think this change is in the same
> > > > > > > spirit, pulling the validation further forward and not letting
> > > errors
> > > > > > > lie dormant.
> > > > > > >
> > > > > > > And to call back to your original concern about interrupting
> > > > > > > connectors that explicitly provide these configurations and
> don't
> > > use
> > > > > > > the worker configs: I expect that operators with a majority of
> > > these
> > > > > > > sorts of clients aren't going to be setting the worker .version
> > > > > > > properties, because it would have no effect on the majority of
> > > their
> > > > > > > connectors. They would be able to rely on
> backwards-compatibility
> > > and
> > > > > > > continue to ignore the class properties.
> > > > > > >
> > > > > > > > CLI
> > > > > > > > and programmatic UI developers will want to develop their own
> > > > tooling
> > > > > > > > layers.
> > > > > > >
> > > > > > > This is a very compelling argument. Snehashis do you want to
> > figure
> > > > > > > out a REST API design for this use-case?
> > > > > > >
> > > > > > > > Regarding the GET /connector-plugins/<plugin>/config
> endpoint,
> > I
> > > > was
> > > > > > > > thinking about the response for non-connector plugins, e.g.,
> > > > > > > > GET /connector-plugins/RegexRouter/config. Would a "version"
> > > > property
> > > > > > > > appear with recommended values?
> > > > > > >
> > > > > > > I intended for the ".version" property to be like other
> framework
> > > > > > > configs (".class", ".type", ".predicate", ".negate") where they
> > are
> > > > > > > inside the plugin namespace, but not part of the plugin config
> > > > itself.
> > > > > > > Perhaps we can deviate from those configs because it would aid
> in
> > > > > > > discovering other valid `GET
> > > > > > > /connector-plugins/<plugin>/config?version=` calls, without
> > calling
> > > > > > > `GET /connector-plugins?connectorsOnly=false`.
> > > > > > > I don't really feel strongly either way.
> > > > > > >
> > > > > > > > if a user changes the
> > > > > > > > connector version in, e.g., a dropdown menu, then the UI
> either
> > > has
> > > > > to
> > > > > > > > re-fetch the ConfigDef for the new version, or risk operating
> > on
> > > > > stale
> > > > > > > > information
> > > > > > >
> > > > > > > This is a really interesting situation, thanks for finding
> that!
> > > This
> > > > > > > is already a footgun with transformations and predicates; Once
> > you
> > > > > > > fill out a transformation class (which includes a recommender)
> > you
> > > > > > > will then see all of the transformations configurations.
> > > > > > > I think this means that UI developers should have already
> > developed
> > > > > > > the infrastructure for handling dynamic recommenders, or else
> > > they've
> > > > > > > had this bug since KIP-66 in 2017. It may require some manual
> > > > > > > attention to roll-out support for the ".version" properties
> > > > > > > specifically, but I don't think that should prevent us from
> > > providing
> > > > > > > this information in a natural place.
> > > > > > >
> > > > > > > > but will it be possible to configure
> > > > > > > > a connector to use two instances of the same transform or
> > > > predicate,
> > > > > but
> > > > > > > > with different versions for each?
> > > > > > >
> > > > > > > Yes, and this is included in the example in the KIP,
> > > > > > > `transforms.flatten-latest` and `transforms.flatten-old`. This
> > will
> > > > > > > work in the natural way, with the two instances having a
> > different
> > > > > > > version if configured for that. If a user wants to pin the same
> > > > > > > version of a plugin for every instance, they will need to
> provide
> > > the
> > > > > > > version config multiple times and keep them in-sync themselves.
> > > > > > >
> > > > > > > > I would have expected the error
> > > > > > > > to only be attributed to the version property, and for the
> > class
> > > > > property
> > > > > > > > to be reported as valid.
> > > > > > >
> > > > > > > I considered this, and then went back and changed it. To quote
> > > > someone
> > > > > > > else, this is to catch "misspelling cat as dog". For example a
> > user
> > > > > > > types DogConverter and meant to type CatConverter, and then
> > types a
> > > > > > > version which is valid for CatConverter, conceptually the error
> > is
> > > in
> > > > > > > the class name and not the version.
> > > > > > > That's a very contrived scenario, but I think similar arguments
> > are
> > > > > > > used for attributing validation errors to
> > endpoints/urls/hostnames
> > > > > > > when the associated credentials are unable to log into the
> remote
> > > > > > > system. Did the user provide the correct testing credentials,
> but
> > > > > > > accidentally typed the production endpoint? Even if the
> > production
> > > > > > > endpoint looks valid (it's a real hostname that is reachable)
> the
> > > > > > > conceptual error is still in the hostname and should have the
> > error
> > > > > > > attributed to it to draw the user's attention.
> > > > > > > If that's not convincing, I think the alternative of only
> > > attributing
> > > > > > > version errors to the version property is also acceptable.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Greg
> > > > > > >
> > > > > > > On Wed, May 15, 2024 at 8:06 AM Chris Egerton
> > > > <chrise@aiven.io.invalid
> > > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Greg,
> > > > > > > >
> > > > > > > > Thanks for your responses! Continuations of existing
> > discussions:
> > > > > > > >
> > > > > > > > Regarding crashing the worker on startup--yes, there is also
> a
> > > risk
> > > > > to
> > > > > > > > allowing it to join the cluster. But this risk is already
> > present
> > > > > with
> > > > > > > > existing error cases, and I don't see anything that justifies
> > > > > changing
> > > > > > > > existing behavior with an invalid converter class, or
> diverging
> > > > from
> > > > > it
> > > > > > > in
> > > > > > > > the case of invalid converter versions. I think we should
> keep
> > > this
> > > > > > > simple
> > > > > > > > and not do anything different for worker startup than we do
> > > > already.
> > > > > > > >
> > > > > > > > As far as REST API vs. metrics go--I think you're right that
> > the
> > > > > original
> > > > > > > > version metrics were added as a "monitoring" detail. However,
> > > this
> > > > > was
> > > > > > > back
> > > > > > > > when plugin versions were managed solely by cluster
> > > administrators.
> > > > > With
> > > > > > > > this KIP, connector users will be able to manage plugin
> > versions,
> > > > > and CLI
> > > > > > > > and programmatic UI developers will want to develop their own
> > > > tooling
> > > > > > > > layers. I think focusing on the REST API as the primary
> > interface
> > > > for
> > > > > > > this
> > > > > > > > KIP would be best for these users.
> > > > > > > >
> > > > > > > > (All that said, I don't object to the metrics that are
> proposed
> > > in
> > > > > the
> > > > > > > KIP;
> > > > > > > > I just think they make more sense in addition to new REST API
> > > > > > > > functionality, as opposed to instead of it.)
> > > > > > > >
> > > > > > > > Regarding the GET /connector-plugins/<plugin>/config
> endpoint,
> > I
> > > > was
> > > > > > > > thinking about the response for non-connector plugins, e.g.,
> > > > > > > > GET /connector-plugins/RegexRouter/config. Would a "version"
> > > > property
> > > > > > > > appear with recommended values?
> > > > > > > >
> > > > > > > >
> > > > > > > > And new thoughts:
> > > > > > > >
> > > > > > > > 1) Regarding the recommended values for "connector.version",
> > this
> > > > > might
> > > > > > > be
> > > > > > > > confusing since there could be differences between the
> > ConfigDefs
> > > > > for the
> > > > > > > > latest version of the connector and prior versions. It also
> > makes
> > > > the
> > > > > > > flow
> > > > > > > > a bit awkward for programmatic UI developers: if a user
> changes
> > > the
> > > > > > > > connector version in, e.g., a dropdown menu, then the UI
> either
> > > has
> > > > > to
> > > > > > > > re-fetch the ConfigDef for the new version, or risk operating
> > on
> > > > > stale
> > > > > > > > information. I'm starting to doubt that exposing the range of
> > > > > available
> > > > > > > > versions via recommended values is the best way to proceed,
> > > instead
> > > > > of a
> > > > > > > > more explicit approach like GET
> > > > > /connector-plugins/<plugin>/versions, or
> > > > > > > > the "Adding new REST API endpoints" rejected alternative.
> > > > > > > >
> > > > > > > > 2) I know that this is a bit heinous, but will it be possible
> > to
> > > > > > > configure
> > > > > > > > a connector to use two instances of the same transform or
> > > > predicate,
> > > > > but
> > > > > > > > with different versions for each? (I don't think this is
> worth
> > > > > > > significant
> > > > > > > > design/implementation effort, so if it would inflate either
> of
> > > > those,
> > > > > > > > please don't feel obligated to support it.)
> > > > > > > >
> > > > > > > > 3) Just out of curiosity, why double-ding during config
> > > validation
> > > > > if the
> > > > > > > > version for a plugin class can't be found? I would have
> > expected
> > > > the
> > > > > > > error
> > > > > > > > to only be attributed to the version property, and for the
> > class
> > > > > property
> > > > > > > > to be reported as valid.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Tue, May 14, 2024 at 1:11 PM Greg Harris
> > > > > <greg.harris@aiven.io.invalid
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Chris,
> > > > > > > > >
> > > > > > > > > Thanks for your comments. I'm glad you like the
> motivations,
> > > > > Snehashis
> > > > > > > > > wrote that part!
> > > > > > > > >
> > > > > > > > > > the configuration syntax for the most basic use case of
> > > > > > > > > > specifying a single desired version is pretty
> > > counterintuitive.
> > > > > > > > >
> > > > > > > > > I agree, and the "soft" requirement scheme is something I
> > > wasn't
> > > > > > > > > explicitly looking for, but would be inherited from the
> > > library.
> > > > > I'm
> > > > > > > > > fine with eliminating the soft requirement semantics, and
> > > having
> > > > > > > > > "1.0.0" behave the same as "[1.0.0]". I'm less inclined to
> > > > include
> > > > > > > > > "range" in the property name, or have two properties.
> > > > > > > > >
> > > > > > > > > > Failing startup is drastic and has the potential to
> disrupt
> > > the
> > > > > > > > > > availability of connectors that would otherwise be able
> to
> > > run
> > > > > > > healthily
> > > > > > > > > > because they were explicitly configured to use valid
> > > converters
> > > > > > > instead
> > > > > > > > > of
> > > > > > > > > > the worker defaults.
> > > > > > > > >
> > > > > > > > > I think this argument cuts both ways. If someone
> > reconfigures a
> > > > > worker
> > > > > > > > > and adds an invalid ".version" string to the worker (or
> > changes
> > > > the
> > > > > > > > > plugin.path to make it invalid), it would be permitted to
> > enter
> > > > the
> > > > > > > > > group, and accept work assignments. If those work
> assignments
> > > > used
> > > > > > > > > these configurations, a set of tasks could transition to
> > FAILED
> > > > and
> > > > > > > > > not be able to recover, because they would be restarted
> again
> > > on
> > > > > the
> > > > > > > > > same worker.
> > > > > > > > >
> > > > > > > > > > Why are metrics utilized to report information about
> plugin
> > > > > versions
> > > > > > > > > > utilized by connectors at runtime instead of publishing
> > this
> > > > > info in
> > > > > > > the
> > > > > > > > > > REST API
> > > > > > > > >
> > > > > > > > > I was following the existing pattern for exposing runtime
> > > > versions
> > > > > for
> > > > > > > > > connectors, and it did seem like a "monitoring" feature. If
> > > that
> > > > > > > > > approach is flawed and should be phased out, I think it
> would
> > > be
> > > > a
> > > > > > > > > good idea to reconsider the REST API rejected alternative.
> > > > > > > > > We would need some additional design work to spec out the
> > REST
> > > > API
> > > > > > > > > interface, as I don't have anything in mind currently.
> > > > > > > > >
> > > > > > > > > > I'm unclear on whether or not it'll be possible to see
> this
> > > > > > > information
> > > > > > > > > via the
> > > > > > > > > > GET /connector-plugins/<plugin>/config endpoint
> > > > > > > > >
> > > > > > > > > There is room in the API to add recommenders for
> > > "key.converter",
> > > > > > > > > "value.converter", and "header.converter", but not for
> > > transforms
> > > > > and
> > > > > > > > > predicates, as they include aliases that depend on an
> actual
> > > > > > > > > configuration. We could explicitly say we're going to do
> > that,
> > > or
> > > > > do
> > > > > > > > > whatever is convenient during the implementation phase, or
> > > leave
> > > > it
> > > > > > > > > open to be improved later.
> > > > > > > > > There will not be any recommenders for ".version"
> properties
> > in
> > > > the
> > > > > > > > > `/config` endpoint, because those recommenders are dynamic
> > and
> > > > > depend
> > > > > > > > > on an actual configuration.
> > > > > > > > >
> > > > > > > > > 5) There are two relevant lines in the KIP: "If a .version
> > > > property
> > > > > > > > > contains a hard requirement, select the latest installed
> > > version
> > > > > which
> > > > > > > > > satisfies the requirement." and "This configuration is
> > > > re-evaluated
> > > > > > > > > each time the connector or task are assigned to a new
> > worker".
> > > I
> > > > > would
> > > > > > > > > call this "eager" upgrade behavior, rather than a "sticky"
> or
> > > > > "lazy"
> > > > > > > > > upgrade behavior.
> > > > > > > > >
> > > > > > > > > 6) Updated!
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Greg
> > > > > > > > >
> > > > > > > > > On Tue, May 14, 2024 at 9:14 AM Chris Egerton
> > > > > <chrise@aiven.io.invalid
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > Thanks Greg for updating the KIP, and thanks Snehashis
> for
> > > > > starting
> > > > > > > the
> > > > > > > > > > work on this originally.
> > > > > > > > > >
> > > > > > > > > > The motivation section makes a pretty convincing case for
> > > this
> > > > > kind
> > > > > > > of
> > > > > > > > > > feature. My thoughts are mostly about specific details:
> > > > > > > > > >
> > > > > > > > > > 1) I like the support for version ranges (the example
> > > > > demonstrating
> > > > > > > how
> > > > > > > > > to
> > > > > > > > > > avoid KAFKA-10574 with the header converter was
> > particularly
> > > > > > > > > entertaining),
> > > > > > > > > > but the configuration syntax for the most basic use case
> of
> > > > > > > specifying a
> > > > > > > > > > single desired version is pretty counterintuitive. People
> > may
> > > > get
> > > > > > > bitten
> > > > > > > > > or
> > > > > > > > > > at least frustrated if they put connector.version=3.8.0
> in
> > a
> > > > > config
> > > > > > > but
> > > > > > > > > > then version 3.7.5 ends up running. I'd like it if we
> could
> > > > > either
> > > > > > > > > > intentionally deviate from Maven ranges when a bare
> version
> > > is
> > > > > > > present,
> > > > > > > > > or
> > > > > > > > > > separate things out into two properties: foo.version
> would
> > be
> > > > the
> > > > > > > single
> > > > > > > > > > accepted version for the foo plugin, and
> foo.version.range
> > > > would
> > > > > use
> > > > > > > > > Maven
> > > > > > > > > > range syntax. Open to other options too, just providing a
> > > > couple
> > > > > to
> > > > > > > get
> > > > > > > > > the
> > > > > > > > > > ball rolling.
> > > > > > > > > >
> > > > > > > > > > 2) Although the current behavior for a worker with an
> > invalid
> > > > > > > > > > key/value/header converter class specified in its config
> > file
> > > > is
> > > > > a
> > > > > > > little
> > > > > > > > > > strange (I was surprised to learn that it wouldn't fail
> on
> > > > > startup),
> > > > > > > I
> > > > > > > > > > don't see a good reason to deviate from this when an
> > invalid
> > > > > version
> > > > > > > is
> > > > > > > > > > specified. Failing startup is drastic and has the
> potential
> > > to
> > > > > > > disrupt
> > > > > > > > > the
> > > > > > > > > > availability of connectors that would otherwise be able
> to
> > > run
> > > > > > > healthily
> > > > > > > > > > because they were explicitly configured to use valid
> > > converters
> > > > > > > instead
> > > > > > > > > of
> > > > > > > > > > the worker defaults.
> > > > > > > > > >
> > > > > > > > > > 3) Why are metrics utilized to report information about
> > > plugin
> > > > > > > versions
> > > > > > > > > > utilized by connectors at runtime instead of publishing
> > this
> > > > > info in
> > > > > > > the
> > > > > > > > > > REST API? I saw that this was mentioned as a rejected
> > > > > alternative,
> > > > > > > but I
> > > > > > > > > > didn't get a sense of why. It seems like the REST API
> would
> > > be
> > > > > > > easier to
> > > > > > > > > > access and more intuitive for most users than new
> metrics.
> > > > > > > > > >
> > > > > > > > > > 4) In the "Validation" section it's stated that "Users
> can
> > > use
> > > > > these
> > > > > > > > > > recommenders to discover the valid plugin classes and
> > > versions,
> > > > > > > without
> > > > > > > > > > requiring an earlier call to GET
> > > > > > > > > /connector-plugins?connectorsOnly=false."
> > > > > > > > > > I really like the creativity and simplicity of reusing
> the
> > > > > > > recommender
> > > > > > > > > > mechanism to expose available versions for plugins via
> the
> > > REST
> > > > > API.
> > > > > > > I'm
> > > > > > > > > > unclear on whether or not it'll be possible to see this
> > > > > information
> > > > > > > via
> > > > > > > > > the
> > > > > > > > > > GET /connector-plugins/<plugin>/config endpoint, though.
> > It'd
> > > > be
> > > > > > > great if
> > > > > > > > > > this were supported, since we learned in KIP-769 [1] that
> > > > people
> > > > > > > really
> > > > > > > > > > want to be able to see configuration options for
> connectors
> > > and
> > > > > their
> > > > > > > > > > plugins via some kind of GET endpoint without having to
> > > > provide a
> > > > > > > > > complete
> > > > > > > > > > connector config for validation.
> > > > > > > > > >
> > > > > > > > > > 5) In the Maven version range docs, it's stated that
> "Maven
> > > > > picks the
> > > > > > > > > > highest version of each project that satisfies all the
> hard
> > > > > > > requirements
> > > > > > > > > of
> > > > > > > > > > the dependencies on that project." I'm guessing this
> > behavior
> > > > > will be
> > > > > > > > > > retained for Connect; i.e., the highest-possible version
> of
> > > > each
> > > > > > > plugin
> > > > > > > > > > that satisfies the user-specified version constraints
> will
> > be
> > > > > run?
> > > > > > > (An
> > > > > > > > > > alternative approach could be to have some kind of
> "sticky"
> > > > logic
> > > > > > > that
> > > > > > > > > only
> > > > > > > > > > restarts connectors/tasks when their currently-used
> version
> > > > > becomes
> > > > > > > > > > incompatible with the configured constraints.)
> > > > > > > > > >
> > > > > > > > > > 6) (Nit) It'd be nice to add a link to the TestPlugins
> > class
> > > or
> > > > > > > somewhere
> > > > > > > > > > in its neighborhood to the testing plan; unfamiliar
> readers
> > > > > probably
> > > > > > > > > won't
> > > > > > > > > > get much out of what's there right now.
> > > > > > > > > >
> > > > > > > > > > [1] -
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
> > > > > > > > > > Chris
> > > > > > > > > >
> > > > > > > > > > On Mon, May 13, 2024 at 2:01 PM Snehashis <
> > > > > snehashisp1995@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Greg,
> > > > > > > > > > >
> > > > > > > > > > > That is much appreciated. No complaints on the
> additional
> > > > > scope, I
> > > > > > > will
> > > > > > > > > > > make some time out to work on this once we have
> approval.
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > > Snehashis
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hey Snehashis,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm glad to hear you're still interested in this KIP!
> > > > > > > > > > > > I'm happy to let you drive this, and I apologize for
> > > > > increasing
> > > > > > > the
> > > > > > > > > > > > scope of work so drastically. To make up for that,
> I'll
> > > > > > > volunteer to
> > > > > > > > > > > > be the primary PR reviewer to help get this done
> > quickly
> > > > once
> > > > > > > the KIP
> > > > > > > > > > > > is approved.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Greg
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, May 10, 2024 at 3:51 AM Snehashis <
> > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the follow up to my original KIP, I am
> in
> > > > > favour of
> > > > > > > the
> > > > > > > > > > > > > additions made to expand its scope, the addition of
> > > range
> > > > > > > versions
> > > > > > > > > > > > > specifically make a lot of sense.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Apologies if I have not publicly worked on this KIP
> > > for a
> > > > > long
> > > > > > > > > time.
> > > > > > > > > > > The
> > > > > > > > > > > > > original work was done when the move to service
> > loading
> > > > > was in
> > > > > > > > > > > discussion
> > > > > > > > > > > > > and I wanted to loop back to this only after that
> > work
> > > > was
> > > > > > > > > completed.
> > > > > > > > > > > > Post
> > > > > > > > > > > > > its conclusion, I have not been able to take this
> up
> > > due
> > > > to
> > > > > > > other
> > > > > > > > > > > > > priorities. If it's okay with you, I would still
> like
> > > to
> > > > > get
> > > > > > > this
> > > > > > > > > > > > > implemented myself, including the additional scope.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks and regards
> > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> > > > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'd like to reboot the discussion on KIP-891:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I've made some changes, most notably:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. Specifying versions for all plugins in
> Connector
> > > > > configs
> > > > > > > > > > > > > > (converters, header converters, transforms, and
> > > > > predicates)
> > > > > > > not
> > > > > > > > > just
> > > > > > > > > > > > > > connectors & tasks
> > > > > > > > > > > > > > 2. Specifying a range of versions instead of an
> > exact
> > > > > match
> > > > > > > > > > > > > > 3. New metrics to observe what versions are
> in-use
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks to Snehashis for the original KIP idea!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <
> > > > > > > > > greg.harris@aiven.io>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Snehashis,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you for the KIP! This is something I've
> > > wanted
> > > > > for a
> > > > > > > long
> > > > > > > > > > > time.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I know the discussion has gone cold, are you
> > still
> > > > > > > interested
> > > > > > > > > in
> > > > > > > > > > > > > > > pursuing this feature? I'll make time to review
> > the
> > > > > KIP if
> > > > > > > you
> > > > > > > > > are
> > > > > > > > > > > > > > > still accepting comments.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis <
> > > > > > > > > > > snehashisp1995@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for the points Sagar.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1) Should we update the GET /connectors
> > > endpoint
> > > > to
> > > > > > > > > include the
> > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > the plugin that is running? It could be
> > useful
> > > to
> > > > > > > figure
> > > > > > > > > out
> > > > > > > > > > > the
> > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > the plugin or I am assuming it gets
> returned
> > by
> > > > the
> > > > > > > > > expand=info
> > > > > > > > > > > > call?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think this is good to have and possible
> > future
> > > > > > > > > enhancement. The
> > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > info will be present in the config of the
> > > connector
> > > > > if
> > > > > > > the
> > > > > > > > > user
> > > > > > > > > > > has
> > > > > > > > > > > > > > > > specified the version. Otherwise it is the
> > latest
> > > > > version
> > > > > > > > > which
> > > > > > > > > > > the
> > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > can find out from the connector-plugin
> > endpoint.
> > > > The
> > > > > > > > > information
> > > > > > > > > > > > can be
> > > > > > > > > > > > > > > > introduced to the response of the GET
> > /connectors
> > > > > > > endpoint
> > > > > > > > > > > itself,
> > > > > > > > > > > > > > however
> > > > > > > > > > > > > > > > the most ideal way of doing this would be to
> > get
> > > > the
> > > > > > > > > currently
> > > > > > > > > > > > running
> > > > > > > > > > > > > > > > instance of the connector and get the version
> > > > > directly
> > > > > > > from
> > > > > > > > > > > there.
> > > > > > > > > > > > > > This is
> > > > > > > > > > > > > > > > slightly tricky as the connector could be
> > running
> > > > in
> > > > > a
> > > > > > > > > different
> > > > > > > > > > > > node.
> > > > > > > > > > > > > > > > One way to do this would be to persist the
> > > version
> > > > > > > > > information in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > status backing store during instantiation of
> > the
> > > > > > > connector.
> > > > > > > > > It
> > > > > > > > > > > > requires
> > > > > > > > > > > > > > > > some more thought and since the version is
> part
> > > of
> > > > > the
> > > > > > > > > configs if
> > > > > > > > > > > > > > provided
> > > > > > > > > > > > > > > > and evident otherwise, I have not included it
> > in
> > > > this
> > > > > > > KIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2) I am not aware of this and hence asking,
> > > can 2
> > > > > > > > > connectors
> > > > > > > > > > > with
> > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > versions have the same name? Does the
> plugin
> > > > > isolation
> > > > > > > > > allow
> > > > > > > > > > > > this?
> > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > could have a bearing when using the
> lifecycle
> > > > > > > endpoints for
> > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > All connectors in a cluster need to have
> > uniquire
> > > > > > > connector
> > > > > > > > > names
> > > > > > > > > > > > > > > > regardless of what version of the plugin the
> > > > > connector is
> > > > > > > > > running
> > > > > > > > > > > > > > > > underneath. This is something enforced by the
> > > > connect
> > > > > > > runtime
> > > > > > > > > > > > itself.
> > > > > > > > > > > > > > All
> > > > > > > > > > > > > > > > connect CRUD operations are keyed on the
> > > connector
> > > > > name
> > > > > > > so
> > > > > > > > > there
> > > > > > > > > > > > will
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > be an issue.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <
> > > > > > > > > sagarmeansocean@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hey Snehashsih,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for the KIP. It looks like a very
> > useful
> > > > > > > feature.
> > > > > > > > > Couple
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > small-ish points, let me know what you
> think:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1) Should we update the GET /connectors
> > > endpoint
> > > > to
> > > > > > > > > include the
> > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > the plugin that is running? It could be
> > useful
> > > to
> > > > > > > figure
> > > > > > > > > out
> > > > > > > > > > > the
> > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > the plugin or I am assuming it gets
> returned
> > by
> > > > the
> > > > > > > > > expand=info
> > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > 2) I am not aware of this and hence asking,
> > > can 2
> > > > > > > > > connectors
> > > > > > > > > > > with
> > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > versions have the same name? Does the
> plugin
> > > > > isolation
> > > > > > > > > allow
> > > > > > > > > > > > this?
> > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > could have a bearing when using the
> lifecycle
> > > > > > > endpoints for
> > > > > > > > > > > > > > connectors like
> > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > Sagar.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin
> > > > > > > > > > > > <apankaj@confluent.io.invalid
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > IIUC (please correct me if I am wrong
> > > here),
> > > > > what
> > > > > > > you
> > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > a versioning scheme for a connector
> config
> > > for
> > > > > the
> > > > > > > same
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > different versions of a connector
> plugin).
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Sorry for not being more precise in my
> > > wording
> > > > > - I
> > > > > > > meant
> > > > > > > > > > > > > > registering
> > > > > > > > > > > > > > > > > > versions of schema for connector config.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Let's take the example of a fictional
> > > connector
> > > > > which
> > > > > > > > > uses a
> > > > > > > > > > > > > > fictional
> > > > > > > > > > > > > > > > > AWS
> > > > > > > > > > > > > > > > > > service.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Fictional Connector Config schema
> > version:2.0
> > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Fictional Connector config schema
> > version:3.0
> > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > "iam_role": {
> > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > "iam_role"
> > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The connector which supports Fictional
> > config
> > > > > schema
> > > > > > > 2.0
> > > > > > > > > > > will
> > > > > > > > > > > > > > validate
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > access key and secret key.
> > > > > > > > > > > > > > > > > > Whereas a connector which supports config
> > > with
> > > > > schema
> > > > > > > > > version
> > > > > > > > > > > > 3.0
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > validate the IAM role.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This is the alternative which I wanted to
> > > > > suggest.
> > > > > > > Each
> > > > > > > > > > > plugin
> > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > register the schema versions of connector
> > > > config
> > > > > > > which it
> > > > > > > > > > > > supports.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The plugin paths may be optionally
> > different
> > > > > i.e we
> > > > > > > > > don't
> > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > mandatorily add a new plugin path to
> > support
> > > a
> > > > > new
> > > > > > > schema
> > > > > > > > > > > > version.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:47 PM
> Snehashis
> > <
> > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on the rejected
> > > > > > > alternatives ?
> > > > > > > > > > > Suppose
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > config is versioned and has a schema.
> > > Then
> > > > a
> > > > > > > single
> > > > > > > > > > > plugin
> > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > dependencies have not changed) can
> > handle
> > > > > > > multiple
> > > > > > > > > config
> > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > IIUC (please correct me if I am wrong
> > > here),
> > > > > what
> > > > > > > you
> > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > a versioning scheme for a connector
> > config
> > > > for
> > > > > the
> > > > > > > same
> > > > > > > > > > > > > > connector (and
> > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > different versions of a connector
> > plugin).
> > > > > That is
> > > > > > > a
> > > > > > > > > > > somewhat
> > > > > > > > > > > > > > > > > tangential
> > > > > > > > > > > > > > > > > > > problem. While it is definitely a
> useful
> > > > > feature to
> > > > > > > > > have,
> > > > > > > > > > > > like a
> > > > > > > > > > > > > > log to
> > > > > > > > > > > > > > > > > > > check what changes were made over time
> to
> > > the
> > > > > > > config
> > > > > > > > > which
> > > > > > > > > > > > might
> > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > easier to do rollbacks, it is not the
> > focus
> > > > > here.
> > > > > > > Here
> > > > > > > > > by
> > > > > > > > > > > > > > version we
> > > > > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > > > to say what underlying version of the
> > > plugin
> > > > > > > should the
> > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > of the connector use. Perhaps it is
> > better
> > > to
> > > > > > > change
> > > > > > > > > the
> > > > > > > > > > > > name of
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > parameter from connector.version to
> > > > > > > > > > > connector.plugin.version
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > plugin.version if it was confusing.
> wdyt?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2. Any plans to support assisted
> > > migration
> > > > > e.g
> > > > > > > if a
> > > > > > > > > user
> > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > connector/config?migrate=latest", the
> > > > latest
> > > > > > > version
> > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > transform the existing config to the
> > > newer
> > > > > > > version.
> > > > > > > > > This
> > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > adding a method like "boolean
> > > > migrate(Version
> > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This is an enhancement we can think of
> > > doing
> > > > in
> > > > > > > future.
> > > > > > > > > > > > Users can
> > > > > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > > a PUT call with the updated config
> which
> > > has
> > > > > the
> > > > > > > > > updated
> > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > number.
> > > > > > > > > > > > > > > > > > > The assisted mode could be handy as the
> > > user
> > > > > does
> > > > > > > not
> > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > > > know the
> > > > > > > > > > > > > > > > > > > config but beyond this it does not seem
> > to
> > > > > justify
> > > > > > > its
> > > > > > > > > > > > existence.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin
> > > > > > > > > > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This is a really useful feature and
> > > thanks
> > > > > for
> > > > > > > > > initiating
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I had the following questions -
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on the rejected
> > > > > > > alternatives ?
> > > > > > > > > > > Suppose
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > config is versioned and has a schema.
> > > Then
> > > > a
> > > > > > > single
> > > > > > > > > > > plugin
> > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > dependencies have not changed) can
> > handle
> > > > > > > multiple
> > > > > > > > > config
> > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2. Any plans to support assisted
> > > migration
> > > > > e.g
> > > > > > > if a
> > > > > > > > > user
> > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > connector/config?migrate=latest", the
> > > > latest
> > > > > > > version
> > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > transform the existing config to the
> > > newer
> > > > > > > version.
> > > > > > > > > This
> > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > adding a method like "boolean
> > > > migrate(Version
> > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM
> > > Snehashis <
> > > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I'd like to start a discussion
> thread
> > > on
> > > > > > > KIP-891:
> > > > > > > > > > > Running
> > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The KIP aims to add the ability for
> > the
> > > > > connect
> > > > > > > > > runtime
> > > > > > > > > > > > to
> > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > versions of a connector.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Please take a look and let me know
> > what
> > > > you
> > > > > > > think.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Snehashis <sn...@gmail.com>.
Hi Greg, Chris
Thanks for the in-depth discussion, I have a couple of discussion points
and would like your thoughts on this.
1) One concern I have with the new addition of 'soft' and 'hard' version
requirements is that there could be a mismatch in the plugin version that
two different tasks are running, if a soft requirement is provided and the
nodes a multi cluster deployment are not in sync w.r.t the plugin versions
that they are configured with. Note that if my assumptions are correct then
this can happen with the existing framework as well, or is there some
safeguard from this happening? So far, we could have pointed to the
misconfigured cluster configuration and somewhat differ this problem to
something outside of connect runtime. With this feature in place perhaps
the expectation is more on connect to not be running with such
inconsistency, especially if a connector version is specified. This is also
a problem with validation if different cluster have different
configurations, as IIRC validations are local to the worker which receives
the rest call for validate. So, we might be validating with a certain
version which is different from the one that will be used to create
connector and tasks. Again, this is likely how the current state is, but
perhaps such inconsistencies warrant a deeper look with the addition of
this feature. The problems associated with them can be somewhat insidious
and hard to diagnose.
2) There was some discussion on the need for a new REST endpoint to provide
information on the versions of running connectors, and I think adding this
information via REST is a valuable addition. The way I see it the version
is an intrinsic property of an instance of a running connector and hence
this should be part of the set of APIs under /connector/<connector-name>
(also the /connectors API should also have this information as it is an
amalgamation of all the individual connector information). We can introduce
a new path under this for version (/connector/connector-name/version), but
perhaps adding this as part of the status is a valid alternative. This is
mentioned as a rejected alternative right now. Also, to go further I think
version information for tasks could also be available, especially if we
choose to not address the pitfalls discussed in my point 1), this will
at-least provide admins a quick and easy way to determine if such and
inconsistent state exist in any of the connectors.
Thanks again for reviving my original KIP and working to improve it.
Looking forward to your thoughts on the points mentioned above.
Regards
Snehashis
On Wed, May 29, 2024 at 9:59 PM Chris Egerton <ch...@aiven.io.invalid>
wrote:
> Hi Greg,
>
> First, an apology! I mistakenly assumed that each plugin appeared only once
> in the responses from GET /connector-plugins?connectorsOnly=false. Thank
> you for correcting me and pointing out that all versions of each plugin
> appear in that response, which does indeed satisfy my desire for users to
> discover this information in at most two REST requests (and in fact, does
> it in only one)!
>
> And secondly, with the revelation about recommenders, I agree that it's
> best to leave the "version" property out of the lists of properties
> returned from the GET /connector-plugins/<plugin>/config endpoint.
>
> With those two points settled, I think the only unresolved item is the
> small change to version parsing added to the KIP (where raw version numbers
> are treated as an exact match, instead of a best-effort match with a
> fallback on the default version). If the KIP is updated with that then I'd
> be ready to vote on it.
>
> Cheers,
>
> Chris
>
> On Wed, May 29, 2024 at 12:00 PM Greg Harris <greg.harris@aiven.io.invalid
> >
> wrote:
>
> > Hey Chris,
> >
> > Thanks for your thoughts.
> >
> > > Won't it still only expose the
> > > latest version for each plugin, instead of the range of versions
> > available?
> >
> > Here is a snippet of the current output of the GET
> > /connector-plugins?connectorsOnly=false endpoint, after I installed two
> > versions of the debezium PostgresConnector:
> >
> > {
> > "class": "io.debezium.connector.postgresql.PostgresConnector",
> > "type": "source",
> > "version": "2.0.1.Final"
> > },
> > {
> > "class": "io.debezium.connector.postgresql.PostgresConnector",
> > "type": "source",
> > "version": "2.6.1.Final"
> > },
> >
> > I think this satisfies your requirement to learn about all plugins and
> all
> > versions in two or fewer REST calls.
> >
> > I tried to get an example of the output of `/config` by hardcoding the
> > Recommender, and realized that Recommenders aren't executed on the
> > `/config` endpoint at all: only during validation, when a configuration
> is
> > actually present.
> > And this led me to discover that the `/config` endpoint returns a
> > List<ConfigKeyInfo>, and ConfigKeyInfo does not contain a
> recommendedValues
> > field. The ConfigValue field is the object which contains
> > recommendedValues, and it is only generated during validation.
> > I think it's out of scope to start calling recommenders on empty configs
> > that might throw exceptions, changing the existing REST entities, or
> > changing the core ConfigDef implementation.
> > Someone could add this functionality later, I don't think it's necessary
> > here.
> >
> > Then the question is: should "version" without recommenders appear in
> > non-connector plugins? I think I'd rather be consistent with "predicate"
> > and "negate" on release, and let a later improvement add them.
> >
> > Thanks,
> > Greg
> >
> > On Wed, May 29, 2024 at 8:06 AM Chris Egerton <ch...@aiven.io.invalid>
> > wrote:
> >
> > > Hi Greg,
> > >
> > > I'm confused about the behavior for GET
> > > /connector-plugins?connectorsOnly=false. Won't it still only expose the
> > > latest version for each plugin, instead of the range of versions
> > available?
> > >
> > > I'm hoping we can provide a flow where people need at most two REST
> calls
> > > to discover 1) the complete set of plugins available on the worker
> (which
> > > is already possible with the endpoint under discussion) and 2) the set
> of
> > > versions available for a specific plugin on the worker (which doesn't
> > > appear to be possible, at least for some plugin types). This wouldn't
> > > require any out-of-band knowledge and would be valuable for connector
> > users
> > > (who may want to, for example, know what their options are when
> > considering
> > > a plugin upgrade) and cluster administrators (who could use it as a
> > sanity
> > > check for the setup of their Kafka Connect clusters without having to
> > pore
> > > over log files).
> > >
> > > As far as modifying the content of the GET
> > > /connector-plugins/<plugin>/config endpoint to include a "version"
> > property
> > > goes, I think your point about returning that property from requests to
> > > that endpoint that include a version query parameter is salient, but it
> > > also unfortunately applies to all types of plugin. I don't think it
> > should
> > > completely disqualify that option. I also wasn't imagining adding
> > anything
> > > besides that single property to that endpoint, so no new "predicate"
> > > property for SMTs, no new "negate" property for predicates, and no new
> > > "type" property for either. I'm not necessarily opposed to adding
> those,
> > > but it can be done without being pulled into the scope of this KIP.
> > >
> > > So to summarize how I imagine people might use the REST API after this
> > KIP:
> > >
> > > 1. Discover the set of plugins on the worker via GET
> > > /connector-plugins?connectorsOnly=false
> > > 2. For any of those plugins, discover the set of available versions by
> > > examining the recommended values for the "version" property after
> hitting
> > > the GET GET /connector-plugins/<plugin>/config endpoint
> > >
> > > I don't think this is necessarily optimal since plucking the "version"
> > > property out of the response from that endpoint might be a PITA for ad
> > hoc
> > > queries via, e.g., curl, but I think it'd be enough for this KIP. Still
> > > open to other options though, and if I've misunderstood anything in the
> > > proposal, please let me know!
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Wed, May 22, 2024 at 4:05 PM Greg Harris
> <greg.harris@aiven.io.invalid
> > >
> > > wrote:
> > >
> > > > Hey Chris,
> > > >
> > > > Thanks for your comments, and I'm glad that it seems like we're
> > > > aligning on the vision here.
> > > >
> > > > > An
> > > > > alternative could be to change existing behavior to fail fast on
> any
> > > > > invalid default converter configuration instead of just for invalid
> > > > > versions
> > > >
> > > > I suppose if this is landing in 4.0, we have the opportunity to break
> > > > compatibility and strictly validate the worker class configs, and
> > > > could have a new consistent behavior instead of inconsistent but
> > > > backwards-compatible behavior.
> > > >
> > > > > no other part of the KIP requires this change.
> > > >
> > > > This is correct, and a compelling argument. I'm fine leaving the
> > > > strict worker validation off of this configuration to potentially be
> > > > added later. If a KIP was raised to perform strict validation of the
> > > > worker config, it would include the version config, and can address
> > > > backwards compatibility for both configs together.
> > > >
> > > > > RE exposing the version property in the
> > > > /connector-plugins/<plugin>/config
> > > > > endpoint, the behavior is inconsistent across plugin types.
> > > >
> > > > Yes it is, and that started in a bugfix, once people started using
> > > > this endpoint: https://issues.apache.org/jira/browse/KAFKA-14843 . I
> > > > wasn't planning on converging the behavior in this KIP, as I
> > > > considered it out-of-scope, and was going to follow the current
> > > > behavior. To summarize:
> > > >
> > > > GET /connector-plugins/<connector>/config will emit:
> > > > * connector.class (already implemented)
> > > > * connector.version (new)
> > > > * key.converter (already implemented)
> > > > * key.converter.version (new)
> > > > * value.converter (already implemented)
> > > > * value.converter.version (new)
> > > > * header.converter (already implemented)
> > > > * header.converter.version (new)
> > > > But will NOT emit:
> > > > * transforms.<alias>.type
> > > > * transforms.<alias>.version
> > > > * transforms.<alias>.predicate
> > > > * predicates.<alias>.type
> > > > * predicates.<alias>.version
> > > > * predicates.<alias>.negate
> > > >
> > > > GET /connector-plugins/<converter>/config will NOT emit:
> > > > * "" (there's not even a ".class" prefix!)
> > > > * version
> > > >
> > > > GET /connector-plugins/<transform>/config will NOT emit:
> > > > * type
> > > > * version
> > > > * predicate
> > > >
> > > > GET /connector-plugins/<predicate>/config will NOT emit:
> > > > * type
> > > > * version
> > > > * negate
> > > >
> > > > Do you want the converter, transform, and predicate endpoints
> changed?
> > > > Do you want just "version", or do you want all of the prefixed
> configs
> > > > including "type", "predicate" and "negate"? How would you want to
> > > > handle the converters?
> > > > And when I say the configs are "not part of the plugin config itself"
> > > > I mean that saying that GET
> > > > /connector-plugins/Flatten$Key/config?version=3.8.0 has a "version"
> > > > config that must be "3.8.0" is a little bit nonsense, as the version
> > > > is already specified.
> > > >
> > > > > IMO
> > > > > it's worth including this information somewhere directly accessible
> > > > without
> > > > > having to provide a full connector config. FWIW I'd be fine with
> GET
> > > > > /connector-plugins/<plugin>/versions as a first-class endpoint
> > > >
> > > > You don't have to provide a configuration to call GET
> > > > /connector-plugins?connectorsOnly=false , is that endpoint not close
> > > > enough to what you have in mind? See also the Rejected Alternative
> > > > "Adding new REST API endpoints"
> > > >
> > > > If you're calling /connector-plugins/<transform>/config, you know the
> > > > name of a plugin right? That either comes from out-of-band knowledge,
> > > > validating a connector config, or calling GET
> > > > /connector-plugins?connectorsOnly=false.
> > > > * If you have out-of-band knowledge of plugin classes, perhaps you
> > > > have out-of-band knowledge of versions too.
> > > > * If you've just validated a connector config, there should be an
> > > > accompanying "version" field there with an accurate default value and
> > > > recommenders.
> > > > * If you've called GET /connector-plugins?connectorsOnly=false, that
> > > > endpoint includes version information.
> > > >
> > > > Thanks,
> > > > Greg
> > > >
> > > > On Wed, May 22, 2024 at 11:05 AM Chris Egerton
> <chrise@aiven.io.invalid
> > >
> > > > wrote:
> > > > >
> > > > > Hi Greg,
> > > > >
> > > > > Hope you had a nice weekend! Gonna try to keep things concise.
> > > > >
> > > > > Concluded points:
> > > > >
> > > > > RE version recommenders, I agree it's likely that programmatic UIs
> > will
> > > > > already be able to handle dynamic configuration definitions, and
> the
> > > > detail
> > > > > about SMTs is a great point. I still anticipate some awkwardness
> with
> > > > > connector versions, though: if the latest version supports some new
> > > > > properties, then a user switches to an earlier version, a UI may
> > > respond
> > > > by
> > > > > wiping values for these properties. I guess we can bite the bullet,
> > > > though.
> > > > >
> > > > > RE double-dinging during preflight validation for invalid
> versions, I
> > > > like
> > > > > the analogy with login credentials. I'm convinced that the proposal
> > in
> > > > the
> > > > > KIP is best 👍
> > > > >
> > > > > Continued points:
> > > > >
> > > > > RE failing on worker startup, sorry, I should be clearer: there is
> no
> > > > _new_
> > > > > justification for it that doesn't also apply to existing behavior.
> We
> > > > > shouldn't diverge from existing behavior solely for this new case.
> An
> > > > > alternative could be to change existing behavior to fail fast on
> any
> > > > > invalid default converter configuration instead of just for invalid
> > > > > versions, but I'd vote to just stick to existing behavior and not
> > > > > complicate things, especially since no other part of the KIP
> requires
> > > > this
> > > > > change.
> > > > >
> > > > > RE exposing the version property in the
> > > > /connector-plugins/<plugin>/config
> > > > > endpoint, the behavior is inconsistent across plugin types. Hitting
> > the
> > > > > endpoint for the FileStreamSinkConnector on version 3.7.0 yields a
> > > > response
> > > > > that includes, among other things, the "topics", "topics.regex",
> and
> > > > > "errors.tolerance" properties. I see that we don't do this
> everywhere
> > > > (the
> > > > > examples you cite for SMT and converter properties are accurate),
> but
> > > IMO
> > > > > it's worth including this information somewhere directly accessible
> > > > without
> > > > > having to provide a full connector config. FWIW I'd be fine with
> GET
> > > > > /connector-plugins/<plugin>/versions as a first-class endpoint
> either
> > > > > instead of or in addition to adding recommended values for all
> plugin
> > > > > versions.
> > > > >
> > > > > Thanks for your continued work on this KIP, and with the progress
> > we're
> > > > > making I'm optimistic about its chances of appearing in 4.0.0.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Wed, May 15, 2024 at 1:22 PM Greg Harris
> > > <greg.harris@aiven.io.invalid
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hey Chris,
> > > > > >
> > > > > > Thanks for your quick follow up.
> > > > > >
> > > > > > > But this risk is already present with
> > > > > > > existing error cases, and I don't see anything that justifies
> > > > changing
> > > > > > > existing behavior with an invalid converter class, or diverging
> > > from
> > > > it
> > > > > > in
> > > > > > > the case of invalid converter versions.
> > > > > >
> > > > > > The justification is to fail-fast, and prevent REST API users
> from
> > > > > > receiving errors from bad configs that they didn't write, or
> maybe
> > > > > > don't even know apply to them.
> > > > > > Up until recently errors in these configurations surfaced as
> > failures
> > > > > > to create the connector, or failures to start, and you made them
> > > > > > fail-fast during validation. I think this change is in the same
> > > > > > spirit, pulling the validation further forward and not letting
> > errors
> > > > > > lie dormant.
> > > > > >
> > > > > > And to call back to your original concern about interrupting
> > > > > > connectors that explicitly provide these configurations and don't
> > use
> > > > > > the worker configs: I expect that operators with a majority of
> > these
> > > > > > sorts of clients aren't going to be setting the worker .version
> > > > > > properties, because it would have no effect on the majority of
> > their
> > > > > > connectors. They would be able to rely on backwards-compatibility
> > and
> > > > > > continue to ignore the class properties.
> > > > > >
> > > > > > > CLI
> > > > > > > and programmatic UI developers will want to develop their own
> > > tooling
> > > > > > > layers.
> > > > > >
> > > > > > This is a very compelling argument. Snehashis do you want to
> figure
> > > > > > out a REST API design for this use-case?
> > > > > >
> > > > > > > Regarding the GET /connector-plugins/<plugin>/config endpoint,
> I
> > > was
> > > > > > > thinking about the response for non-connector plugins, e.g.,
> > > > > > > GET /connector-plugins/RegexRouter/config. Would a "version"
> > > property
> > > > > > > appear with recommended values?
> > > > > >
> > > > > > I intended for the ".version" property to be like other framework
> > > > > > configs (".class", ".type", ".predicate", ".negate") where they
> are
> > > > > > inside the plugin namespace, but not part of the plugin config
> > > itself.
> > > > > > Perhaps we can deviate from those configs because it would aid in
> > > > > > discovering other valid `GET
> > > > > > /connector-plugins/<plugin>/config?version=` calls, without
> calling
> > > > > > `GET /connector-plugins?connectorsOnly=false`.
> > > > > > I don't really feel strongly either way.
> > > > > >
> > > > > > > if a user changes the
> > > > > > > connector version in, e.g., a dropdown menu, then the UI either
> > has
> > > > to
> > > > > > > re-fetch the ConfigDef for the new version, or risk operating
> on
> > > > stale
> > > > > > > information
> > > > > >
> > > > > > This is a really interesting situation, thanks for finding that!
> > This
> > > > > > is already a footgun with transformations and predicates; Once
> you
> > > > > > fill out a transformation class (which includes a recommender)
> you
> > > > > > will then see all of the transformations configurations.
> > > > > > I think this means that UI developers should have already
> developed
> > > > > > the infrastructure for handling dynamic recommenders, or else
> > they've
> > > > > > had this bug since KIP-66 in 2017. It may require some manual
> > > > > > attention to roll-out support for the ".version" properties
> > > > > > specifically, but I don't think that should prevent us from
> > providing
> > > > > > this information in a natural place.
> > > > > >
> > > > > > > but will it be possible to configure
> > > > > > > a connector to use two instances of the same transform or
> > > predicate,
> > > > but
> > > > > > > with different versions for each?
> > > > > >
> > > > > > Yes, and this is included in the example in the KIP,
> > > > > > `transforms.flatten-latest` and `transforms.flatten-old`. This
> will
> > > > > > work in the natural way, with the two instances having a
> different
> > > > > > version if configured for that. If a user wants to pin the same
> > > > > > version of a plugin for every instance, they will need to provide
> > the
> > > > > > version config multiple times and keep them in-sync themselves.
> > > > > >
> > > > > > > I would have expected the error
> > > > > > > to only be attributed to the version property, and for the
> class
> > > > property
> > > > > > > to be reported as valid.
> > > > > >
> > > > > > I considered this, and then went back and changed it. To quote
> > > someone
> > > > > > else, this is to catch "misspelling cat as dog". For example a
> user
> > > > > > types DogConverter and meant to type CatConverter, and then
> types a
> > > > > > version which is valid for CatConverter, conceptually the error
> is
> > in
> > > > > > the class name and not the version.
> > > > > > That's a very contrived scenario, but I think similar arguments
> are
> > > > > > used for attributing validation errors to
> endpoints/urls/hostnames
> > > > > > when the associated credentials are unable to log into the remote
> > > > > > system. Did the user provide the correct testing credentials, but
> > > > > > accidentally typed the production endpoint? Even if the
> production
> > > > > > endpoint looks valid (it's a real hostname that is reachable) the
> > > > > > conceptual error is still in the hostname and should have the
> error
> > > > > > attributed to it to draw the user's attention.
> > > > > > If that's not convincing, I think the alternative of only
> > attributing
> > > > > > version errors to the version property is also acceptable.
> > > > > >
> > > > > > Thanks,
> > > > > > Greg
> > > > > >
> > > > > > On Wed, May 15, 2024 at 8:06 AM Chris Egerton
> > > <chrise@aiven.io.invalid
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Greg,
> > > > > > >
> > > > > > > Thanks for your responses! Continuations of existing
> discussions:
> > > > > > >
> > > > > > > Regarding crashing the worker on startup--yes, there is also a
> > risk
> > > > to
> > > > > > > allowing it to join the cluster. But this risk is already
> present
> > > > with
> > > > > > > existing error cases, and I don't see anything that justifies
> > > > changing
> > > > > > > existing behavior with an invalid converter class, or diverging
> > > from
> > > > it
> > > > > > in
> > > > > > > the case of invalid converter versions. I think we should keep
> > this
> > > > > > simple
> > > > > > > and not do anything different for worker startup than we do
> > > already.
> > > > > > >
> > > > > > > As far as REST API vs. metrics go--I think you're right that
> the
> > > > original
> > > > > > > version metrics were added as a "monitoring" detail. However,
> > this
> > > > was
> > > > > > back
> > > > > > > when plugin versions were managed solely by cluster
> > administrators.
> > > > With
> > > > > > > this KIP, connector users will be able to manage plugin
> versions,
> > > > and CLI
> > > > > > > and programmatic UI developers will want to develop their own
> > > tooling
> > > > > > > layers. I think focusing on the REST API as the primary
> interface
> > > for
> > > > > > this
> > > > > > > KIP would be best for these users.
> > > > > > >
> > > > > > > (All that said, I don't object to the metrics that are proposed
> > in
> > > > the
> > > > > > KIP;
> > > > > > > I just think they make more sense in addition to new REST API
> > > > > > > functionality, as opposed to instead of it.)
> > > > > > >
> > > > > > > Regarding the GET /connector-plugins/<plugin>/config endpoint,
> I
> > > was
> > > > > > > thinking about the response for non-connector plugins, e.g.,
> > > > > > > GET /connector-plugins/RegexRouter/config. Would a "version"
> > > property
> > > > > > > appear with recommended values?
> > > > > > >
> > > > > > >
> > > > > > > And new thoughts:
> > > > > > >
> > > > > > > 1) Regarding the recommended values for "connector.version",
> this
> > > > might
> > > > > > be
> > > > > > > confusing since there could be differences between the
> ConfigDefs
> > > > for the
> > > > > > > latest version of the connector and prior versions. It also
> makes
> > > the
> > > > > > flow
> > > > > > > a bit awkward for programmatic UI developers: if a user changes
> > the
> > > > > > > connector version in, e.g., a dropdown menu, then the UI either
> > has
> > > > to
> > > > > > > re-fetch the ConfigDef for the new version, or risk operating
> on
> > > > stale
> > > > > > > information. I'm starting to doubt that exposing the range of
> > > > available
> > > > > > > versions via recommended values is the best way to proceed,
> > instead
> > > > of a
> > > > > > > more explicit approach like GET
> > > > /connector-plugins/<plugin>/versions, or
> > > > > > > the "Adding new REST API endpoints" rejected alternative.
> > > > > > >
> > > > > > > 2) I know that this is a bit heinous, but will it be possible
> to
> > > > > > configure
> > > > > > > a connector to use two instances of the same transform or
> > > predicate,
> > > > but
> > > > > > > with different versions for each? (I don't think this is worth
> > > > > > significant
> > > > > > > design/implementation effort, so if it would inflate either of
> > > those,
> > > > > > > please don't feel obligated to support it.)
> > > > > > >
> > > > > > > 3) Just out of curiosity, why double-ding during config
> > validation
> > > > if the
> > > > > > > version for a plugin class can't be found? I would have
> expected
> > > the
> > > > > > error
> > > > > > > to only be attributed to the version property, and for the
> class
> > > > property
> > > > > > > to be reported as valid.
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Tue, May 14, 2024 at 1:11 PM Greg Harris
> > > > <greg.harris@aiven.io.invalid
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey Chris,
> > > > > > > >
> > > > > > > > Thanks for your comments. I'm glad you like the motivations,
> > > > Snehashis
> > > > > > > > wrote that part!
> > > > > > > >
> > > > > > > > > the configuration syntax for the most basic use case of
> > > > > > > > > specifying a single desired version is pretty
> > counterintuitive.
> > > > > > > >
> > > > > > > > I agree, and the "soft" requirement scheme is something I
> > wasn't
> > > > > > > > explicitly looking for, but would be inherited from the
> > library.
> > > > I'm
> > > > > > > > fine with eliminating the soft requirement semantics, and
> > having
> > > > > > > > "1.0.0" behave the same as "[1.0.0]". I'm less inclined to
> > > include
> > > > > > > > "range" in the property name, or have two properties.
> > > > > > > >
> > > > > > > > > Failing startup is drastic and has the potential to disrupt
> > the
> > > > > > > > > availability of connectors that would otherwise be able to
> > run
> > > > > > healthily
> > > > > > > > > because they were explicitly configured to use valid
> > converters
> > > > > > instead
> > > > > > > > of
> > > > > > > > > the worker defaults.
> > > > > > > >
> > > > > > > > I think this argument cuts both ways. If someone
> reconfigures a
> > > > worker
> > > > > > > > and adds an invalid ".version" string to the worker (or
> changes
> > > the
> > > > > > > > plugin.path to make it invalid), it would be permitted to
> enter
> > > the
> > > > > > > > group, and accept work assignments. If those work assignments
> > > used
> > > > > > > > these configurations, a set of tasks could transition to
> FAILED
> > > and
> > > > > > > > not be able to recover, because they would be restarted again
> > on
> > > > the
> > > > > > > > same worker.
> > > > > > > >
> > > > > > > > > Why are metrics utilized to report information about plugin
> > > > versions
> > > > > > > > > utilized by connectors at runtime instead of publishing
> this
> > > > info in
> > > > > > the
> > > > > > > > > REST API
> > > > > > > >
> > > > > > > > I was following the existing pattern for exposing runtime
> > > versions
> > > > for
> > > > > > > > connectors, and it did seem like a "monitoring" feature. If
> > that
> > > > > > > > approach is flawed and should be phased out, I think it would
> > be
> > > a
> > > > > > > > good idea to reconsider the REST API rejected alternative.
> > > > > > > > We would need some additional design work to spec out the
> REST
> > > API
> > > > > > > > interface, as I don't have anything in mind currently.
> > > > > > > >
> > > > > > > > > I'm unclear on whether or not it'll be possible to see this
> > > > > > information
> > > > > > > > via the
> > > > > > > > > GET /connector-plugins/<plugin>/config endpoint
> > > > > > > >
> > > > > > > > There is room in the API to add recommenders for
> > "key.converter",
> > > > > > > > "value.converter", and "header.converter", but not for
> > transforms
> > > > and
> > > > > > > > predicates, as they include aliases that depend on an actual
> > > > > > > > configuration. We could explicitly say we're going to do
> that,
> > or
> > > > do
> > > > > > > > whatever is convenient during the implementation phase, or
> > leave
> > > it
> > > > > > > > open to be improved later.
> > > > > > > > There will not be any recommenders for ".version" properties
> in
> > > the
> > > > > > > > `/config` endpoint, because those recommenders are dynamic
> and
> > > > depend
> > > > > > > > on an actual configuration.
> > > > > > > >
> > > > > > > > 5) There are two relevant lines in the KIP: "If a .version
> > > property
> > > > > > > > contains a hard requirement, select the latest installed
> > version
> > > > which
> > > > > > > > satisfies the requirement." and "This configuration is
> > > re-evaluated
> > > > > > > > each time the connector or task are assigned to a new
> worker".
> > I
> > > > would
> > > > > > > > call this "eager" upgrade behavior, rather than a "sticky" or
> > > > "lazy"
> > > > > > > > upgrade behavior.
> > > > > > > >
> > > > > > > > 6) Updated!
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Greg
> > > > > > > >
> > > > > > > > On Tue, May 14, 2024 at 9:14 AM Chris Egerton
> > > > <chrise@aiven.io.invalid
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > Thanks Greg for updating the KIP, and thanks Snehashis for
> > > > starting
> > > > > > the
> > > > > > > > > work on this originally.
> > > > > > > > >
> > > > > > > > > The motivation section makes a pretty convincing case for
> > this
> > > > kind
> > > > > > of
> > > > > > > > > feature. My thoughts are mostly about specific details:
> > > > > > > > >
> > > > > > > > > 1) I like the support for version ranges (the example
> > > > demonstrating
> > > > > > how
> > > > > > > > to
> > > > > > > > > avoid KAFKA-10574 with the header converter was
> particularly
> > > > > > > > entertaining),
> > > > > > > > > but the configuration syntax for the most basic use case of
> > > > > > specifying a
> > > > > > > > > single desired version is pretty counterintuitive. People
> may
> > > get
> > > > > > bitten
> > > > > > > > or
> > > > > > > > > at least frustrated if they put connector.version=3.8.0 in
> a
> > > > config
> > > > > > but
> > > > > > > > > then version 3.7.5 ends up running. I'd like it if we could
> > > > either
> > > > > > > > > intentionally deviate from Maven ranges when a bare version
> > is
> > > > > > present,
> > > > > > > > or
> > > > > > > > > separate things out into two properties: foo.version would
> be
> > > the
> > > > > > single
> > > > > > > > > accepted version for the foo plugin, and foo.version.range
> > > would
> > > > use
> > > > > > > > Maven
> > > > > > > > > range syntax. Open to other options too, just providing a
> > > couple
> > > > to
> > > > > > get
> > > > > > > > the
> > > > > > > > > ball rolling.
> > > > > > > > >
> > > > > > > > > 2) Although the current behavior for a worker with an
> invalid
> > > > > > > > > key/value/header converter class specified in its config
> file
> > > is
> > > > a
> > > > > > little
> > > > > > > > > strange (I was surprised to learn that it wouldn't fail on
> > > > startup),
> > > > > > I
> > > > > > > > > don't see a good reason to deviate from this when an
> invalid
> > > > version
> > > > > > is
> > > > > > > > > specified. Failing startup is drastic and has the potential
> > to
> > > > > > disrupt
> > > > > > > > the
> > > > > > > > > availability of connectors that would otherwise be able to
> > run
> > > > > > healthily
> > > > > > > > > because they were explicitly configured to use valid
> > converters
> > > > > > instead
> > > > > > > > of
> > > > > > > > > the worker defaults.
> > > > > > > > >
> > > > > > > > > 3) Why are metrics utilized to report information about
> > plugin
> > > > > > versions
> > > > > > > > > utilized by connectors at runtime instead of publishing
> this
> > > > info in
> > > > > > the
> > > > > > > > > REST API? I saw that this was mentioned as a rejected
> > > > alternative,
> > > > > > but I
> > > > > > > > > didn't get a sense of why. It seems like the REST API would
> > be
> > > > > > easier to
> > > > > > > > > access and more intuitive for most users than new metrics.
> > > > > > > > >
> > > > > > > > > 4) In the "Validation" section it's stated that "Users can
> > use
> > > > these
> > > > > > > > > recommenders to discover the valid plugin classes and
> > versions,
> > > > > > without
> > > > > > > > > requiring an earlier call to GET
> > > > > > > > /connector-plugins?connectorsOnly=false."
> > > > > > > > > I really like the creativity and simplicity of reusing the
> > > > > > recommender
> > > > > > > > > mechanism to expose available versions for plugins via the
> > REST
> > > > API.
> > > > > > I'm
> > > > > > > > > unclear on whether or not it'll be possible to see this
> > > > information
> > > > > > via
> > > > > > > > the
> > > > > > > > > GET /connector-plugins/<plugin>/config endpoint, though.
> It'd
> > > be
> > > > > > great if
> > > > > > > > > this were supported, since we learned in KIP-769 [1] that
> > > people
> > > > > > really
> > > > > > > > > want to be able to see configuration options for connectors
> > and
> > > > their
> > > > > > > > > plugins via some kind of GET endpoint without having to
> > > provide a
> > > > > > > > complete
> > > > > > > > > connector config for validation.
> > > > > > > > >
> > > > > > > > > 5) In the Maven version range docs, it's stated that "Maven
> > > > picks the
> > > > > > > > > highest version of each project that satisfies all the hard
> > > > > > requirements
> > > > > > > > of
> > > > > > > > > the dependencies on that project." I'm guessing this
> behavior
> > > > will be
> > > > > > > > > retained for Connect; i.e., the highest-possible version of
> > > each
> > > > > > plugin
> > > > > > > > > that satisfies the user-specified version constraints will
> be
> > > > run?
> > > > > > (An
> > > > > > > > > alternative approach could be to have some kind of "sticky"
> > > logic
> > > > > > that
> > > > > > > > only
> > > > > > > > > restarts connectors/tasks when their currently-used version
> > > > becomes
> > > > > > > > > incompatible with the configured constraints.)
> > > > > > > > >
> > > > > > > > > 6) (Nit) It'd be nice to add a link to the TestPlugins
> class
> > or
> > > > > > somewhere
> > > > > > > > > in its neighborhood to the testing plan; unfamiliar readers
> > > > probably
> > > > > > > > won't
> > > > > > > > > get much out of what's there right now.
> > > > > > > > >
> > > > > > > > > [1] -
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Mon, May 13, 2024 at 2:01 PM Snehashis <
> > > > snehashisp1995@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Greg,
> > > > > > > > > >
> > > > > > > > > > That is much appreciated. No complaints on the additional
> > > > scope, I
> > > > > > will
> > > > > > > > > > make some time out to work on this once we have approval.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Snehashis
> > > > > > > > > >
> > > > > > > > > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey Snehashis,
> > > > > > > > > > >
> > > > > > > > > > > I'm glad to hear you're still interested in this KIP!
> > > > > > > > > > > I'm happy to let you drive this, and I apologize for
> > > > increasing
> > > > > > the
> > > > > > > > > > > scope of work so drastically. To make up for that, I'll
> > > > > > volunteer to
> > > > > > > > > > > be the primary PR reviewer to help get this done
> quickly
> > > once
> > > > > > the KIP
> > > > > > > > > > > is approved.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Greg
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 10, 2024 at 3:51 AM Snehashis <
> > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the follow up to my original KIP, I am in
> > > > favour of
> > > > > > the
> > > > > > > > > > > > additions made to expand its scope, the addition of
> > range
> > > > > > versions
> > > > > > > > > > > > specifically make a lot of sense.
> > > > > > > > > > > >
> > > > > > > > > > > > Apologies if I have not publicly worked on this KIP
> > for a
> > > > long
> > > > > > > > time.
> > > > > > > > > > The
> > > > > > > > > > > > original work was done when the move to service
> loading
> > > > was in
> > > > > > > > > > discussion
> > > > > > > > > > > > and I wanted to loop back to this only after that
> work
> > > was
> > > > > > > > completed.
> > > > > > > > > > > Post
> > > > > > > > > > > > its conclusion, I have not been able to take this up
> > due
> > > to
> > > > > > other
> > > > > > > > > > > > priorities. If it's okay with you, I would still like
> > to
> > > > get
> > > > > > this
> > > > > > > > > > > > implemented myself, including the additional scope.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks and regards
> > > > > > > > > > > > Snehashis
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> > > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'd like to reboot the discussion on KIP-891:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > > > > > > > >
> > > > > > > > > > > > > I've made some changes, most notably:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Specifying versions for all plugins in Connector
> > > > configs
> > > > > > > > > > > > > (converters, header converters, transforms, and
> > > > predicates)
> > > > > > not
> > > > > > > > just
> > > > > > > > > > > > > connectors & tasks
> > > > > > > > > > > > > 2. Specifying a range of versions instead of an
> exact
> > > > match
> > > > > > > > > > > > > 3. New metrics to observe what versions are in-use
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks to Snehashis for the original KIP idea!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Greg
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <
> > > > > > > > greg.harris@aiven.io>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Snehashis,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you for the KIP! This is something I've
> > wanted
> > > > for a
> > > > > > long
> > > > > > > > > > time.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I know the discussion has gone cold, are you
> still
> > > > > > interested
> > > > > > > > in
> > > > > > > > > > > > > > pursuing this feature? I'll make time to review
> the
> > > > KIP if
> > > > > > you
> > > > > > > > are
> > > > > > > > > > > > > > still accepting comments.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis <
> > > > > > > > > > snehashisp1995@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the points Sagar.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1) Should we update the GET /connectors
> > endpoint
> > > to
> > > > > > > > include the
> > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > the plugin that is running? It could be
> useful
> > to
> > > > > > figure
> > > > > > > > out
> > > > > > > > > > the
> > > > > > > > > > > > > version
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the plugin or I am assuming it gets returned
> by
> > > the
> > > > > > > > expand=info
> > > > > > > > > > > call?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think this is good to have and possible
> future
> > > > > > > > enhancement. The
> > > > > > > > > > > > > version
> > > > > > > > > > > > > > > info will be present in the config of the
> > connector
> > > > if
> > > > > > the
> > > > > > > > user
> > > > > > > > > > has
> > > > > > > > > > > > > > > specified the version. Otherwise it is the
> latest
> > > > version
> > > > > > > > which
> > > > > > > > > > the
> > > > > > > > > > > > > user
> > > > > > > > > > > > > > > can find out from the connector-plugin
> endpoint.
> > > The
> > > > > > > > information
> > > > > > > > > > > can be
> > > > > > > > > > > > > > > introduced to the response of the GET
> /connectors
> > > > > > endpoint
> > > > > > > > > > itself,
> > > > > > > > > > > > > however
> > > > > > > > > > > > > > > the most ideal way of doing this would be to
> get
> > > the
> > > > > > > > currently
> > > > > > > > > > > running
> > > > > > > > > > > > > > > instance of the connector and get the version
> > > > directly
> > > > > > from
> > > > > > > > > > there.
> > > > > > > > > > > > > This is
> > > > > > > > > > > > > > > slightly tricky as the connector could be
> running
> > > in
> > > > a
> > > > > > > > different
> > > > > > > > > > > node.
> > > > > > > > > > > > > > > One way to do this would be to persist the
> > version
> > > > > > > > information in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > status backing store during instantiation of
> the
> > > > > > connector.
> > > > > > > > It
> > > > > > > > > > > requires
> > > > > > > > > > > > > > > some more thought and since the version is part
> > of
> > > > the
> > > > > > > > configs if
> > > > > > > > > > > > > provided
> > > > > > > > > > > > > > > and evident otherwise, I have not included it
> in
> > > this
> > > > > > KIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2) I am not aware of this and hence asking,
> > can 2
> > > > > > > > connectors
> > > > > > > > > > with
> > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > versions have the same name? Does the plugin
> > > > isolation
> > > > > > > > allow
> > > > > > > > > > > this?
> > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > could have a bearing when using the lifecycle
> > > > > > endpoints for
> > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > All connectors in a cluster need to have
> uniquire
> > > > > > connector
> > > > > > > > names
> > > > > > > > > > > > > > > regardless of what version of the plugin the
> > > > connector is
> > > > > > > > running
> > > > > > > > > > > > > > > underneath. This is something enforced by the
> > > connect
> > > > > > runtime
> > > > > > > > > > > itself.
> > > > > > > > > > > > > All
> > > > > > > > > > > > > > > connect CRUD operations are keyed on the
> > connector
> > > > name
> > > > > > so
> > > > > > > > there
> > > > > > > > > > > will
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > be an issue.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <
> > > > > > > > sagarmeansocean@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hey Snehashsih,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for the KIP. It looks like a very
> useful
> > > > > > feature.
> > > > > > > > Couple
> > > > > > > > > > > of
> > > > > > > > > > > > > > > > small-ish points, let me know what you think:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1) Should we update the GET /connectors
> > endpoint
> > > to
> > > > > > > > include the
> > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > the plugin that is running? It could be
> useful
> > to
> > > > > > figure
> > > > > > > > out
> > > > > > > > > > the
> > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > the plugin or I am assuming it gets returned
> by
> > > the
> > > > > > > > expand=info
> > > > > > > > > > > call?
> > > > > > > > > > > > > > > > 2) I am not aware of this and hence asking,
> > can 2
> > > > > > > > connectors
> > > > > > > > > > with
> > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > versions have the same name? Does the plugin
> > > > isolation
> > > > > > > > allow
> > > > > > > > > > > this?
> > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > could have a bearing when using the lifecycle
> > > > > > endpoints for
> > > > > > > > > > > > > connectors like
> > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > Sagar.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin
> > > > > > > > > > > <apankaj@confluent.io.invalid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > IIUC (please correct me if I am wrong
> > here),
> > > > what
> > > > > > you
> > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > a versioning scheme for a connector config
> > for
> > > > the
> > > > > > same
> > > > > > > > > > > connector
> > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > different versions of a connector plugin).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Sorry for not being more precise in my
> > wording
> > > > - I
> > > > > > meant
> > > > > > > > > > > > > registering
> > > > > > > > > > > > > > > > > versions of schema for connector config.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Let's take the example of a fictional
> > connector
> > > > which
> > > > > > > > uses a
> > > > > > > > > > > > > fictional
> > > > > > > > > > > > > > > > AWS
> > > > > > > > > > > > > > > > > service.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Fictional Connector Config schema
> version:2.0
> > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > "$schema": "
> > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Fictional Connector config schema
> version:3.0
> > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > "$schema": "
> > > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > "iam_role": {
> > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > "iam_role"
> > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The connector which supports Fictional
> config
> > > > schema
> > > > > > 2.0
> > > > > > > > > > will
> > > > > > > > > > > > > validate
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > access key and secret key.
> > > > > > > > > > > > > > > > > Whereas a connector which supports config
> > with
> > > > schema
> > > > > > > > version
> > > > > > > > > > > 3.0
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > validate the IAM role.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This is the alternative which I wanted to
> > > > suggest.
> > > > > > Each
> > > > > > > > > > plugin
> > > > > > > > > > > will
> > > > > > > > > > > > > > > > > register the schema versions of connector
> > > config
> > > > > > which it
> > > > > > > > > > > supports.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The plugin paths may be optionally
> different
> > > > i.e we
> > > > > > > > don't
> > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > mandatorily add a new plugin path to
> support
> > a
> > > > new
> > > > > > schema
> > > > > > > > > > > version.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis
> <
> > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. Can you elaborate on the rejected
> > > > > > alternatives ?
> > > > > > > > > > Suppose
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > config is versioned and has a schema.
> > Then
> > > a
> > > > > > single
> > > > > > > > > > plugin
> > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > dependencies have not changed) can
> handle
> > > > > > multiple
> > > > > > > > config
> > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > IIUC (please correct me if I am wrong
> > here),
> > > > what
> > > > > > you
> > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > a versioning scheme for a connector
> config
> > > for
> > > > the
> > > > > > same
> > > > > > > > > > > > > connector (and
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > different versions of a connector
> plugin).
> > > > That is
> > > > > > a
> > > > > > > > > > somewhat
> > > > > > > > > > > > > > > > tangential
> > > > > > > > > > > > > > > > > > problem. While it is definitely a useful
> > > > feature to
> > > > > > > > have,
> > > > > > > > > > > like a
> > > > > > > > > > > > > log to
> > > > > > > > > > > > > > > > > > check what changes were made over time to
> > the
> > > > > > config
> > > > > > > > which
> > > > > > > > > > > might
> > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > easier to do rollbacks, it is not the
> focus
> > > > here.
> > > > > > Here
> > > > > > > > by
> > > > > > > > > > > > > version we
> > > > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > > to say what underlying version of the
> > plugin
> > > > > > should the
> > > > > > > > > > given
> > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > of the connector use. Perhaps it is
> better
> > to
> > > > > > change
> > > > > > > > the
> > > > > > > > > > > name of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > parameter from connector.version to
> > > > > > > > > > connector.plugin.version
> > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > plugin.version if it was confusing. wdyt?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2. Any plans to support assisted
> > migration
> > > > e.g
> > > > > > if a
> > > > > > > > user
> > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > connector/config?migrate=latest", the
> > > latest
> > > > > > version
> > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > transform the existing config to the
> > newer
> > > > > > version.
> > > > > > > > This
> > > > > > > > > > > would
> > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > adding a method like "boolean
> > > migrate(Version
> > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This is an enhancement we can think of
> > doing
> > > in
> > > > > > future.
> > > > > > > > > > > Users can
> > > > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > a PUT call with the updated config which
> > has
> > > > the
> > > > > > > > updated
> > > > > > > > > > > version
> > > > > > > > > > > > > > > > number.
> > > > > > > > > > > > > > > > > > The assisted mode could be handy as the
> > user
> > > > does
> > > > > > not
> > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > > > know the
> > > > > > > > > > > > > > > > > > config but beyond this it does not seem
> to
> > > > justify
> > > > > > its
> > > > > > > > > > > existence.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin
> > > > > > > > > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This is a really useful feature and
> > thanks
> > > > for
> > > > > > > > initiating
> > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I had the following questions -
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. Can you elaborate on the rejected
> > > > > > alternatives ?
> > > > > > > > > > Suppose
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > config is versioned and has a schema.
> > Then
> > > a
> > > > > > single
> > > > > > > > > > plugin
> > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > dependencies have not changed) can
> handle
> > > > > > multiple
> > > > > > > > config
> > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2. Any plans to support assisted
> > migration
> > > > e.g
> > > > > > if a
> > > > > > > > user
> > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > connector/config?migrate=latest", the
> > > latest
> > > > > > version
> > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > transform the existing config to the
> > newer
> > > > > > version.
> > > > > > > > This
> > > > > > > > > > > would
> > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > adding a method like "boolean
> > > migrate(Version
> > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM
> > Snehashis <
> > > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I'd like to start a discussion thread
> > on
> > > > > > KIP-891:
> > > > > > > > > > Running
> > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The KIP aims to add the ability for
> the
> > > > connect
> > > > > > > > runtime
> > > > > > > > > > > to
> > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > versions of a connector.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Please take a look and let me know
> what
> > > you
> > > > > > think.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Chris Egerton <ch...@aiven.io.INVALID>.
Hi Greg,
First, an apology! I mistakenly assumed that each plugin appeared only once
in the responses from GET /connector-plugins?connectorsOnly=false. Thank
you for correcting me and pointing out that all versions of each plugin
appear in that response, which does indeed satisfy my desire for users to
discover this information in at most two REST requests (and in fact, does
it in only one)!
And secondly, with the revelation about recommenders, I agree that it's
best to leave the "version" property out of the lists of properties
returned from the GET /connector-plugins/<plugin>/config endpoint.
With those two points settled, I think the only unresolved item is the
small change to version parsing added to the KIP (where raw version numbers
are treated as an exact match, instead of a best-effort match with a
fallback on the default version). If the KIP is updated with that then I'd
be ready to vote on it.
Cheers,
Chris
On Wed, May 29, 2024 at 12:00 PM Greg Harris <gr...@aiven.io.invalid>
wrote:
> Hey Chris,
>
> Thanks for your thoughts.
>
> > Won't it still only expose the
> > latest version for each plugin, instead of the range of versions
> available?
>
> Here is a snippet of the current output of the GET
> /connector-plugins?connectorsOnly=false endpoint, after I installed two
> versions of the debezium PostgresConnector:
>
> {
> "class": "io.debezium.connector.postgresql.PostgresConnector",
> "type": "source",
> "version": "2.0.1.Final"
> },
> {
> "class": "io.debezium.connector.postgresql.PostgresConnector",
> "type": "source",
> "version": "2.6.1.Final"
> },
>
> I think this satisfies your requirement to learn about all plugins and all
> versions in two or fewer REST calls.
>
> I tried to get an example of the output of `/config` by hardcoding the
> Recommender, and realized that Recommenders aren't executed on the
> `/config` endpoint at all: only during validation, when a configuration is
> actually present.
> And this led me to discover that the `/config` endpoint returns a
> List<ConfigKeyInfo>, and ConfigKeyInfo does not contain a recommendedValues
> field. The ConfigValue field is the object which contains
> recommendedValues, and it is only generated during validation.
> I think it's out of scope to start calling recommenders on empty configs
> that might throw exceptions, changing the existing REST entities, or
> changing the core ConfigDef implementation.
> Someone could add this functionality later, I don't think it's necessary
> here.
>
> Then the question is: should "version" without recommenders appear in
> non-connector plugins? I think I'd rather be consistent with "predicate"
> and "negate" on release, and let a later improvement add them.
>
> Thanks,
> Greg
>
> On Wed, May 29, 2024 at 8:06 AM Chris Egerton <ch...@aiven.io.invalid>
> wrote:
>
> > Hi Greg,
> >
> > I'm confused about the behavior for GET
> > /connector-plugins?connectorsOnly=false. Won't it still only expose the
> > latest version for each plugin, instead of the range of versions
> available?
> >
> > I'm hoping we can provide a flow where people need at most two REST calls
> > to discover 1) the complete set of plugins available on the worker (which
> > is already possible with the endpoint under discussion) and 2) the set of
> > versions available for a specific plugin on the worker (which doesn't
> > appear to be possible, at least for some plugin types). This wouldn't
> > require any out-of-band knowledge and would be valuable for connector
> users
> > (who may want to, for example, know what their options are when
> considering
> > a plugin upgrade) and cluster administrators (who could use it as a
> sanity
> > check for the setup of their Kafka Connect clusters without having to
> pore
> > over log files).
> >
> > As far as modifying the content of the GET
> > /connector-plugins/<plugin>/config endpoint to include a "version"
> property
> > goes, I think your point about returning that property from requests to
> > that endpoint that include a version query parameter is salient, but it
> > also unfortunately applies to all types of plugin. I don't think it
> should
> > completely disqualify that option. I also wasn't imagining adding
> anything
> > besides that single property to that endpoint, so no new "predicate"
> > property for SMTs, no new "negate" property for predicates, and no new
> > "type" property for either. I'm not necessarily opposed to adding those,
> > but it can be done without being pulled into the scope of this KIP.
> >
> > So to summarize how I imagine people might use the REST API after this
> KIP:
> >
> > 1. Discover the set of plugins on the worker via GET
> > /connector-plugins?connectorsOnly=false
> > 2. For any of those plugins, discover the set of available versions by
> > examining the recommended values for the "version" property after hitting
> > the GET GET /connector-plugins/<plugin>/config endpoint
> >
> > I don't think this is necessarily optimal since plucking the "version"
> > property out of the response from that endpoint might be a PITA for ad
> hoc
> > queries via, e.g., curl, but I think it'd be enough for this KIP. Still
> > open to other options though, and if I've misunderstood anything in the
> > proposal, please let me know!
> >
> > Cheers,
> >
> > Chris
> >
> > On Wed, May 22, 2024 at 4:05 PM Greg Harris <greg.harris@aiven.io.invalid
> >
> > wrote:
> >
> > > Hey Chris,
> > >
> > > Thanks for your comments, and I'm glad that it seems like we're
> > > aligning on the vision here.
> > >
> > > > An
> > > > alternative could be to change existing behavior to fail fast on any
> > > > invalid default converter configuration instead of just for invalid
> > > > versions
> > >
> > > I suppose if this is landing in 4.0, we have the opportunity to break
> > > compatibility and strictly validate the worker class configs, and
> > > could have a new consistent behavior instead of inconsistent but
> > > backwards-compatible behavior.
> > >
> > > > no other part of the KIP requires this change.
> > >
> > > This is correct, and a compelling argument. I'm fine leaving the
> > > strict worker validation off of this configuration to potentially be
> > > added later. If a KIP was raised to perform strict validation of the
> > > worker config, it would include the version config, and can address
> > > backwards compatibility for both configs together.
> > >
> > > > RE exposing the version property in the
> > > /connector-plugins/<plugin>/config
> > > > endpoint, the behavior is inconsistent across plugin types.
> > >
> > > Yes it is, and that started in a bugfix, once people started using
> > > this endpoint: https://issues.apache.org/jira/browse/KAFKA-14843 . I
> > > wasn't planning on converging the behavior in this KIP, as I
> > > considered it out-of-scope, and was going to follow the current
> > > behavior. To summarize:
> > >
> > > GET /connector-plugins/<connector>/config will emit:
> > > * connector.class (already implemented)
> > > * connector.version (new)
> > > * key.converter (already implemented)
> > > * key.converter.version (new)
> > > * value.converter (already implemented)
> > > * value.converter.version (new)
> > > * header.converter (already implemented)
> > > * header.converter.version (new)
> > > But will NOT emit:
> > > * transforms.<alias>.type
> > > * transforms.<alias>.version
> > > * transforms.<alias>.predicate
> > > * predicates.<alias>.type
> > > * predicates.<alias>.version
> > > * predicates.<alias>.negate
> > >
> > > GET /connector-plugins/<converter>/config will NOT emit:
> > > * "" (there's not even a ".class" prefix!)
> > > * version
> > >
> > > GET /connector-plugins/<transform>/config will NOT emit:
> > > * type
> > > * version
> > > * predicate
> > >
> > > GET /connector-plugins/<predicate>/config will NOT emit:
> > > * type
> > > * version
> > > * negate
> > >
> > > Do you want the converter, transform, and predicate endpoints changed?
> > > Do you want just "version", or do you want all of the prefixed configs
> > > including "type", "predicate" and "negate"? How would you want to
> > > handle the converters?
> > > And when I say the configs are "not part of the plugin config itself"
> > > I mean that saying that GET
> > > /connector-plugins/Flatten$Key/config?version=3.8.0 has a "version"
> > > config that must be "3.8.0" is a little bit nonsense, as the version
> > > is already specified.
> > >
> > > > IMO
> > > > it's worth including this information somewhere directly accessible
> > > without
> > > > having to provide a full connector config. FWIW I'd be fine with GET
> > > > /connector-plugins/<plugin>/versions as a first-class endpoint
> > >
> > > You don't have to provide a configuration to call GET
> > > /connector-plugins?connectorsOnly=false , is that endpoint not close
> > > enough to what you have in mind? See also the Rejected Alternative
> > > "Adding new REST API endpoints"
> > >
> > > If you're calling /connector-plugins/<transform>/config, you know the
> > > name of a plugin right? That either comes from out-of-band knowledge,
> > > validating a connector config, or calling GET
> > > /connector-plugins?connectorsOnly=false.
> > > * If you have out-of-band knowledge of plugin classes, perhaps you
> > > have out-of-band knowledge of versions too.
> > > * If you've just validated a connector config, there should be an
> > > accompanying "version" field there with an accurate default value and
> > > recommenders.
> > > * If you've called GET /connector-plugins?connectorsOnly=false, that
> > > endpoint includes version information.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Wed, May 22, 2024 at 11:05 AM Chris Egerton <chrise@aiven.io.invalid
> >
> > > wrote:
> > > >
> > > > Hi Greg,
> > > >
> > > > Hope you had a nice weekend! Gonna try to keep things concise.
> > > >
> > > > Concluded points:
> > > >
> > > > RE version recommenders, I agree it's likely that programmatic UIs
> will
> > > > already be able to handle dynamic configuration definitions, and the
> > > detail
> > > > about SMTs is a great point. I still anticipate some awkwardness with
> > > > connector versions, though: if the latest version supports some new
> > > > properties, then a user switches to an earlier version, a UI may
> > respond
> > > by
> > > > wiping values for these properties. I guess we can bite the bullet,
> > > though.
> > > >
> > > > RE double-dinging during preflight validation for invalid versions, I
> > > like
> > > > the analogy with login credentials. I'm convinced that the proposal
> in
> > > the
> > > > KIP is best 👍
> > > >
> > > > Continued points:
> > > >
> > > > RE failing on worker startup, sorry, I should be clearer: there is no
> > > _new_
> > > > justification for it that doesn't also apply to existing behavior. We
> > > > shouldn't diverge from existing behavior solely for this new case. An
> > > > alternative could be to change existing behavior to fail fast on any
> > > > invalid default converter configuration instead of just for invalid
> > > > versions, but I'd vote to just stick to existing behavior and not
> > > > complicate things, especially since no other part of the KIP requires
> > > this
> > > > change.
> > > >
> > > > RE exposing the version property in the
> > > /connector-plugins/<plugin>/config
> > > > endpoint, the behavior is inconsistent across plugin types. Hitting
> the
> > > > endpoint for the FileStreamSinkConnector on version 3.7.0 yields a
> > > response
> > > > that includes, among other things, the "topics", "topics.regex", and
> > > > "errors.tolerance" properties. I see that we don't do this everywhere
> > > (the
> > > > examples you cite for SMT and converter properties are accurate), but
> > IMO
> > > > it's worth including this information somewhere directly accessible
> > > without
> > > > having to provide a full connector config. FWIW I'd be fine with GET
> > > > /connector-plugins/<plugin>/versions as a first-class endpoint either
> > > > instead of or in addition to adding recommended values for all plugin
> > > > versions.
> > > >
> > > > Thanks for your continued work on this KIP, and with the progress
> we're
> > > > making I'm optimistic about its chances of appearing in 4.0.0.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Wed, May 15, 2024 at 1:22 PM Greg Harris
> > <greg.harris@aiven.io.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hey Chris,
> > > > >
> > > > > Thanks for your quick follow up.
> > > > >
> > > > > > But this risk is already present with
> > > > > > existing error cases, and I don't see anything that justifies
> > > changing
> > > > > > existing behavior with an invalid converter class, or diverging
> > from
> > > it
> > > > > in
> > > > > > the case of invalid converter versions.
> > > > >
> > > > > The justification is to fail-fast, and prevent REST API users from
> > > > > receiving errors from bad configs that they didn't write, or maybe
> > > > > don't even know apply to them.
> > > > > Up until recently errors in these configurations surfaced as
> failures
> > > > > to create the connector, or failures to start, and you made them
> > > > > fail-fast during validation. I think this change is in the same
> > > > > spirit, pulling the validation further forward and not letting
> errors
> > > > > lie dormant.
> > > > >
> > > > > And to call back to your original concern about interrupting
> > > > > connectors that explicitly provide these configurations and don't
> use
> > > > > the worker configs: I expect that operators with a majority of
> these
> > > > > sorts of clients aren't going to be setting the worker .version
> > > > > properties, because it would have no effect on the majority of
> their
> > > > > connectors. They would be able to rely on backwards-compatibility
> and
> > > > > continue to ignore the class properties.
> > > > >
> > > > > > CLI
> > > > > > and programmatic UI developers will want to develop their own
> > tooling
> > > > > > layers.
> > > > >
> > > > > This is a very compelling argument. Snehashis do you want to figure
> > > > > out a REST API design for this use-case?
> > > > >
> > > > > > Regarding the GET /connector-plugins/<plugin>/config endpoint, I
> > was
> > > > > > thinking about the response for non-connector plugins, e.g.,
> > > > > > GET /connector-plugins/RegexRouter/config. Would a "version"
> > property
> > > > > > appear with recommended values?
> > > > >
> > > > > I intended for the ".version" property to be like other framework
> > > > > configs (".class", ".type", ".predicate", ".negate") where they are
> > > > > inside the plugin namespace, but not part of the plugin config
> > itself.
> > > > > Perhaps we can deviate from those configs because it would aid in
> > > > > discovering other valid `GET
> > > > > /connector-plugins/<plugin>/config?version=` calls, without calling
> > > > > `GET /connector-plugins?connectorsOnly=false`.
> > > > > I don't really feel strongly either way.
> > > > >
> > > > > > if a user changes the
> > > > > > connector version in, e.g., a dropdown menu, then the UI either
> has
> > > to
> > > > > > re-fetch the ConfigDef for the new version, or risk operating on
> > > stale
> > > > > > information
> > > > >
> > > > > This is a really interesting situation, thanks for finding that!
> This
> > > > > is already a footgun with transformations and predicates; Once you
> > > > > fill out a transformation class (which includes a recommender) you
> > > > > will then see all of the transformations configurations.
> > > > > I think this means that UI developers should have already developed
> > > > > the infrastructure for handling dynamic recommenders, or else
> they've
> > > > > had this bug since KIP-66 in 2017. It may require some manual
> > > > > attention to roll-out support for the ".version" properties
> > > > > specifically, but I don't think that should prevent us from
> providing
> > > > > this information in a natural place.
> > > > >
> > > > > > but will it be possible to configure
> > > > > > a connector to use two instances of the same transform or
> > predicate,
> > > but
> > > > > > with different versions for each?
> > > > >
> > > > > Yes, and this is included in the example in the KIP,
> > > > > `transforms.flatten-latest` and `transforms.flatten-old`. This will
> > > > > work in the natural way, with the two instances having a different
> > > > > version if configured for that. If a user wants to pin the same
> > > > > version of a plugin for every instance, they will need to provide
> the
> > > > > version config multiple times and keep them in-sync themselves.
> > > > >
> > > > > > I would have expected the error
> > > > > > to only be attributed to the version property, and for the class
> > > property
> > > > > > to be reported as valid.
> > > > >
> > > > > I considered this, and then went back and changed it. To quote
> > someone
> > > > > else, this is to catch "misspelling cat as dog". For example a user
> > > > > types DogConverter and meant to type CatConverter, and then types a
> > > > > version which is valid for CatConverter, conceptually the error is
> in
> > > > > the class name and not the version.
> > > > > That's a very contrived scenario, but I think similar arguments are
> > > > > used for attributing validation errors to endpoints/urls/hostnames
> > > > > when the associated credentials are unable to log into the remote
> > > > > system. Did the user provide the correct testing credentials, but
> > > > > accidentally typed the production endpoint? Even if the production
> > > > > endpoint looks valid (it's a real hostname that is reachable) the
> > > > > conceptual error is still in the hostname and should have the error
> > > > > attributed to it to draw the user's attention.
> > > > > If that's not convincing, I think the alternative of only
> attributing
> > > > > version errors to the version property is also acceptable.
> > > > >
> > > > > Thanks,
> > > > > Greg
> > > > >
> > > > > On Wed, May 15, 2024 at 8:06 AM Chris Egerton
> > <chrise@aiven.io.invalid
> > > >
> > > > > wrote:
> > > > > >
> > > > > > Hi Greg,
> > > > > >
> > > > > > Thanks for your responses! Continuations of existing discussions:
> > > > > >
> > > > > > Regarding crashing the worker on startup--yes, there is also a
> risk
> > > to
> > > > > > allowing it to join the cluster. But this risk is already present
> > > with
> > > > > > existing error cases, and I don't see anything that justifies
> > > changing
> > > > > > existing behavior with an invalid converter class, or diverging
> > from
> > > it
> > > > > in
> > > > > > the case of invalid converter versions. I think we should keep
> this
> > > > > simple
> > > > > > and not do anything different for worker startup than we do
> > already.
> > > > > >
> > > > > > As far as REST API vs. metrics go--I think you're right that the
> > > original
> > > > > > version metrics were added as a "monitoring" detail. However,
> this
> > > was
> > > > > back
> > > > > > when plugin versions were managed solely by cluster
> administrators.
> > > With
> > > > > > this KIP, connector users will be able to manage plugin versions,
> > > and CLI
> > > > > > and programmatic UI developers will want to develop their own
> > tooling
> > > > > > layers. I think focusing on the REST API as the primary interface
> > for
> > > > > this
> > > > > > KIP would be best for these users.
> > > > > >
> > > > > > (All that said, I don't object to the metrics that are proposed
> in
> > > the
> > > > > KIP;
> > > > > > I just think they make more sense in addition to new REST API
> > > > > > functionality, as opposed to instead of it.)
> > > > > >
> > > > > > Regarding the GET /connector-plugins/<plugin>/config endpoint, I
> > was
> > > > > > thinking about the response for non-connector plugins, e.g.,
> > > > > > GET /connector-plugins/RegexRouter/config. Would a "version"
> > property
> > > > > > appear with recommended values?
> > > > > >
> > > > > >
> > > > > > And new thoughts:
> > > > > >
> > > > > > 1) Regarding the recommended values for "connector.version", this
> > > might
> > > > > be
> > > > > > confusing since there could be differences between the ConfigDefs
> > > for the
> > > > > > latest version of the connector and prior versions. It also makes
> > the
> > > > > flow
> > > > > > a bit awkward for programmatic UI developers: if a user changes
> the
> > > > > > connector version in, e.g., a dropdown menu, then the UI either
> has
> > > to
> > > > > > re-fetch the ConfigDef for the new version, or risk operating on
> > > stale
> > > > > > information. I'm starting to doubt that exposing the range of
> > > available
> > > > > > versions via recommended values is the best way to proceed,
> instead
> > > of a
> > > > > > more explicit approach like GET
> > > /connector-plugins/<plugin>/versions, or
> > > > > > the "Adding new REST API endpoints" rejected alternative.
> > > > > >
> > > > > > 2) I know that this is a bit heinous, but will it be possible to
> > > > > configure
> > > > > > a connector to use two instances of the same transform or
> > predicate,
> > > but
> > > > > > with different versions for each? (I don't think this is worth
> > > > > significant
> > > > > > design/implementation effort, so if it would inflate either of
> > those,
> > > > > > please don't feel obligated to support it.)
> > > > > >
> > > > > > 3) Just out of curiosity, why double-ding during config
> validation
> > > if the
> > > > > > version for a plugin class can't be found? I would have expected
> > the
> > > > > error
> > > > > > to only be attributed to the version property, and for the class
> > > property
> > > > > > to be reported as valid.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Tue, May 14, 2024 at 1:11 PM Greg Harris
> > > <greg.harris@aiven.io.invalid
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hey Chris,
> > > > > > >
> > > > > > > Thanks for your comments. I'm glad you like the motivations,
> > > Snehashis
> > > > > > > wrote that part!
> > > > > > >
> > > > > > > > the configuration syntax for the most basic use case of
> > > > > > > > specifying a single desired version is pretty
> counterintuitive.
> > > > > > >
> > > > > > > I agree, and the "soft" requirement scheme is something I
> wasn't
> > > > > > > explicitly looking for, but would be inherited from the
> library.
> > > I'm
> > > > > > > fine with eliminating the soft requirement semantics, and
> having
> > > > > > > "1.0.0" behave the same as "[1.0.0]". I'm less inclined to
> > include
> > > > > > > "range" in the property name, or have two properties.
> > > > > > >
> > > > > > > > Failing startup is drastic and has the potential to disrupt
> the
> > > > > > > > availability of connectors that would otherwise be able to
> run
> > > > > healthily
> > > > > > > > because they were explicitly configured to use valid
> converters
> > > > > instead
> > > > > > > of
> > > > > > > > the worker defaults.
> > > > > > >
> > > > > > > I think this argument cuts both ways. If someone reconfigures a
> > > worker
> > > > > > > and adds an invalid ".version" string to the worker (or changes
> > the
> > > > > > > plugin.path to make it invalid), it would be permitted to enter
> > the
> > > > > > > group, and accept work assignments. If those work assignments
> > used
> > > > > > > these configurations, a set of tasks could transition to FAILED
> > and
> > > > > > > not be able to recover, because they would be restarted again
> on
> > > the
> > > > > > > same worker.
> > > > > > >
> > > > > > > > Why are metrics utilized to report information about plugin
> > > versions
> > > > > > > > utilized by connectors at runtime instead of publishing this
> > > info in
> > > > > the
> > > > > > > > REST API
> > > > > > >
> > > > > > > I was following the existing pattern for exposing runtime
> > versions
> > > for
> > > > > > > connectors, and it did seem like a "monitoring" feature. If
> that
> > > > > > > approach is flawed and should be phased out, I think it would
> be
> > a
> > > > > > > good idea to reconsider the REST API rejected alternative.
> > > > > > > We would need some additional design work to spec out the REST
> > API
> > > > > > > interface, as I don't have anything in mind currently.
> > > > > > >
> > > > > > > > I'm unclear on whether or not it'll be possible to see this
> > > > > information
> > > > > > > via the
> > > > > > > > GET /connector-plugins/<plugin>/config endpoint
> > > > > > >
> > > > > > > There is room in the API to add recommenders for
> "key.converter",
> > > > > > > "value.converter", and "header.converter", but not for
> transforms
> > > and
> > > > > > > predicates, as they include aliases that depend on an actual
> > > > > > > configuration. We could explicitly say we're going to do that,
> or
> > > do
> > > > > > > whatever is convenient during the implementation phase, or
> leave
> > it
> > > > > > > open to be improved later.
> > > > > > > There will not be any recommenders for ".version" properties in
> > the
> > > > > > > `/config` endpoint, because those recommenders are dynamic and
> > > depend
> > > > > > > on an actual configuration.
> > > > > > >
> > > > > > > 5) There are two relevant lines in the KIP: "If a .version
> > property
> > > > > > > contains a hard requirement, select the latest installed
> version
> > > which
> > > > > > > satisfies the requirement." and "This configuration is
> > re-evaluated
> > > > > > > each time the connector or task are assigned to a new worker".
> I
> > > would
> > > > > > > call this "eager" upgrade behavior, rather than a "sticky" or
> > > "lazy"
> > > > > > > upgrade behavior.
> > > > > > >
> > > > > > > 6) Updated!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Greg
> > > > > > >
> > > > > > > On Tue, May 14, 2024 at 9:14 AM Chris Egerton
> > > <chrise@aiven.io.invalid
> > > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Thanks Greg for updating the KIP, and thanks Snehashis for
> > > starting
> > > > > the
> > > > > > > > work on this originally.
> > > > > > > >
> > > > > > > > The motivation section makes a pretty convincing case for
> this
> > > kind
> > > > > of
> > > > > > > > feature. My thoughts are mostly about specific details:
> > > > > > > >
> > > > > > > > 1) I like the support for version ranges (the example
> > > demonstrating
> > > > > how
> > > > > > > to
> > > > > > > > avoid KAFKA-10574 with the header converter was particularly
> > > > > > > entertaining),
> > > > > > > > but the configuration syntax for the most basic use case of
> > > > > specifying a
> > > > > > > > single desired version is pretty counterintuitive. People may
> > get
> > > > > bitten
> > > > > > > or
> > > > > > > > at least frustrated if they put connector.version=3.8.0 in a
> > > config
> > > > > but
> > > > > > > > then version 3.7.5 ends up running. I'd like it if we could
> > > either
> > > > > > > > intentionally deviate from Maven ranges when a bare version
> is
> > > > > present,
> > > > > > > or
> > > > > > > > separate things out into two properties: foo.version would be
> > the
> > > > > single
> > > > > > > > accepted version for the foo plugin, and foo.version.range
> > would
> > > use
> > > > > > > Maven
> > > > > > > > range syntax. Open to other options too, just providing a
> > couple
> > > to
> > > > > get
> > > > > > > the
> > > > > > > > ball rolling.
> > > > > > > >
> > > > > > > > 2) Although the current behavior for a worker with an invalid
> > > > > > > > key/value/header converter class specified in its config file
> > is
> > > a
> > > > > little
> > > > > > > > strange (I was surprised to learn that it wouldn't fail on
> > > startup),
> > > > > I
> > > > > > > > don't see a good reason to deviate from this when an invalid
> > > version
> > > > > is
> > > > > > > > specified. Failing startup is drastic and has the potential
> to
> > > > > disrupt
> > > > > > > the
> > > > > > > > availability of connectors that would otherwise be able to
> run
> > > > > healthily
> > > > > > > > because they were explicitly configured to use valid
> converters
> > > > > instead
> > > > > > > of
> > > > > > > > the worker defaults.
> > > > > > > >
> > > > > > > > 3) Why are metrics utilized to report information about
> plugin
> > > > > versions
> > > > > > > > utilized by connectors at runtime instead of publishing this
> > > info in
> > > > > the
> > > > > > > > REST API? I saw that this was mentioned as a rejected
> > > alternative,
> > > > > but I
> > > > > > > > didn't get a sense of why. It seems like the REST API would
> be
> > > > > easier to
> > > > > > > > access and more intuitive for most users than new metrics.
> > > > > > > >
> > > > > > > > 4) In the "Validation" section it's stated that "Users can
> use
> > > these
> > > > > > > > recommenders to discover the valid plugin classes and
> versions,
> > > > > without
> > > > > > > > requiring an earlier call to GET
> > > > > > > /connector-plugins?connectorsOnly=false."
> > > > > > > > I really like the creativity and simplicity of reusing the
> > > > > recommender
> > > > > > > > mechanism to expose available versions for plugins via the
> REST
> > > API.
> > > > > I'm
> > > > > > > > unclear on whether or not it'll be possible to see this
> > > information
> > > > > via
> > > > > > > the
> > > > > > > > GET /connector-plugins/<plugin>/config endpoint, though. It'd
> > be
> > > > > great if
> > > > > > > > this were supported, since we learned in KIP-769 [1] that
> > people
> > > > > really
> > > > > > > > want to be able to see configuration options for connectors
> and
> > > their
> > > > > > > > plugins via some kind of GET endpoint without having to
> > provide a
> > > > > > > complete
> > > > > > > > connector config for validation.
> > > > > > > >
> > > > > > > > 5) In the Maven version range docs, it's stated that "Maven
> > > picks the
> > > > > > > > highest version of each project that satisfies all the hard
> > > > > requirements
> > > > > > > of
> > > > > > > > the dependencies on that project." I'm guessing this behavior
> > > will be
> > > > > > > > retained for Connect; i.e., the highest-possible version of
> > each
> > > > > plugin
> > > > > > > > that satisfies the user-specified version constraints will be
> > > run?
> > > > > (An
> > > > > > > > alternative approach could be to have some kind of "sticky"
> > logic
> > > > > that
> > > > > > > only
> > > > > > > > restarts connectors/tasks when their currently-used version
> > > becomes
> > > > > > > > incompatible with the configured constraints.)
> > > > > > > >
> > > > > > > > 6) (Nit) It'd be nice to add a link to the TestPlugins class
> or
> > > > > somewhere
> > > > > > > > in its neighborhood to the testing plan; unfamiliar readers
> > > probably
> > > > > > > won't
> > > > > > > > get much out of what's there right now.
> > > > > > > >
> > > > > > > > [1] -
> > > > > > > >
> > > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Mon, May 13, 2024 at 2:01 PM Snehashis <
> > > snehashisp1995@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Greg,
> > > > > > > > >
> > > > > > > > > That is much appreciated. No complaints on the additional
> > > scope, I
> > > > > will
> > > > > > > > > make some time out to work on this once we have approval.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Snehashis
> > > > > > > > >
> > > > > > > > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hey Snehashis,
> > > > > > > > > >
> > > > > > > > > > I'm glad to hear you're still interested in this KIP!
> > > > > > > > > > I'm happy to let you drive this, and I apologize for
> > > increasing
> > > > > the
> > > > > > > > > > scope of work so drastically. To make up for that, I'll
> > > > > volunteer to
> > > > > > > > > > be the primary PR reviewer to help get this done quickly
> > once
> > > > > the KIP
> > > > > > > > > > is approved.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Greg
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, May 10, 2024 at 3:51 AM Snehashis <
> > > > > snehashisp1995@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Greg,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the follow up to my original KIP, I am in
> > > favour of
> > > > > the
> > > > > > > > > > > additions made to expand its scope, the addition of
> range
> > > > > versions
> > > > > > > > > > > specifically make a lot of sense.
> > > > > > > > > > >
> > > > > > > > > > > Apologies if I have not publicly worked on this KIP
> for a
> > > long
> > > > > > > time.
> > > > > > > > > The
> > > > > > > > > > > original work was done when the move to service loading
> > > was in
> > > > > > > > > discussion
> > > > > > > > > > > and I wanted to loop back to this only after that work
> > was
> > > > > > > completed.
> > > > > > > > > > Post
> > > > > > > > > > > its conclusion, I have not been able to take this up
> due
> > to
> > > > > other
> > > > > > > > > > > priorities. If it's okay with you, I would still like
> to
> > > get
> > > > > this
> > > > > > > > > > > implemented myself, including the additional scope.
> > > > > > > > > > >
> > > > > > > > > > > Thanks and regards
> > > > > > > > > > > Snehashis
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> > > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I'd like to reboot the discussion on KIP-891:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > > > > > > >
> > > > > > > > > > > > I've made some changes, most notably:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Specifying versions for all plugins in Connector
> > > configs
> > > > > > > > > > > > (converters, header converters, transforms, and
> > > predicates)
> > > > > not
> > > > > > > just
> > > > > > > > > > > > connectors & tasks
> > > > > > > > > > > > 2. Specifying a range of versions instead of an exact
> > > match
> > > > > > > > > > > > 3. New metrics to observe what versions are in-use
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks to Snehashis for the original KIP idea!
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Greg
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <
> > > > > > > greg.harris@aiven.io>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Snehashis,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you for the KIP! This is something I've
> wanted
> > > for a
> > > > > long
> > > > > > > > > time.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I know the discussion has gone cold, are you still
> > > > > interested
> > > > > > > in
> > > > > > > > > > > > > pursuing this feature? I'll make time to review the
> > > KIP if
> > > > > you
> > > > > > > are
> > > > > > > > > > > > > still accepting comments.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Greg
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis <
> > > > > > > > > snehashisp1995@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the points Sagar.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1) Should we update the GET /connectors
> endpoint
> > to
> > > > > > > include the
> > > > > > > > > > > > version of
> > > > > > > > > > > > > > > the plugin that is running? It could be useful
> to
> > > > > figure
> > > > > > > out
> > > > > > > > > the
> > > > > > > > > > > > version
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the plugin or I am assuming it gets returned by
> > the
> > > > > > > expand=info
> > > > > > > > > > call?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think this is good to have and possible future
> > > > > > > enhancement. The
> > > > > > > > > > > > version
> > > > > > > > > > > > > > info will be present in the config of the
> connector
> > > if
> > > > > the
> > > > > > > user
> > > > > > > > > has
> > > > > > > > > > > > > > specified the version. Otherwise it is the latest
> > > version
> > > > > > > which
> > > > > > > > > the
> > > > > > > > > > > > user
> > > > > > > > > > > > > > can find out from the connector-plugin endpoint.
> > The
> > > > > > > information
> > > > > > > > > > can be
> > > > > > > > > > > > > > introduced to the response of the GET /connectors
> > > > > endpoint
> > > > > > > > > itself,
> > > > > > > > > > > > however
> > > > > > > > > > > > > > the most ideal way of doing this would be to get
> > the
> > > > > > > currently
> > > > > > > > > > running
> > > > > > > > > > > > > > instance of the connector and get the version
> > > directly
> > > > > from
> > > > > > > > > there.
> > > > > > > > > > > > This is
> > > > > > > > > > > > > > slightly tricky as the connector could be running
> > in
> > > a
> > > > > > > different
> > > > > > > > > > node.
> > > > > > > > > > > > > > One way to do this would be to persist the
> version
> > > > > > > information in
> > > > > > > > > > the
> > > > > > > > > > > > > > status backing store during instantiation of the
> > > > > connector.
> > > > > > > It
> > > > > > > > > > requires
> > > > > > > > > > > > > > some more thought and since the version is part
> of
> > > the
> > > > > > > configs if
> > > > > > > > > > > > provided
> > > > > > > > > > > > > > and evident otherwise, I have not included it in
> > this
> > > > > KIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2) I am not aware of this and hence asking,
> can 2
> > > > > > > connectors
> > > > > > > > > with
> > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > versions have the same name? Does the plugin
> > > isolation
> > > > > > > allow
> > > > > > > > > > this?
> > > > > > > > > > > > This
> > > > > > > > > > > > > > > could have a bearing when using the lifecycle
> > > > > endpoints for
> > > > > > > > > > > > connectors
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > All connectors in a cluster need to have uniquire
> > > > > connector
> > > > > > > names
> > > > > > > > > > > > > > regardless of what version of the plugin the
> > > connector is
> > > > > > > running
> > > > > > > > > > > > > > underneath. This is something enforced by the
> > connect
> > > > > runtime
> > > > > > > > > > itself.
> > > > > > > > > > > > All
> > > > > > > > > > > > > > connect CRUD operations are keyed on the
> connector
> > > name
> > > > > so
> > > > > > > there
> > > > > > > > > > will
> > > > > > > > > > > > not
> > > > > > > > > > > > > > be an issue.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <
> > > > > > > sagarmeansocean@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hey Snehashsih,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the KIP. It looks like a very useful
> > > > > feature.
> > > > > > > Couple
> > > > > > > > > > of
> > > > > > > > > > > > > > > small-ish points, let me know what you think:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1) Should we update the GET /connectors
> endpoint
> > to
> > > > > > > include the
> > > > > > > > > > > > version of
> > > > > > > > > > > > > > > the plugin that is running? It could be useful
> to
> > > > > figure
> > > > > > > out
> > > > > > > > > the
> > > > > > > > > > > > version of
> > > > > > > > > > > > > > > the plugin or I am assuming it gets returned by
> > the
> > > > > > > expand=info
> > > > > > > > > > call?
> > > > > > > > > > > > > > > 2) I am not aware of this and hence asking,
> can 2
> > > > > > > connectors
> > > > > > > > > with
> > > > > > > > > > > > different
> > > > > > > > > > > > > > > versions have the same name? Does the plugin
> > > isolation
> > > > > > > allow
> > > > > > > > > > this?
> > > > > > > > > > > > This
> > > > > > > > > > > > > > > could have a bearing when using the lifecycle
> > > > > endpoints for
> > > > > > > > > > > > connectors like
> > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > Sagar.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin
> > > > > > > > > > <apankaj@confluent.io.invalid
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > IIUC (please correct me if I am wrong
> here),
> > > what
> > > > > you
> > > > > > > > > > highlighted
> > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > a versioning scheme for a connector config
> for
> > > the
> > > > > same
> > > > > > > > > > connector
> > > > > > > > > > > > (and
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > different versions of a connector plugin).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Sorry for not being more precise in my
> wording
> > > - I
> > > > > meant
> > > > > > > > > > > > registering
> > > > > > > > > > > > > > > > versions of schema for connector config.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Let's take the example of a fictional
> connector
> > > which
> > > > > > > uses a
> > > > > > > > > > > > fictional
> > > > > > > > > > > > > > > AWS
> > > > > > > > > > > > > > > > service.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Fictional Connector Config schema version:2.0
> > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > "$schema": "
> > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Fictional Connector config schema version:3.0
> > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > "$schema": "
> > > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > "iam_role": {
> > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > "iam_role"
> > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The connector which supports Fictional config
> > > schema
> > > > > 2.0
> > > > > > > > > will
> > > > > > > > > > > > validate
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > access key and secret key.
> > > > > > > > > > > > > > > > Whereas a connector which supports config
> with
> > > schema
> > > > > > > version
> > > > > > > > > > 3.0
> > > > > > > > > > > > will
> > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > validate the IAM role.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This is the alternative which I wanted to
> > > suggest.
> > > > > Each
> > > > > > > > > plugin
> > > > > > > > > > will
> > > > > > > > > > > > > > > > register the schema versions of connector
> > config
> > > > > which it
> > > > > > > > > > supports.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The plugin paths may be optionally different
> > > i.e we
> > > > > > > don't
> > > > > > > > > > have to
> > > > > > > > > > > > > > > > mandatorily add a new plugin path to support
> a
> > > new
> > > > > schema
> > > > > > > > > > version.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <
> > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. Can you elaborate on the rejected
> > > > > alternatives ?
> > > > > > > > > Suppose
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > config is versioned and has a schema.
> Then
> > a
> > > > > single
> > > > > > > > > plugin
> > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > dependencies have not changed) can handle
> > > > > multiple
> > > > > > > config
> > > > > > > > > > > > versions
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > IIUC (please correct me if I am wrong
> here),
> > > what
> > > > > you
> > > > > > > > > > highlighted
> > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > a versioning scheme for a connector config
> > for
> > > the
> > > > > same
> > > > > > > > > > > > connector (and
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > different versions of a connector plugin).
> > > That is
> > > > > a
> > > > > > > > > somewhat
> > > > > > > > > > > > > > > tangential
> > > > > > > > > > > > > > > > > problem. While it is definitely a useful
> > > feature to
> > > > > > > have,
> > > > > > > > > > like a
> > > > > > > > > > > > log to
> > > > > > > > > > > > > > > > > check what changes were made over time to
> the
> > > > > config
> > > > > > > which
> > > > > > > > > > might
> > > > > > > > > > > > make
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > easier to do rollbacks, it is not the focus
> > > here.
> > > > > Here
> > > > > > > by
> > > > > > > > > > > > version we
> > > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > to say what underlying version of the
> plugin
> > > > > should the
> > > > > > > > > given
> > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > of the connector use. Perhaps it is better
> to
> > > > > change
> > > > > > > the
> > > > > > > > > > name of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > parameter from connector.version to
> > > > > > > > > connector.plugin.version
> > > > > > > > > > or
> > > > > > > > > > > > > > > > > plugin.version if it was confusing. wdyt?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2. Any plans to support assisted
> migration
> > > e.g
> > > > > if a
> > > > > > > user
> > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > connector/config?migrate=latest", the
> > latest
> > > > > version
> > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > transform the existing config to the
> newer
> > > > > version.
> > > > > > > This
> > > > > > > > > > would
> > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > adding a method like "boolean
> > migrate(Version
> > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This is an enhancement we can think of
> doing
> > in
> > > > > future.
> > > > > > > > > > Users can
> > > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > a PUT call with the updated config which
> has
> > > the
> > > > > > > updated
> > > > > > > > > > version
> > > > > > > > > > > > > > > number.
> > > > > > > > > > > > > > > > > The assisted mode could be handy as the
> user
> > > does
> > > > > not
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > > know the
> > > > > > > > > > > > > > > > > config but beyond this it does not seem to
> > > justify
> > > > > its
> > > > > > > > > > existence.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin
> > > > > > > > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This is a really useful feature and
> thanks
> > > for
> > > > > > > initiating
> > > > > > > > > > this
> > > > > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I had the following questions -
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. Can you elaborate on the rejected
> > > > > alternatives ?
> > > > > > > > > Suppose
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > config is versioned and has a schema.
> Then
> > a
> > > > > single
> > > > > > > > > plugin
> > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > dependencies have not changed) can handle
> > > > > multiple
> > > > > > > config
> > > > > > > > > > > > versions
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2. Any plans to support assisted
> migration
> > > e.g
> > > > > if a
> > > > > > > user
> > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > connector/config?migrate=latest", the
> > latest
> > > > > version
> > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > transform the existing config to the
> newer
> > > > > version.
> > > > > > > This
> > > > > > > > > > would
> > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > adding a method like "boolean
> > migrate(Version
> > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM
> Snehashis <
> > > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I'd like to start a discussion thread
> on
> > > > > KIP-891:
> > > > > > > > > Running
> > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The KIP aims to add the ability for the
> > > connect
> > > > > > > runtime
> > > > > > > > > > to
> > > > > > > > > > > > run
> > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > versions of a connector.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Please take a look and let me know what
> > you
> > > > > think.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> > >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Hey Chris,
Thanks for your thoughts.
> Won't it still only expose the
> latest version for each plugin, instead of the range of versions
available?
Here is a snippet of the current output of the GET
/connector-plugins?connectorsOnly=false endpoint, after I installed two
versions of the debezium PostgresConnector:
{
"class": "io.debezium.connector.postgresql.PostgresConnector",
"type": "source",
"version": "2.0.1.Final"
},
{
"class": "io.debezium.connector.postgresql.PostgresConnector",
"type": "source",
"version": "2.6.1.Final"
},
I think this satisfies your requirement to learn about all plugins and all
versions in two or fewer REST calls.
I tried to get an example of the output of `/config` by hardcoding the
Recommender, and realized that Recommenders aren't executed on the
`/config` endpoint at all: only during validation, when a configuration is
actually present.
And this led me to discover that the `/config` endpoint returns a
List<ConfigKeyInfo>, and ConfigKeyInfo does not contain a recommendedValues
field. The ConfigValue field is the object which contains
recommendedValues, and it is only generated during validation.
I think it's out of scope to start calling recommenders on empty configs
that might throw exceptions, changing the existing REST entities, or
changing the core ConfigDef implementation.
Someone could add this functionality later, I don't think it's necessary
here.
Then the question is: should "version" without recommenders appear in
non-connector plugins? I think I'd rather be consistent with "predicate"
and "negate" on release, and let a later improvement add them.
Thanks,
Greg
On Wed, May 29, 2024 at 8:06 AM Chris Egerton <ch...@aiven.io.invalid>
wrote:
> Hi Greg,
>
> I'm confused about the behavior for GET
> /connector-plugins?connectorsOnly=false. Won't it still only expose the
> latest version for each plugin, instead of the range of versions available?
>
> I'm hoping we can provide a flow where people need at most two REST calls
> to discover 1) the complete set of plugins available on the worker (which
> is already possible with the endpoint under discussion) and 2) the set of
> versions available for a specific plugin on the worker (which doesn't
> appear to be possible, at least for some plugin types). This wouldn't
> require any out-of-band knowledge and would be valuable for connector users
> (who may want to, for example, know what their options are when considering
> a plugin upgrade) and cluster administrators (who could use it as a sanity
> check for the setup of their Kafka Connect clusters without having to pore
> over log files).
>
> As far as modifying the content of the GET
> /connector-plugins/<plugin>/config endpoint to include a "version" property
> goes, I think your point about returning that property from requests to
> that endpoint that include a version query parameter is salient, but it
> also unfortunately applies to all types of plugin. I don't think it should
> completely disqualify that option. I also wasn't imagining adding anything
> besides that single property to that endpoint, so no new "predicate"
> property for SMTs, no new "negate" property for predicates, and no new
> "type" property for either. I'm not necessarily opposed to adding those,
> but it can be done without being pulled into the scope of this KIP.
>
> So to summarize how I imagine people might use the REST API after this KIP:
>
> 1. Discover the set of plugins on the worker via GET
> /connector-plugins?connectorsOnly=false
> 2. For any of those plugins, discover the set of available versions by
> examining the recommended values for the "version" property after hitting
> the GET GET /connector-plugins/<plugin>/config endpoint
>
> I don't think this is necessarily optimal since plucking the "version"
> property out of the response from that endpoint might be a PITA for ad hoc
> queries via, e.g., curl, but I think it'd be enough for this KIP. Still
> open to other options though, and if I've misunderstood anything in the
> proposal, please let me know!
>
> Cheers,
>
> Chris
>
> On Wed, May 22, 2024 at 4:05 PM Greg Harris <gr...@aiven.io.invalid>
> wrote:
>
> > Hey Chris,
> >
> > Thanks for your comments, and I'm glad that it seems like we're
> > aligning on the vision here.
> >
> > > An
> > > alternative could be to change existing behavior to fail fast on any
> > > invalid default converter configuration instead of just for invalid
> > > versions
> >
> > I suppose if this is landing in 4.0, we have the opportunity to break
> > compatibility and strictly validate the worker class configs, and
> > could have a new consistent behavior instead of inconsistent but
> > backwards-compatible behavior.
> >
> > > no other part of the KIP requires this change.
> >
> > This is correct, and a compelling argument. I'm fine leaving the
> > strict worker validation off of this configuration to potentially be
> > added later. If a KIP was raised to perform strict validation of the
> > worker config, it would include the version config, and can address
> > backwards compatibility for both configs together.
> >
> > > RE exposing the version property in the
> > /connector-plugins/<plugin>/config
> > > endpoint, the behavior is inconsistent across plugin types.
> >
> > Yes it is, and that started in a bugfix, once people started using
> > this endpoint: https://issues.apache.org/jira/browse/KAFKA-14843 . I
> > wasn't planning on converging the behavior in this KIP, as I
> > considered it out-of-scope, and was going to follow the current
> > behavior. To summarize:
> >
> > GET /connector-plugins/<connector>/config will emit:
> > * connector.class (already implemented)
> > * connector.version (new)
> > * key.converter (already implemented)
> > * key.converter.version (new)
> > * value.converter (already implemented)
> > * value.converter.version (new)
> > * header.converter (already implemented)
> > * header.converter.version (new)
> > But will NOT emit:
> > * transforms.<alias>.type
> > * transforms.<alias>.version
> > * transforms.<alias>.predicate
> > * predicates.<alias>.type
> > * predicates.<alias>.version
> > * predicates.<alias>.negate
> >
> > GET /connector-plugins/<converter>/config will NOT emit:
> > * "" (there's not even a ".class" prefix!)
> > * version
> >
> > GET /connector-plugins/<transform>/config will NOT emit:
> > * type
> > * version
> > * predicate
> >
> > GET /connector-plugins/<predicate>/config will NOT emit:
> > * type
> > * version
> > * negate
> >
> > Do you want the converter, transform, and predicate endpoints changed?
> > Do you want just "version", or do you want all of the prefixed configs
> > including "type", "predicate" and "negate"? How would you want to
> > handle the converters?
> > And when I say the configs are "not part of the plugin config itself"
> > I mean that saying that GET
> > /connector-plugins/Flatten$Key/config?version=3.8.0 has a "version"
> > config that must be "3.8.0" is a little bit nonsense, as the version
> > is already specified.
> >
> > > IMO
> > > it's worth including this information somewhere directly accessible
> > without
> > > having to provide a full connector config. FWIW I'd be fine with GET
> > > /connector-plugins/<plugin>/versions as a first-class endpoint
> >
> > You don't have to provide a configuration to call GET
> > /connector-plugins?connectorsOnly=false , is that endpoint not close
> > enough to what you have in mind? See also the Rejected Alternative
> > "Adding new REST API endpoints"
> >
> > If you're calling /connector-plugins/<transform>/config, you know the
> > name of a plugin right? That either comes from out-of-band knowledge,
> > validating a connector config, or calling GET
> > /connector-plugins?connectorsOnly=false.
> > * If you have out-of-band knowledge of plugin classes, perhaps you
> > have out-of-band knowledge of versions too.
> > * If you've just validated a connector config, there should be an
> > accompanying "version" field there with an accurate default value and
> > recommenders.
> > * If you've called GET /connector-plugins?connectorsOnly=false, that
> > endpoint includes version information.
> >
> > Thanks,
> > Greg
> >
> > On Wed, May 22, 2024 at 11:05 AM Chris Egerton <ch...@aiven.io.invalid>
> > wrote:
> > >
> > > Hi Greg,
> > >
> > > Hope you had a nice weekend! Gonna try to keep things concise.
> > >
> > > Concluded points:
> > >
> > > RE version recommenders, I agree it's likely that programmatic UIs will
> > > already be able to handle dynamic configuration definitions, and the
> > detail
> > > about SMTs is a great point. I still anticipate some awkwardness with
> > > connector versions, though: if the latest version supports some new
> > > properties, then a user switches to an earlier version, a UI may
> respond
> > by
> > > wiping values for these properties. I guess we can bite the bullet,
> > though.
> > >
> > > RE double-dinging during preflight validation for invalid versions, I
> > like
> > > the analogy with login credentials. I'm convinced that the proposal in
> > the
> > > KIP is best 👍
> > >
> > > Continued points:
> > >
> > > RE failing on worker startup, sorry, I should be clearer: there is no
> > _new_
> > > justification for it that doesn't also apply to existing behavior. We
> > > shouldn't diverge from existing behavior solely for this new case. An
> > > alternative could be to change existing behavior to fail fast on any
> > > invalid default converter configuration instead of just for invalid
> > > versions, but I'd vote to just stick to existing behavior and not
> > > complicate things, especially since no other part of the KIP requires
> > this
> > > change.
> > >
> > > RE exposing the version property in the
> > /connector-plugins/<plugin>/config
> > > endpoint, the behavior is inconsistent across plugin types. Hitting the
> > > endpoint for the FileStreamSinkConnector on version 3.7.0 yields a
> > response
> > > that includes, among other things, the "topics", "topics.regex", and
> > > "errors.tolerance" properties. I see that we don't do this everywhere
> > (the
> > > examples you cite for SMT and converter properties are accurate), but
> IMO
> > > it's worth including this information somewhere directly accessible
> > without
> > > having to provide a full connector config. FWIW I'd be fine with GET
> > > /connector-plugins/<plugin>/versions as a first-class endpoint either
> > > instead of or in addition to adding recommended values for all plugin
> > > versions.
> > >
> > > Thanks for your continued work on this KIP, and with the progress we're
> > > making I'm optimistic about its chances of appearing in 4.0.0.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Wed, May 15, 2024 at 1:22 PM Greg Harris
> <greg.harris@aiven.io.invalid
> > >
> > > wrote:
> > >
> > > > Hey Chris,
> > > >
> > > > Thanks for your quick follow up.
> > > >
> > > > > But this risk is already present with
> > > > > existing error cases, and I don't see anything that justifies
> > changing
> > > > > existing behavior with an invalid converter class, or diverging
> from
> > it
> > > > in
> > > > > the case of invalid converter versions.
> > > >
> > > > The justification is to fail-fast, and prevent REST API users from
> > > > receiving errors from bad configs that they didn't write, or maybe
> > > > don't even know apply to them.
> > > > Up until recently errors in these configurations surfaced as failures
> > > > to create the connector, or failures to start, and you made them
> > > > fail-fast during validation. I think this change is in the same
> > > > spirit, pulling the validation further forward and not letting errors
> > > > lie dormant.
> > > >
> > > > And to call back to your original concern about interrupting
> > > > connectors that explicitly provide these configurations and don't use
> > > > the worker configs: I expect that operators with a majority of these
> > > > sorts of clients aren't going to be setting the worker .version
> > > > properties, because it would have no effect on the majority of their
> > > > connectors. They would be able to rely on backwards-compatibility and
> > > > continue to ignore the class properties.
> > > >
> > > > > CLI
> > > > > and programmatic UI developers will want to develop their own
> tooling
> > > > > layers.
> > > >
> > > > This is a very compelling argument. Snehashis do you want to figure
> > > > out a REST API design for this use-case?
> > > >
> > > > > Regarding the GET /connector-plugins/<plugin>/config endpoint, I
> was
> > > > > thinking about the response for non-connector plugins, e.g.,
> > > > > GET /connector-plugins/RegexRouter/config. Would a "version"
> property
> > > > > appear with recommended values?
> > > >
> > > > I intended for the ".version" property to be like other framework
> > > > configs (".class", ".type", ".predicate", ".negate") where they are
> > > > inside the plugin namespace, but not part of the plugin config
> itself.
> > > > Perhaps we can deviate from those configs because it would aid in
> > > > discovering other valid `GET
> > > > /connector-plugins/<plugin>/config?version=` calls, without calling
> > > > `GET /connector-plugins?connectorsOnly=false`.
> > > > I don't really feel strongly either way.
> > > >
> > > > > if a user changes the
> > > > > connector version in, e.g., a dropdown menu, then the UI either has
> > to
> > > > > re-fetch the ConfigDef for the new version, or risk operating on
> > stale
> > > > > information
> > > >
> > > > This is a really interesting situation, thanks for finding that! This
> > > > is already a footgun with transformations and predicates; Once you
> > > > fill out a transformation class (which includes a recommender) you
> > > > will then see all of the transformations configurations.
> > > > I think this means that UI developers should have already developed
> > > > the infrastructure for handling dynamic recommenders, or else they've
> > > > had this bug since KIP-66 in 2017. It may require some manual
> > > > attention to roll-out support for the ".version" properties
> > > > specifically, but I don't think that should prevent us from providing
> > > > this information in a natural place.
> > > >
> > > > > but will it be possible to configure
> > > > > a connector to use two instances of the same transform or
> predicate,
> > but
> > > > > with different versions for each?
> > > >
> > > > Yes, and this is included in the example in the KIP,
> > > > `transforms.flatten-latest` and `transforms.flatten-old`. This will
> > > > work in the natural way, with the two instances having a different
> > > > version if configured for that. If a user wants to pin the same
> > > > version of a plugin for every instance, they will need to provide the
> > > > version config multiple times and keep them in-sync themselves.
> > > >
> > > > > I would have expected the error
> > > > > to only be attributed to the version property, and for the class
> > property
> > > > > to be reported as valid.
> > > >
> > > > I considered this, and then went back and changed it. To quote
> someone
> > > > else, this is to catch "misspelling cat as dog". For example a user
> > > > types DogConverter and meant to type CatConverter, and then types a
> > > > version which is valid for CatConverter, conceptually the error is in
> > > > the class name and not the version.
> > > > That's a very contrived scenario, but I think similar arguments are
> > > > used for attributing validation errors to endpoints/urls/hostnames
> > > > when the associated credentials are unable to log into the remote
> > > > system. Did the user provide the correct testing credentials, but
> > > > accidentally typed the production endpoint? Even if the production
> > > > endpoint looks valid (it's a real hostname that is reachable) the
> > > > conceptual error is still in the hostname and should have the error
> > > > attributed to it to draw the user's attention.
> > > > If that's not convincing, I think the alternative of only attributing
> > > > version errors to the version property is also acceptable.
> > > >
> > > > Thanks,
> > > > Greg
> > > >
> > > > On Wed, May 15, 2024 at 8:06 AM Chris Egerton
> <chrise@aiven.io.invalid
> > >
> > > > wrote:
> > > > >
> > > > > Hi Greg,
> > > > >
> > > > > Thanks for your responses! Continuations of existing discussions:
> > > > >
> > > > > Regarding crashing the worker on startup--yes, there is also a risk
> > to
> > > > > allowing it to join the cluster. But this risk is already present
> > with
> > > > > existing error cases, and I don't see anything that justifies
> > changing
> > > > > existing behavior with an invalid converter class, or diverging
> from
> > it
> > > > in
> > > > > the case of invalid converter versions. I think we should keep this
> > > > simple
> > > > > and not do anything different for worker startup than we do
> already.
> > > > >
> > > > > As far as REST API vs. metrics go--I think you're right that the
> > original
> > > > > version metrics were added as a "monitoring" detail. However, this
> > was
> > > > back
> > > > > when plugin versions were managed solely by cluster administrators.
> > With
> > > > > this KIP, connector users will be able to manage plugin versions,
> > and CLI
> > > > > and programmatic UI developers will want to develop their own
> tooling
> > > > > layers. I think focusing on the REST API as the primary interface
> for
> > > > this
> > > > > KIP would be best for these users.
> > > > >
> > > > > (All that said, I don't object to the metrics that are proposed in
> > the
> > > > KIP;
> > > > > I just think they make more sense in addition to new REST API
> > > > > functionality, as opposed to instead of it.)
> > > > >
> > > > > Regarding the GET /connector-plugins/<plugin>/config endpoint, I
> was
> > > > > thinking about the response for non-connector plugins, e.g.,
> > > > > GET /connector-plugins/RegexRouter/config. Would a "version"
> property
> > > > > appear with recommended values?
> > > > >
> > > > >
> > > > > And new thoughts:
> > > > >
> > > > > 1) Regarding the recommended values for "connector.version", this
> > might
> > > > be
> > > > > confusing since there could be differences between the ConfigDefs
> > for the
> > > > > latest version of the connector and prior versions. It also makes
> the
> > > > flow
> > > > > a bit awkward for programmatic UI developers: if a user changes the
> > > > > connector version in, e.g., a dropdown menu, then the UI either has
> > to
> > > > > re-fetch the ConfigDef for the new version, or risk operating on
> > stale
> > > > > information. I'm starting to doubt that exposing the range of
> > available
> > > > > versions via recommended values is the best way to proceed, instead
> > of a
> > > > > more explicit approach like GET
> > /connector-plugins/<plugin>/versions, or
> > > > > the "Adding new REST API endpoints" rejected alternative.
> > > > >
> > > > > 2) I know that this is a bit heinous, but will it be possible to
> > > > configure
> > > > > a connector to use two instances of the same transform or
> predicate,
> > but
> > > > > with different versions for each? (I don't think this is worth
> > > > significant
> > > > > design/implementation effort, so if it would inflate either of
> those,
> > > > > please don't feel obligated to support it.)
> > > > >
> > > > > 3) Just out of curiosity, why double-ding during config validation
> > if the
> > > > > version for a plugin class can't be found? I would have expected
> the
> > > > error
> > > > > to only be attributed to the version property, and for the class
> > property
> > > > > to be reported as valid.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Tue, May 14, 2024 at 1:11 PM Greg Harris
> > <greg.harris@aiven.io.invalid
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hey Chris,
> > > > > >
> > > > > > Thanks for your comments. I'm glad you like the motivations,
> > Snehashis
> > > > > > wrote that part!
> > > > > >
> > > > > > > the configuration syntax for the most basic use case of
> > > > > > > specifying a single desired version is pretty counterintuitive.
> > > > > >
> > > > > > I agree, and the "soft" requirement scheme is something I wasn't
> > > > > > explicitly looking for, but would be inherited from the library.
> > I'm
> > > > > > fine with eliminating the soft requirement semantics, and having
> > > > > > "1.0.0" behave the same as "[1.0.0]". I'm less inclined to
> include
> > > > > > "range" in the property name, or have two properties.
> > > > > >
> > > > > > > Failing startup is drastic and has the potential to disrupt the
> > > > > > > availability of connectors that would otherwise be able to run
> > > > healthily
> > > > > > > because they were explicitly configured to use valid converters
> > > > instead
> > > > > > of
> > > > > > > the worker defaults.
> > > > > >
> > > > > > I think this argument cuts both ways. If someone reconfigures a
> > worker
> > > > > > and adds an invalid ".version" string to the worker (or changes
> the
> > > > > > plugin.path to make it invalid), it would be permitted to enter
> the
> > > > > > group, and accept work assignments. If those work assignments
> used
> > > > > > these configurations, a set of tasks could transition to FAILED
> and
> > > > > > not be able to recover, because they would be restarted again on
> > the
> > > > > > same worker.
> > > > > >
> > > > > > > Why are metrics utilized to report information about plugin
> > versions
> > > > > > > utilized by connectors at runtime instead of publishing this
> > info in
> > > > the
> > > > > > > REST API
> > > > > >
> > > > > > I was following the existing pattern for exposing runtime
> versions
> > for
> > > > > > connectors, and it did seem like a "monitoring" feature. If that
> > > > > > approach is flawed and should be phased out, I think it would be
> a
> > > > > > good idea to reconsider the REST API rejected alternative.
> > > > > > We would need some additional design work to spec out the REST
> API
> > > > > > interface, as I don't have anything in mind currently.
> > > > > >
> > > > > > > I'm unclear on whether or not it'll be possible to see this
> > > > information
> > > > > > via the
> > > > > > > GET /connector-plugins/<plugin>/config endpoint
> > > > > >
> > > > > > There is room in the API to add recommenders for "key.converter",
> > > > > > "value.converter", and "header.converter", but not for transforms
> > and
> > > > > > predicates, as they include aliases that depend on an actual
> > > > > > configuration. We could explicitly say we're going to do that, or
> > do
> > > > > > whatever is convenient during the implementation phase, or leave
> it
> > > > > > open to be improved later.
> > > > > > There will not be any recommenders for ".version" properties in
> the
> > > > > > `/config` endpoint, because those recommenders are dynamic and
> > depend
> > > > > > on an actual configuration.
> > > > > >
> > > > > > 5) There are two relevant lines in the KIP: "If a .version
> property
> > > > > > contains a hard requirement, select the latest installed version
> > which
> > > > > > satisfies the requirement." and "This configuration is
> re-evaluated
> > > > > > each time the connector or task are assigned to a new worker". I
> > would
> > > > > > call this "eager" upgrade behavior, rather than a "sticky" or
> > "lazy"
> > > > > > upgrade behavior.
> > > > > >
> > > > > > 6) Updated!
> > > > > >
> > > > > > Thanks,
> > > > > > Greg
> > > > > >
> > > > > > On Tue, May 14, 2024 at 9:14 AM Chris Egerton
> > <chrise@aiven.io.invalid
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Thanks Greg for updating the KIP, and thanks Snehashis for
> > starting
> > > > the
> > > > > > > work on this originally.
> > > > > > >
> > > > > > > The motivation section makes a pretty convincing case for this
> > kind
> > > > of
> > > > > > > feature. My thoughts are mostly about specific details:
> > > > > > >
> > > > > > > 1) I like the support for version ranges (the example
> > demonstrating
> > > > how
> > > > > > to
> > > > > > > avoid KAFKA-10574 with the header converter was particularly
> > > > > > entertaining),
> > > > > > > but the configuration syntax for the most basic use case of
> > > > specifying a
> > > > > > > single desired version is pretty counterintuitive. People may
> get
> > > > bitten
> > > > > > or
> > > > > > > at least frustrated if they put connector.version=3.8.0 in a
> > config
> > > > but
> > > > > > > then version 3.7.5 ends up running. I'd like it if we could
> > either
> > > > > > > intentionally deviate from Maven ranges when a bare version is
> > > > present,
> > > > > > or
> > > > > > > separate things out into two properties: foo.version would be
> the
> > > > single
> > > > > > > accepted version for the foo plugin, and foo.version.range
> would
> > use
> > > > > > Maven
> > > > > > > range syntax. Open to other options too, just providing a
> couple
> > to
> > > > get
> > > > > > the
> > > > > > > ball rolling.
> > > > > > >
> > > > > > > 2) Although the current behavior for a worker with an invalid
> > > > > > > key/value/header converter class specified in its config file
> is
> > a
> > > > little
> > > > > > > strange (I was surprised to learn that it wouldn't fail on
> > startup),
> > > > I
> > > > > > > don't see a good reason to deviate from this when an invalid
> > version
> > > > is
> > > > > > > specified. Failing startup is drastic and has the potential to
> > > > disrupt
> > > > > > the
> > > > > > > availability of connectors that would otherwise be able to run
> > > > healthily
> > > > > > > because they were explicitly configured to use valid converters
> > > > instead
> > > > > > of
> > > > > > > the worker defaults.
> > > > > > >
> > > > > > > 3) Why are metrics utilized to report information about plugin
> > > > versions
> > > > > > > utilized by connectors at runtime instead of publishing this
> > info in
> > > > the
> > > > > > > REST API? I saw that this was mentioned as a rejected
> > alternative,
> > > > but I
> > > > > > > didn't get a sense of why. It seems like the REST API would be
> > > > easier to
> > > > > > > access and more intuitive for most users than new metrics.
> > > > > > >
> > > > > > > 4) In the "Validation" section it's stated that "Users can use
> > these
> > > > > > > recommenders to discover the valid plugin classes and versions,
> > > > without
> > > > > > > requiring an earlier call to GET
> > > > > > /connector-plugins?connectorsOnly=false."
> > > > > > > I really like the creativity and simplicity of reusing the
> > > > recommender
> > > > > > > mechanism to expose available versions for plugins via the REST
> > API.
> > > > I'm
> > > > > > > unclear on whether or not it'll be possible to see this
> > information
> > > > via
> > > > > > the
> > > > > > > GET /connector-plugins/<plugin>/config endpoint, though. It'd
> be
> > > > great if
> > > > > > > this were supported, since we learned in KIP-769 [1] that
> people
> > > > really
> > > > > > > want to be able to see configuration options for connectors and
> > their
> > > > > > > plugins via some kind of GET endpoint without having to
> provide a
> > > > > > complete
> > > > > > > connector config for validation.
> > > > > > >
> > > > > > > 5) In the Maven version range docs, it's stated that "Maven
> > picks the
> > > > > > > highest version of each project that satisfies all the hard
> > > > requirements
> > > > > > of
> > > > > > > the dependencies on that project." I'm guessing this behavior
> > will be
> > > > > > > retained for Connect; i.e., the highest-possible version of
> each
> > > > plugin
> > > > > > > that satisfies the user-specified version constraints will be
> > run?
> > > > (An
> > > > > > > alternative approach could be to have some kind of "sticky"
> logic
> > > > that
> > > > > > only
> > > > > > > restarts connectors/tasks when their currently-used version
> > becomes
> > > > > > > incompatible with the configured constraints.)
> > > > > > >
> > > > > > > 6) (Nit) It'd be nice to add a link to the TestPlugins class or
> > > > somewhere
> > > > > > > in its neighborhood to the testing plan; unfamiliar readers
> > probably
> > > > > > won't
> > > > > > > get much out of what's there right now.
> > > > > > >
> > > > > > > [1] -
> > > > > > >
> > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Mon, May 13, 2024 at 2:01 PM Snehashis <
> > snehashisp1995@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Greg,
> > > > > > > >
> > > > > > > > That is much appreciated. No complaints on the additional
> > scope, I
> > > > will
> > > > > > > > make some time out to work on this once we have approval.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Snehashis
> > > > > > > >
> > > > > > > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> > > > > > <gr...@aiven.io.invalid>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Snehashis,
> > > > > > > > >
> > > > > > > > > I'm glad to hear you're still interested in this KIP!
> > > > > > > > > I'm happy to let you drive this, and I apologize for
> > increasing
> > > > the
> > > > > > > > > scope of work so drastically. To make up for that, I'll
> > > > volunteer to
> > > > > > > > > be the primary PR reviewer to help get this done quickly
> once
> > > > the KIP
> > > > > > > > > is approved.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Greg
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, May 10, 2024 at 3:51 AM Snehashis <
> > > > snehashisp1995@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Greg,
> > > > > > > > > >
> > > > > > > > > > Thanks for the follow up to my original KIP, I am in
> > favour of
> > > > the
> > > > > > > > > > additions made to expand its scope, the addition of range
> > > > versions
> > > > > > > > > > specifically make a lot of sense.
> > > > > > > > > >
> > > > > > > > > > Apologies if I have not publicly worked on this KIP for a
> > long
> > > > > > time.
> > > > > > > > The
> > > > > > > > > > original work was done when the move to service loading
> > was in
> > > > > > > > discussion
> > > > > > > > > > and I wanted to loop back to this only after that work
> was
> > > > > > completed.
> > > > > > > > > Post
> > > > > > > > > > its conclusion, I have not been able to take this up due
> to
> > > > other
> > > > > > > > > > priorities. If it's okay with you, I would still like to
> > get
> > > > this
> > > > > > > > > > implemented myself, including the additional scope.
> > > > > > > > > >
> > > > > > > > > > Thanks and regards
> > > > > > > > > > Snehashis
> > > > > > > > > >
> > > > > > > > > > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> > > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I'd like to reboot the discussion on KIP-891:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > > > > > >
> > > > > > > > > > > I've made some changes, most notably:
> > > > > > > > > > >
> > > > > > > > > > > 1. Specifying versions for all plugins in Connector
> > configs
> > > > > > > > > > > (converters, header converters, transforms, and
> > predicates)
> > > > not
> > > > > > just
> > > > > > > > > > > connectors & tasks
> > > > > > > > > > > 2. Specifying a range of versions instead of an exact
> > match
> > > > > > > > > > > 3. New metrics to observe what versions are in-use
> > > > > > > > > > >
> > > > > > > > > > > Thanks to Snehashis for the original KIP idea!
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Greg
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <
> > > > > > greg.harris@aiven.io>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Snehashis,
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you for the KIP! This is something I've wanted
> > for a
> > > > long
> > > > > > > > time.
> > > > > > > > > > > >
> > > > > > > > > > > > I know the discussion has gone cold, are you still
> > > > interested
> > > > > > in
> > > > > > > > > > > > pursuing this feature? I'll make time to review the
> > KIP if
> > > > you
> > > > > > are
> > > > > > > > > > > > still accepting comments.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Greg
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis <
> > > > > > > > snehashisp1995@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the points Sagar.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 1) Should we update the GET /connectors endpoint
> to
> > > > > > include the
> > > > > > > > > > > version of
> > > > > > > > > > > > > > the plugin that is running? It could be useful to
> > > > figure
> > > > > > out
> > > > > > > > the
> > > > > > > > > > > version
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > the plugin or I am assuming it gets returned by
> the
> > > > > > expand=info
> > > > > > > > > call?
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think this is good to have and possible future
> > > > > > enhancement. The
> > > > > > > > > > > version
> > > > > > > > > > > > > info will be present in the config of the connector
> > if
> > > > the
> > > > > > user
> > > > > > > > has
> > > > > > > > > > > > > specified the version. Otherwise it is the latest
> > version
> > > > > > which
> > > > > > > > the
> > > > > > > > > > > user
> > > > > > > > > > > > > can find out from the connector-plugin endpoint.
> The
> > > > > > information
> > > > > > > > > can be
> > > > > > > > > > > > > introduced to the response of the GET /connectors
> > > > endpoint
> > > > > > > > itself,
> > > > > > > > > > > however
> > > > > > > > > > > > > the most ideal way of doing this would be to get
> the
> > > > > > currently
> > > > > > > > > running
> > > > > > > > > > > > > instance of the connector and get the version
> > directly
> > > > from
> > > > > > > > there.
> > > > > > > > > > > This is
> > > > > > > > > > > > > slightly tricky as the connector could be running
> in
> > a
> > > > > > different
> > > > > > > > > node.
> > > > > > > > > > > > > One way to do this would be to persist the version
> > > > > > information in
> > > > > > > > > the
> > > > > > > > > > > > > status backing store during instantiation of the
> > > > connector.
> > > > > > It
> > > > > > > > > requires
> > > > > > > > > > > > > some more thought and since the version is part of
> > the
> > > > > > configs if
> > > > > > > > > > > provided
> > > > > > > > > > > > > and evident otherwise, I have not included it in
> this
> > > > KIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 2) I am not aware of this and hence asking, can 2
> > > > > > connectors
> > > > > > > > with
> > > > > > > > > > > > > different
> > > > > > > > > > > > > > versions have the same name? Does the plugin
> > isolation
> > > > > > allow
> > > > > > > > > this?
> > > > > > > > > > > This
> > > > > > > > > > > > > > could have a bearing when using the lifecycle
> > > > endpoints for
> > > > > > > > > > > connectors
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > >
> > > > > > > > > > > > > All connectors in a cluster need to have uniquire
> > > > connector
> > > > > > names
> > > > > > > > > > > > > regardless of what version of the plugin the
> > connector is
> > > > > > running
> > > > > > > > > > > > > underneath. This is something enforced by the
> connect
> > > > runtime
> > > > > > > > > itself.
> > > > > > > > > > > All
> > > > > > > > > > > > > connect CRUD operations are keyed on the connector
> > name
> > > > so
> > > > > > there
> > > > > > > > > will
> > > > > > > > > > > not
> > > > > > > > > > > > > be an issue.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards
> > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <
> > > > > > sagarmeansocean@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hey Snehashsih,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the KIP. It looks like a very useful
> > > > feature.
> > > > > > Couple
> > > > > > > > > of
> > > > > > > > > > > > > > small-ish points, let me know what you think:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1) Should we update the GET /connectors endpoint
> to
> > > > > > include the
> > > > > > > > > > > version of
> > > > > > > > > > > > > > the plugin that is running? It could be useful to
> > > > figure
> > > > > > out
> > > > > > > > the
> > > > > > > > > > > version of
> > > > > > > > > > > > > > the plugin or I am assuming it gets returned by
> the
> > > > > > expand=info
> > > > > > > > > call?
> > > > > > > > > > > > > > 2) I am not aware of this and hence asking, can 2
> > > > > > connectors
> > > > > > > > with
> > > > > > > > > > > different
> > > > > > > > > > > > > > versions have the same name? Does the plugin
> > isolation
> > > > > > allow
> > > > > > > > > this?
> > > > > > > > > > > This
> > > > > > > > > > > > > > could have a bearing when using the lifecycle
> > > > endpoints for
> > > > > > > > > > > connectors like
> > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > Sagar.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin
> > > > > > > > > <apankaj@confluent.io.invalid
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > IIUC (please correct me if I am wrong here),
> > what
> > > > you
> > > > > > > > > highlighted
> > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > a versioning scheme for a connector config for
> > the
> > > > same
> > > > > > > > > connector
> > > > > > > > > > > (and
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > different versions of a connector plugin).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Sorry for not being more precise in my wording
> > - I
> > > > meant
> > > > > > > > > > > registering
> > > > > > > > > > > > > > > versions of schema for connector config.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Let's take the example of a fictional connector
> > which
> > > > > > uses a
> > > > > > > > > > > fictional
> > > > > > > > > > > > > > AWS
> > > > > > > > > > > > > > > service.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Fictional Connector Config schema version:2.0
> > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > "$schema": "
> > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Fictional Connector config schema version:3.0
> > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > "$schema": "
> > > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > "iam_role": {
> > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > "iam_role"
> > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The connector which supports Fictional config
> > schema
> > > > 2.0
> > > > > > > > will
> > > > > > > > > > > validate
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > access key and secret key.
> > > > > > > > > > > > > > > Whereas a connector which supports config with
> > schema
> > > > > > version
> > > > > > > > > 3.0
> > > > > > > > > > > will
> > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > validate the IAM role.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This is the alternative which I wanted to
> > suggest.
> > > > Each
> > > > > > > > plugin
> > > > > > > > > will
> > > > > > > > > > > > > > > register the schema versions of connector
> config
> > > > which it
> > > > > > > > > supports.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The plugin paths may be optionally different
> > i.e we
> > > > > > don't
> > > > > > > > > have to
> > > > > > > > > > > > > > > mandatorily add a new plugin path to support a
> > new
> > > > schema
> > > > > > > > > version.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <
> > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. Can you elaborate on the rejected
> > > > alternatives ?
> > > > > > > > Suppose
> > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > config is versioned and has a schema. Then
> a
> > > > single
> > > > > > > > plugin
> > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > dependencies have not changed) can handle
> > > > multiple
> > > > > > config
> > > > > > > > > > > versions
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > IIUC (please correct me if I am wrong here),
> > what
> > > > you
> > > > > > > > > highlighted
> > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > a versioning scheme for a connector config
> for
> > the
> > > > same
> > > > > > > > > > > connector (and
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > different versions of a connector plugin).
> > That is
> > > > a
> > > > > > > > somewhat
> > > > > > > > > > > > > > tangential
> > > > > > > > > > > > > > > > problem. While it is definitely a useful
> > feature to
> > > > > > have,
> > > > > > > > > like a
> > > > > > > > > > > log to
> > > > > > > > > > > > > > > > check what changes were made over time to the
> > > > config
> > > > > > which
> > > > > > > > > might
> > > > > > > > > > > make
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > easier to do rollbacks, it is not the focus
> > here.
> > > > Here
> > > > > > by
> > > > > > > > > > > version we
> > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > to say what underlying version of the plugin
> > > > should the
> > > > > > > > given
> > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > of the connector use. Perhaps it is better to
> > > > change
> > > > > > the
> > > > > > > > > name of
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > parameter from connector.version to
> > > > > > > > connector.plugin.version
> > > > > > > > > or
> > > > > > > > > > > > > > > > plugin.version if it was confusing. wdyt?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2. Any plans to support assisted migration
> > e.g
> > > > if a
> > > > > > user
> > > > > > > > > > > invokes
> > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > connector/config?migrate=latest", the
> latest
> > > > version
> > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > transform the existing config to the newer
> > > > version.
> > > > > > This
> > > > > > > > > would
> > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > adding a method like "boolean
> migrate(Version
> > > > > > > > > fromVersion)" to
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This is an enhancement we can think of doing
> in
> > > > future.
> > > > > > > > > Users can
> > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > a PUT call with the updated config which has
> > the
> > > > > > updated
> > > > > > > > > version
> > > > > > > > > > > > > > number.
> > > > > > > > > > > > > > > > The assisted mode could be handy as the user
> > does
> > > > not
> > > > > > need
> > > > > > > > to
> > > > > > > > > > > know the
> > > > > > > > > > > > > > > > config but beyond this it does not seem to
> > justify
> > > > its
> > > > > > > > > existence.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin
> > > > > > > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This is a really useful feature and thanks
> > for
> > > > > > initiating
> > > > > > > > > this
> > > > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I had the following questions -
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. Can you elaborate on the rejected
> > > > alternatives ?
> > > > > > > > Suppose
> > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > config is versioned and has a schema. Then
> a
> > > > single
> > > > > > > > plugin
> > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > dependencies have not changed) can handle
> > > > multiple
> > > > > > config
> > > > > > > > > > > versions
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2. Any plans to support assisted migration
> > e.g
> > > > if a
> > > > > > user
> > > > > > > > > > > invokes
> > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > connector/config?migrate=latest", the
> latest
> > > > version
> > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > transform the existing config to the newer
> > > > version.
> > > > > > This
> > > > > > > > > would
> > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > adding a method like "boolean
> migrate(Version
> > > > > > > > > fromVersion)" to
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <
> > > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'd like to start a discussion thread on
> > > > KIP-891:
> > > > > > > > Running
> > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The KIP aims to add the ability for the
> > connect
> > > > > > runtime
> > > > > > > > > to
> > > > > > > > > > > run
> > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > versions of a connector.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Please take a look and let me know what
> you
> > > > think.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Chris Egerton <ch...@aiven.io.INVALID>.
Hi Greg,
I'm confused about the behavior for GET
/connector-plugins?connectorsOnly=false. Won't it still only expose the
latest version for each plugin, instead of the range of versions available?
I'm hoping we can provide a flow where people need at most two REST calls
to discover 1) the complete set of plugins available on the worker (which
is already possible with the endpoint under discussion) and 2) the set of
versions available for a specific plugin on the worker (which doesn't
appear to be possible, at least for some plugin types). This wouldn't
require any out-of-band knowledge and would be valuable for connector users
(who may want to, for example, know what their options are when considering
a plugin upgrade) and cluster administrators (who could use it as a sanity
check for the setup of their Kafka Connect clusters without having to pore
over log files).
As far as modifying the content of the GET
/connector-plugins/<plugin>/config endpoint to include a "version" property
goes, I think your point about returning that property from requests to
that endpoint that include a version query parameter is salient, but it
also unfortunately applies to all types of plugin. I don't think it should
completely disqualify that option. I also wasn't imagining adding anything
besides that single property to that endpoint, so no new "predicate"
property for SMTs, no new "negate" property for predicates, and no new
"type" property for either. I'm not necessarily opposed to adding those,
but it can be done without being pulled into the scope of this KIP.
So to summarize how I imagine people might use the REST API after this KIP:
1. Discover the set of plugins on the worker via GET
/connector-plugins?connectorsOnly=false
2. For any of those plugins, discover the set of available versions by
examining the recommended values for the "version" property after hitting
the GET GET /connector-plugins/<plugin>/config endpoint
I don't think this is necessarily optimal since plucking the "version"
property out of the response from that endpoint might be a PITA for ad hoc
queries via, e.g., curl, but I think it'd be enough for this KIP. Still
open to other options though, and if I've misunderstood anything in the
proposal, please let me know!
Cheers,
Chris
On Wed, May 22, 2024 at 4:05 PM Greg Harris <gr...@aiven.io.invalid>
wrote:
> Hey Chris,
>
> Thanks for your comments, and I'm glad that it seems like we're
> aligning on the vision here.
>
> > An
> > alternative could be to change existing behavior to fail fast on any
> > invalid default converter configuration instead of just for invalid
> > versions
>
> I suppose if this is landing in 4.0, we have the opportunity to break
> compatibility and strictly validate the worker class configs, and
> could have a new consistent behavior instead of inconsistent but
> backwards-compatible behavior.
>
> > no other part of the KIP requires this change.
>
> This is correct, and a compelling argument. I'm fine leaving the
> strict worker validation off of this configuration to potentially be
> added later. If a KIP was raised to perform strict validation of the
> worker config, it would include the version config, and can address
> backwards compatibility for both configs together.
>
> > RE exposing the version property in the
> /connector-plugins/<plugin>/config
> > endpoint, the behavior is inconsistent across plugin types.
>
> Yes it is, and that started in a bugfix, once people started using
> this endpoint: https://issues.apache.org/jira/browse/KAFKA-14843 . I
> wasn't planning on converging the behavior in this KIP, as I
> considered it out-of-scope, and was going to follow the current
> behavior. To summarize:
>
> GET /connector-plugins/<connector>/config will emit:
> * connector.class (already implemented)
> * connector.version (new)
> * key.converter (already implemented)
> * key.converter.version (new)
> * value.converter (already implemented)
> * value.converter.version (new)
> * header.converter (already implemented)
> * header.converter.version (new)
> But will NOT emit:
> * transforms.<alias>.type
> * transforms.<alias>.version
> * transforms.<alias>.predicate
> * predicates.<alias>.type
> * predicates.<alias>.version
> * predicates.<alias>.negate
>
> GET /connector-plugins/<converter>/config will NOT emit:
> * "" (there's not even a ".class" prefix!)
> * version
>
> GET /connector-plugins/<transform>/config will NOT emit:
> * type
> * version
> * predicate
>
> GET /connector-plugins/<predicate>/config will NOT emit:
> * type
> * version
> * negate
>
> Do you want the converter, transform, and predicate endpoints changed?
> Do you want just "version", or do you want all of the prefixed configs
> including "type", "predicate" and "negate"? How would you want to
> handle the converters?
> And when I say the configs are "not part of the plugin config itself"
> I mean that saying that GET
> /connector-plugins/Flatten$Key/config?version=3.8.0 has a "version"
> config that must be "3.8.0" is a little bit nonsense, as the version
> is already specified.
>
> > IMO
> > it's worth including this information somewhere directly accessible
> without
> > having to provide a full connector config. FWIW I'd be fine with GET
> > /connector-plugins/<plugin>/versions as a first-class endpoint
>
> You don't have to provide a configuration to call GET
> /connector-plugins?connectorsOnly=false , is that endpoint not close
> enough to what you have in mind? See also the Rejected Alternative
> "Adding new REST API endpoints"
>
> If you're calling /connector-plugins/<transform>/config, you know the
> name of a plugin right? That either comes from out-of-band knowledge,
> validating a connector config, or calling GET
> /connector-plugins?connectorsOnly=false.
> * If you have out-of-band knowledge of plugin classes, perhaps you
> have out-of-band knowledge of versions too.
> * If you've just validated a connector config, there should be an
> accompanying "version" field there with an accurate default value and
> recommenders.
> * If you've called GET /connector-plugins?connectorsOnly=false, that
> endpoint includes version information.
>
> Thanks,
> Greg
>
> On Wed, May 22, 2024 at 11:05 AM Chris Egerton <ch...@aiven.io.invalid>
> wrote:
> >
> > Hi Greg,
> >
> > Hope you had a nice weekend! Gonna try to keep things concise.
> >
> > Concluded points:
> >
> > RE version recommenders, I agree it's likely that programmatic UIs will
> > already be able to handle dynamic configuration definitions, and the
> detail
> > about SMTs is a great point. I still anticipate some awkwardness with
> > connector versions, though: if the latest version supports some new
> > properties, then a user switches to an earlier version, a UI may respond
> by
> > wiping values for these properties. I guess we can bite the bullet,
> though.
> >
> > RE double-dinging during preflight validation for invalid versions, I
> like
> > the analogy with login credentials. I'm convinced that the proposal in
> the
> > KIP is best 👍
> >
> > Continued points:
> >
> > RE failing on worker startup, sorry, I should be clearer: there is no
> _new_
> > justification for it that doesn't also apply to existing behavior. We
> > shouldn't diverge from existing behavior solely for this new case. An
> > alternative could be to change existing behavior to fail fast on any
> > invalid default converter configuration instead of just for invalid
> > versions, but I'd vote to just stick to existing behavior and not
> > complicate things, especially since no other part of the KIP requires
> this
> > change.
> >
> > RE exposing the version property in the
> /connector-plugins/<plugin>/config
> > endpoint, the behavior is inconsistent across plugin types. Hitting the
> > endpoint for the FileStreamSinkConnector on version 3.7.0 yields a
> response
> > that includes, among other things, the "topics", "topics.regex", and
> > "errors.tolerance" properties. I see that we don't do this everywhere
> (the
> > examples you cite for SMT and converter properties are accurate), but IMO
> > it's worth including this information somewhere directly accessible
> without
> > having to provide a full connector config. FWIW I'd be fine with GET
> > /connector-plugins/<plugin>/versions as a first-class endpoint either
> > instead of or in addition to adding recommended values for all plugin
> > versions.
> >
> > Thanks for your continued work on this KIP, and with the progress we're
> > making I'm optimistic about its chances of appearing in 4.0.0.
> >
> > Cheers,
> >
> > Chris
> >
> > On Wed, May 15, 2024 at 1:22 PM Greg Harris <greg.harris@aiven.io.invalid
> >
> > wrote:
> >
> > > Hey Chris,
> > >
> > > Thanks for your quick follow up.
> > >
> > > > But this risk is already present with
> > > > existing error cases, and I don't see anything that justifies
> changing
> > > > existing behavior with an invalid converter class, or diverging from
> it
> > > in
> > > > the case of invalid converter versions.
> > >
> > > The justification is to fail-fast, and prevent REST API users from
> > > receiving errors from bad configs that they didn't write, or maybe
> > > don't even know apply to them.
> > > Up until recently errors in these configurations surfaced as failures
> > > to create the connector, or failures to start, and you made them
> > > fail-fast during validation. I think this change is in the same
> > > spirit, pulling the validation further forward and not letting errors
> > > lie dormant.
> > >
> > > And to call back to your original concern about interrupting
> > > connectors that explicitly provide these configurations and don't use
> > > the worker configs: I expect that operators with a majority of these
> > > sorts of clients aren't going to be setting the worker .version
> > > properties, because it would have no effect on the majority of their
> > > connectors. They would be able to rely on backwards-compatibility and
> > > continue to ignore the class properties.
> > >
> > > > CLI
> > > > and programmatic UI developers will want to develop their own tooling
> > > > layers.
> > >
> > > This is a very compelling argument. Snehashis do you want to figure
> > > out a REST API design for this use-case?
> > >
> > > > Regarding the GET /connector-plugins/<plugin>/config endpoint, I was
> > > > thinking about the response for non-connector plugins, e.g.,
> > > > GET /connector-plugins/RegexRouter/config. Would a "version" property
> > > > appear with recommended values?
> > >
> > > I intended for the ".version" property to be like other framework
> > > configs (".class", ".type", ".predicate", ".negate") where they are
> > > inside the plugin namespace, but not part of the plugin config itself.
> > > Perhaps we can deviate from those configs because it would aid in
> > > discovering other valid `GET
> > > /connector-plugins/<plugin>/config?version=` calls, without calling
> > > `GET /connector-plugins?connectorsOnly=false`.
> > > I don't really feel strongly either way.
> > >
> > > > if a user changes the
> > > > connector version in, e.g., a dropdown menu, then the UI either has
> to
> > > > re-fetch the ConfigDef for the new version, or risk operating on
> stale
> > > > information
> > >
> > > This is a really interesting situation, thanks for finding that! This
> > > is already a footgun with transformations and predicates; Once you
> > > fill out a transformation class (which includes a recommender) you
> > > will then see all of the transformations configurations.
> > > I think this means that UI developers should have already developed
> > > the infrastructure for handling dynamic recommenders, or else they've
> > > had this bug since KIP-66 in 2017. It may require some manual
> > > attention to roll-out support for the ".version" properties
> > > specifically, but I don't think that should prevent us from providing
> > > this information in a natural place.
> > >
> > > > but will it be possible to configure
> > > > a connector to use two instances of the same transform or predicate,
> but
> > > > with different versions for each?
> > >
> > > Yes, and this is included in the example in the KIP,
> > > `transforms.flatten-latest` and `transforms.flatten-old`. This will
> > > work in the natural way, with the two instances having a different
> > > version if configured for that. If a user wants to pin the same
> > > version of a plugin for every instance, they will need to provide the
> > > version config multiple times and keep them in-sync themselves.
> > >
> > > > I would have expected the error
> > > > to only be attributed to the version property, and for the class
> property
> > > > to be reported as valid.
> > >
> > > I considered this, and then went back and changed it. To quote someone
> > > else, this is to catch "misspelling cat as dog". For example a user
> > > types DogConverter and meant to type CatConverter, and then types a
> > > version which is valid for CatConverter, conceptually the error is in
> > > the class name and not the version.
> > > That's a very contrived scenario, but I think similar arguments are
> > > used for attributing validation errors to endpoints/urls/hostnames
> > > when the associated credentials are unable to log into the remote
> > > system. Did the user provide the correct testing credentials, but
> > > accidentally typed the production endpoint? Even if the production
> > > endpoint looks valid (it's a real hostname that is reachable) the
> > > conceptual error is still in the hostname and should have the error
> > > attributed to it to draw the user's attention.
> > > If that's not convincing, I think the alternative of only attributing
> > > version errors to the version property is also acceptable.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Wed, May 15, 2024 at 8:06 AM Chris Egerton <chrise@aiven.io.invalid
> >
> > > wrote:
> > > >
> > > > Hi Greg,
> > > >
> > > > Thanks for your responses! Continuations of existing discussions:
> > > >
> > > > Regarding crashing the worker on startup--yes, there is also a risk
> to
> > > > allowing it to join the cluster. But this risk is already present
> with
> > > > existing error cases, and I don't see anything that justifies
> changing
> > > > existing behavior with an invalid converter class, or diverging from
> it
> > > in
> > > > the case of invalid converter versions. I think we should keep this
> > > simple
> > > > and not do anything different for worker startup than we do already.
> > > >
> > > > As far as REST API vs. metrics go--I think you're right that the
> original
> > > > version metrics were added as a "monitoring" detail. However, this
> was
> > > back
> > > > when plugin versions were managed solely by cluster administrators.
> With
> > > > this KIP, connector users will be able to manage plugin versions,
> and CLI
> > > > and programmatic UI developers will want to develop their own tooling
> > > > layers. I think focusing on the REST API as the primary interface for
> > > this
> > > > KIP would be best for these users.
> > > >
> > > > (All that said, I don't object to the metrics that are proposed in
> the
> > > KIP;
> > > > I just think they make more sense in addition to new REST API
> > > > functionality, as opposed to instead of it.)
> > > >
> > > > Regarding the GET /connector-plugins/<plugin>/config endpoint, I was
> > > > thinking about the response for non-connector plugins, e.g.,
> > > > GET /connector-plugins/RegexRouter/config. Would a "version" property
> > > > appear with recommended values?
> > > >
> > > >
> > > > And new thoughts:
> > > >
> > > > 1) Regarding the recommended values for "connector.version", this
> might
> > > be
> > > > confusing since there could be differences between the ConfigDefs
> for the
> > > > latest version of the connector and prior versions. It also makes the
> > > flow
> > > > a bit awkward for programmatic UI developers: if a user changes the
> > > > connector version in, e.g., a dropdown menu, then the UI either has
> to
> > > > re-fetch the ConfigDef for the new version, or risk operating on
> stale
> > > > information. I'm starting to doubt that exposing the range of
> available
> > > > versions via recommended values is the best way to proceed, instead
> of a
> > > > more explicit approach like GET
> /connector-plugins/<plugin>/versions, or
> > > > the "Adding new REST API endpoints" rejected alternative.
> > > >
> > > > 2) I know that this is a bit heinous, but will it be possible to
> > > configure
> > > > a connector to use two instances of the same transform or predicate,
> but
> > > > with different versions for each? (I don't think this is worth
> > > significant
> > > > design/implementation effort, so if it would inflate either of those,
> > > > please don't feel obligated to support it.)
> > > >
> > > > 3) Just out of curiosity, why double-ding during config validation
> if the
> > > > version for a plugin class can't be found? I would have expected the
> > > error
> > > > to only be attributed to the version property, and for the class
> property
> > > > to be reported as valid.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Tue, May 14, 2024 at 1:11 PM Greg Harris
> <greg.harris@aiven.io.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hey Chris,
> > > > >
> > > > > Thanks for your comments. I'm glad you like the motivations,
> Snehashis
> > > > > wrote that part!
> > > > >
> > > > > > the configuration syntax for the most basic use case of
> > > > > > specifying a single desired version is pretty counterintuitive.
> > > > >
> > > > > I agree, and the "soft" requirement scheme is something I wasn't
> > > > > explicitly looking for, but would be inherited from the library.
> I'm
> > > > > fine with eliminating the soft requirement semantics, and having
> > > > > "1.0.0" behave the same as "[1.0.0]". I'm less inclined to include
> > > > > "range" in the property name, or have two properties.
> > > > >
> > > > > > Failing startup is drastic and has the potential to disrupt the
> > > > > > availability of connectors that would otherwise be able to run
> > > healthily
> > > > > > because they were explicitly configured to use valid converters
> > > instead
> > > > > of
> > > > > > the worker defaults.
> > > > >
> > > > > I think this argument cuts both ways. If someone reconfigures a
> worker
> > > > > and adds an invalid ".version" string to the worker (or changes the
> > > > > plugin.path to make it invalid), it would be permitted to enter the
> > > > > group, and accept work assignments. If those work assignments used
> > > > > these configurations, a set of tasks could transition to FAILED and
> > > > > not be able to recover, because they would be restarted again on
> the
> > > > > same worker.
> > > > >
> > > > > > Why are metrics utilized to report information about plugin
> versions
> > > > > > utilized by connectors at runtime instead of publishing this
> info in
> > > the
> > > > > > REST API
> > > > >
> > > > > I was following the existing pattern for exposing runtime versions
> for
> > > > > connectors, and it did seem like a "monitoring" feature. If that
> > > > > approach is flawed and should be phased out, I think it would be a
> > > > > good idea to reconsider the REST API rejected alternative.
> > > > > We would need some additional design work to spec out the REST API
> > > > > interface, as I don't have anything in mind currently.
> > > > >
> > > > > > I'm unclear on whether or not it'll be possible to see this
> > > information
> > > > > via the
> > > > > > GET /connector-plugins/<plugin>/config endpoint
> > > > >
> > > > > There is room in the API to add recommenders for "key.converter",
> > > > > "value.converter", and "header.converter", but not for transforms
> and
> > > > > predicates, as they include aliases that depend on an actual
> > > > > configuration. We could explicitly say we're going to do that, or
> do
> > > > > whatever is convenient during the implementation phase, or leave it
> > > > > open to be improved later.
> > > > > There will not be any recommenders for ".version" properties in the
> > > > > `/config` endpoint, because those recommenders are dynamic and
> depend
> > > > > on an actual configuration.
> > > > >
> > > > > 5) There are two relevant lines in the KIP: "If a .version property
> > > > > contains a hard requirement, select the latest installed version
> which
> > > > > satisfies the requirement." and "This configuration is re-evaluated
> > > > > each time the connector or task are assigned to a new worker". I
> would
> > > > > call this "eager" upgrade behavior, rather than a "sticky" or
> "lazy"
> > > > > upgrade behavior.
> > > > >
> > > > > 6) Updated!
> > > > >
> > > > > Thanks,
> > > > > Greg
> > > > >
> > > > > On Tue, May 14, 2024 at 9:14 AM Chris Egerton
> <chrise@aiven.io.invalid
> > > >
> > > > > wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Thanks Greg for updating the KIP, and thanks Snehashis for
> starting
> > > the
> > > > > > work on this originally.
> > > > > >
> > > > > > The motivation section makes a pretty convincing case for this
> kind
> > > of
> > > > > > feature. My thoughts are mostly about specific details:
> > > > > >
> > > > > > 1) I like the support for version ranges (the example
> demonstrating
> > > how
> > > > > to
> > > > > > avoid KAFKA-10574 with the header converter was particularly
> > > > > entertaining),
> > > > > > but the configuration syntax for the most basic use case of
> > > specifying a
> > > > > > single desired version is pretty counterintuitive. People may get
> > > bitten
> > > > > or
> > > > > > at least frustrated if they put connector.version=3.8.0 in a
> config
> > > but
> > > > > > then version 3.7.5 ends up running. I'd like it if we could
> either
> > > > > > intentionally deviate from Maven ranges when a bare version is
> > > present,
> > > > > or
> > > > > > separate things out into two properties: foo.version would be the
> > > single
> > > > > > accepted version for the foo plugin, and foo.version.range would
> use
> > > > > Maven
> > > > > > range syntax. Open to other options too, just providing a couple
> to
> > > get
> > > > > the
> > > > > > ball rolling.
> > > > > >
> > > > > > 2) Although the current behavior for a worker with an invalid
> > > > > > key/value/header converter class specified in its config file is
> a
> > > little
> > > > > > strange (I was surprised to learn that it wouldn't fail on
> startup),
> > > I
> > > > > > don't see a good reason to deviate from this when an invalid
> version
> > > is
> > > > > > specified. Failing startup is drastic and has the potential to
> > > disrupt
> > > > > the
> > > > > > availability of connectors that would otherwise be able to run
> > > healthily
> > > > > > because they were explicitly configured to use valid converters
> > > instead
> > > > > of
> > > > > > the worker defaults.
> > > > > >
> > > > > > 3) Why are metrics utilized to report information about plugin
> > > versions
> > > > > > utilized by connectors at runtime instead of publishing this
> info in
> > > the
> > > > > > REST API? I saw that this was mentioned as a rejected
> alternative,
> > > but I
> > > > > > didn't get a sense of why. It seems like the REST API would be
> > > easier to
> > > > > > access and more intuitive for most users than new metrics.
> > > > > >
> > > > > > 4) In the "Validation" section it's stated that "Users can use
> these
> > > > > > recommenders to discover the valid plugin classes and versions,
> > > without
> > > > > > requiring an earlier call to GET
> > > > > /connector-plugins?connectorsOnly=false."
> > > > > > I really like the creativity and simplicity of reusing the
> > > recommender
> > > > > > mechanism to expose available versions for plugins via the REST
> API.
> > > I'm
> > > > > > unclear on whether or not it'll be possible to see this
> information
> > > via
> > > > > the
> > > > > > GET /connector-plugins/<plugin>/config endpoint, though. It'd be
> > > great if
> > > > > > this were supported, since we learned in KIP-769 [1] that people
> > > really
> > > > > > want to be able to see configuration options for connectors and
> their
> > > > > > plugins via some kind of GET endpoint without having to provide a
> > > > > complete
> > > > > > connector config for validation.
> > > > > >
> > > > > > 5) In the Maven version range docs, it's stated that "Maven
> picks the
> > > > > > highest version of each project that satisfies all the hard
> > > requirements
> > > > > of
> > > > > > the dependencies on that project." I'm guessing this behavior
> will be
> > > > > > retained for Connect; i.e., the highest-possible version of each
> > > plugin
> > > > > > that satisfies the user-specified version constraints will be
> run?
> > > (An
> > > > > > alternative approach could be to have some kind of "sticky" logic
> > > that
> > > > > only
> > > > > > restarts connectors/tasks when their currently-used version
> becomes
> > > > > > incompatible with the configured constraints.)
> > > > > >
> > > > > > 6) (Nit) It'd be nice to add a link to the TestPlugins class or
> > > somewhere
> > > > > > in its neighborhood to the testing plan; unfamiliar readers
> probably
> > > > > won't
> > > > > > get much out of what's there right now.
> > > > > >
> > > > > > [1] -
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Mon, May 13, 2024 at 2:01 PM Snehashis <
> snehashisp1995@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Greg,
> > > > > > >
> > > > > > > That is much appreciated. No complaints on the additional
> scope, I
> > > will
> > > > > > > make some time out to work on this once we have approval.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Snehashis
> > > > > > >
> > > > > > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> > > > > <gr...@aiven.io.invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey Snehashis,
> > > > > > > >
> > > > > > > > I'm glad to hear you're still interested in this KIP!
> > > > > > > > I'm happy to let you drive this, and I apologize for
> increasing
> > > the
> > > > > > > > scope of work so drastically. To make up for that, I'll
> > > volunteer to
> > > > > > > > be the primary PR reviewer to help get this done quickly once
> > > the KIP
> > > > > > > > is approved.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Greg
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, May 10, 2024 at 3:51 AM Snehashis <
> > > snehashisp1995@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Greg,
> > > > > > > > >
> > > > > > > > > Thanks for the follow up to my original KIP, I am in
> favour of
> > > the
> > > > > > > > > additions made to expand its scope, the addition of range
> > > versions
> > > > > > > > > specifically make a lot of sense.
> > > > > > > > >
> > > > > > > > > Apologies if I have not publicly worked on this KIP for a
> long
> > > > > time.
> > > > > > > The
> > > > > > > > > original work was done when the move to service loading
> was in
> > > > > > > discussion
> > > > > > > > > and I wanted to loop back to this only after that work was
> > > > > completed.
> > > > > > > > Post
> > > > > > > > > its conclusion, I have not been able to take this up due to
> > > other
> > > > > > > > > priorities. If it's okay with you, I would still like to
> get
> > > this
> > > > > > > > > implemented myself, including the additional scope.
> > > > > > > > >
> > > > > > > > > Thanks and regards
> > > > > > > > > Snehashis
> > > > > > > > >
> > > > > > > > > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> > > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I'd like to reboot the discussion on KIP-891:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > > > > >
> > > > > > > > > > I've made some changes, most notably:
> > > > > > > > > >
> > > > > > > > > > 1. Specifying versions for all plugins in Connector
> configs
> > > > > > > > > > (converters, header converters, transforms, and
> predicates)
> > > not
> > > > > just
> > > > > > > > > > connectors & tasks
> > > > > > > > > > 2. Specifying a range of versions instead of an exact
> match
> > > > > > > > > > 3. New metrics to observe what versions are in-use
> > > > > > > > > >
> > > > > > > > > > Thanks to Snehashis for the original KIP idea!
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Greg
> > > > > > > > > >
> > > > > > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <
> > > > > greg.harris@aiven.io>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Snehashis,
> > > > > > > > > > >
> > > > > > > > > > > Thank you for the KIP! This is something I've wanted
> for a
> > > long
> > > > > > > time.
> > > > > > > > > > >
> > > > > > > > > > > I know the discussion has gone cold, are you still
> > > interested
> > > > > in
> > > > > > > > > > > pursuing this feature? I'll make time to review the
> KIP if
> > > you
> > > > > are
> > > > > > > > > > > still accepting comments.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Greg
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis <
> > > > > > > snehashisp1995@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the points Sagar.
> > > > > > > > > > > >
> > > > > > > > > > > > > 1) Should we update the GET /connectors endpoint to
> > > > > include the
> > > > > > > > > > version of
> > > > > > > > > > > > > the plugin that is running? It could be useful to
> > > figure
> > > > > out
> > > > > > > the
> > > > > > > > > > version
> > > > > > > > > > > > of
> > > > > > > > > > > > > the plugin or I am assuming it gets returned by the
> > > > > expand=info
> > > > > > > > call?
> > > > > > > > > > > >
> > > > > > > > > > > > I think this is good to have and possible future
> > > > > enhancement. The
> > > > > > > > > > version
> > > > > > > > > > > > info will be present in the config of the connector
> if
> > > the
> > > > > user
> > > > > > > has
> > > > > > > > > > > > specified the version. Otherwise it is the latest
> version
> > > > > which
> > > > > > > the
> > > > > > > > > > user
> > > > > > > > > > > > can find out from the connector-plugin endpoint. The
> > > > > information
> > > > > > > > can be
> > > > > > > > > > > > introduced to the response of the GET /connectors
> > > endpoint
> > > > > > > itself,
> > > > > > > > > > however
> > > > > > > > > > > > the most ideal way of doing this would be to get the
> > > > > currently
> > > > > > > > running
> > > > > > > > > > > > instance of the connector and get the version
> directly
> > > from
> > > > > > > there.
> > > > > > > > > > This is
> > > > > > > > > > > > slightly tricky as the connector could be running in
> a
> > > > > different
> > > > > > > > node.
> > > > > > > > > > > > One way to do this would be to persist the version
> > > > > information in
> > > > > > > > the
> > > > > > > > > > > > status backing store during instantiation of the
> > > connector.
> > > > > It
> > > > > > > > requires
> > > > > > > > > > > > some more thought and since the version is part of
> the
> > > > > configs if
> > > > > > > > > > provided
> > > > > > > > > > > > and evident otherwise, I have not included it in this
> > > KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > > 2) I am not aware of this and hence asking, can 2
> > > > > connectors
> > > > > > > with
> > > > > > > > > > > > different
> > > > > > > > > > > > > versions have the same name? Does the plugin
> isolation
> > > > > allow
> > > > > > > > this?
> > > > > > > > > > This
> > > > > > > > > > > > > could have a bearing when using the lifecycle
> > > endpoints for
> > > > > > > > > > connectors
> > > > > > > > > > > > like
> > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > >
> > > > > > > > > > > > All connectors in a cluster need to have uniquire
> > > connector
> > > > > names
> > > > > > > > > > > > regardless of what version of the plugin the
> connector is
> > > > > running
> > > > > > > > > > > > underneath. This is something enforced by the connect
> > > runtime
> > > > > > > > itself.
> > > > > > > > > > All
> > > > > > > > > > > > connect CRUD operations are keyed on the connector
> name
> > > so
> > > > > there
> > > > > > > > will
> > > > > > > > > > not
> > > > > > > > > > > > be an issue.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards
> > > > > > > > > > > > Snehashis
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <
> > > > > sagarmeansocean@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hey Snehashsih,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the KIP. It looks like a very useful
> > > feature.
> > > > > Couple
> > > > > > > > of
> > > > > > > > > > > > > small-ish points, let me know what you think:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1) Should we update the GET /connectors endpoint to
> > > > > include the
> > > > > > > > > > version of
> > > > > > > > > > > > > the plugin that is running? It could be useful to
> > > figure
> > > > > out
> > > > > > > the
> > > > > > > > > > version of
> > > > > > > > > > > > > the plugin or I am assuming it gets returned by the
> > > > > expand=info
> > > > > > > > call?
> > > > > > > > > > > > > 2) I am not aware of this and hence asking, can 2
> > > > > connectors
> > > > > > > with
> > > > > > > > > > different
> > > > > > > > > > > > > versions have the same name? Does the plugin
> isolation
> > > > > allow
> > > > > > > > this?
> > > > > > > > > > This
> > > > > > > > > > > > > could have a bearing when using the lifecycle
> > > endpoints for
> > > > > > > > > > connectors like
> > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > Sagar.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin
> > > > > > > > <apankaj@confluent.io.invalid
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > IIUC (please correct me if I am wrong here),
> what
> > > you
> > > > > > > > highlighted
> > > > > > > > > > > > > above,
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > a versioning scheme for a connector config for
> the
> > > same
> > > > > > > > connector
> > > > > > > > > > (and
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > different versions of a connector plugin).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Sorry for not being more precise in my wording
> - I
> > > meant
> > > > > > > > > > registering
> > > > > > > > > > > > > > versions of schema for connector config.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Let's take the example of a fictional connector
> which
> > > > > uses a
> > > > > > > > > > fictional
> > > > > > > > > > > > > AWS
> > > > > > > > > > > > > > service.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Fictional Connector Config schema version:2.0
> > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > {
> > > > > > > > > > > > > > "$schema": "
> > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > },
> > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > },
> > > > > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > },
> > > > > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > > },
> > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Fictional Connector config schema version:3.0
> > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > {
> > > > > > > > > > > > > > "$schema": "
> > > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > },
> > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > },
> > > > > > > > > > > > > > "iam_role": {
> > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > > },
> > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > "iam_role"
> > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The connector which supports Fictional config
> schema
> > > 2.0
> > > > > > > will
> > > > > > > > > > validate
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > access key and secret key.
> > > > > > > > > > > > > > Whereas a connector which supports config with
> schema
> > > > > version
> > > > > > > > 3.0
> > > > > > > > > > will
> > > > > > > > > > > > > only
> > > > > > > > > > > > > > validate the IAM role.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is the alternative which I wanted to
> suggest.
> > > Each
> > > > > > > plugin
> > > > > > > > will
> > > > > > > > > > > > > > register the schema versions of connector config
> > > which it
> > > > > > > > supports.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The plugin paths may be optionally different
> i.e we
> > > > > don't
> > > > > > > > have to
> > > > > > > > > > > > > > mandatorily add a new plugin path to support a
> new
> > > schema
> > > > > > > > version.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <
> > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. Can you elaborate on the rejected
> > > alternatives ?
> > > > > > > Suppose
> > > > > > > > > > connector
> > > > > > > > > > > > > > > > config is versioned and has a schema. Then a
> > > single
> > > > > > > plugin
> > > > > > > > > > (whose
> > > > > > > > > > > > > > > > dependencies have not changed) can handle
> > > multiple
> > > > > config
> > > > > > > > > > versions
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > IIUC (please correct me if I am wrong here),
> what
> > > you
> > > > > > > > highlighted
> > > > > > > > > > > > > above,
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > a versioning scheme for a connector config for
> the
> > > same
> > > > > > > > > > connector (and
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > different versions of a connector plugin).
> That is
> > > a
> > > > > > > somewhat
> > > > > > > > > > > > > tangential
> > > > > > > > > > > > > > > problem. While it is definitely a useful
> feature to
> > > > > have,
> > > > > > > > like a
> > > > > > > > > > log to
> > > > > > > > > > > > > > > check what changes were made over time to the
> > > config
> > > > > which
> > > > > > > > might
> > > > > > > > > > make
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > easier to do rollbacks, it is not the focus
> here.
> > > Here
> > > > > by
> > > > > > > > > > version we
> > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > to say what underlying version of the plugin
> > > should the
> > > > > > > given
> > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > of the connector use. Perhaps it is better to
> > > change
> > > > > the
> > > > > > > > name of
> > > > > > > > > > the
> > > > > > > > > > > > > > > parameter from connector.version to
> > > > > > > connector.plugin.version
> > > > > > > > or
> > > > > > > > > > > > > > > plugin.version if it was confusing. wdyt?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2. Any plans to support assisted migration
> e.g
> > > if a
> > > > > user
> > > > > > > > > > invokes
> > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > connector/config?migrate=latest", the latest
> > > version
> > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > transform the existing config to the newer
> > > version.
> > > > > This
> > > > > > > > would
> > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > adding a method like "boolean migrate(Version
> > > > > > > > fromVersion)" to
> > > > > > > > > > the
> > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This is an enhancement we can think of doing in
> > > future.
> > > > > > > > Users can
> > > > > > > > > > > > > simply
> > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > a PUT call with the updated config which has
> the
> > > > > updated
> > > > > > > > version
> > > > > > > > > > > > > number.
> > > > > > > > > > > > > > > The assisted mode could be handy as the user
> does
> > > not
> > > > > need
> > > > > > > to
> > > > > > > > > > know the
> > > > > > > > > > > > > > > config but beyond this it does not seem to
> justify
> > > its
> > > > > > > > existence.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin
> > > > > > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This is a really useful feature and thanks
> for
> > > > > initiating
> > > > > > > > this
> > > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I had the following questions -
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. Can you elaborate on the rejected
> > > alternatives ?
> > > > > > > Suppose
> > > > > > > > > > connector
> > > > > > > > > > > > > > > > config is versioned and has a schema. Then a
> > > single
> > > > > > > plugin
> > > > > > > > > > (whose
> > > > > > > > > > > > > > > > dependencies have not changed) can handle
> > > multiple
> > > > > config
> > > > > > > > > > versions
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2. Any plans to support assisted migration
> e.g
> > > if a
> > > > > user
> > > > > > > > > > invokes
> > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > connector/config?migrate=latest", the latest
> > > version
> > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > transform the existing config to the newer
> > > version.
> > > > > This
> > > > > > > > would
> > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > adding a method like "boolean migrate(Version
> > > > > > > > fromVersion)" to
> > > > > > > > > > the
> > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <
> > > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'd like to start a discussion thread on
> > > KIP-891:
> > > > > > > Running
> > > > > > > > > > multiple
> > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The KIP aims to add the ability for the
> connect
> > > > > runtime
> > > > > > > > to
> > > > > > > > > > run
> > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > versions of a connector.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Please take a look and let me know what you
> > > think.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
>
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Hey Chris,
Thanks for your comments, and I'm glad that it seems like we're
aligning on the vision here.
> An
> alternative could be to change existing behavior to fail fast on any
> invalid default converter configuration instead of just for invalid
> versions
I suppose if this is landing in 4.0, we have the opportunity to break
compatibility and strictly validate the worker class configs, and
could have a new consistent behavior instead of inconsistent but
backwards-compatible behavior.
> no other part of the KIP requires this change.
This is correct, and a compelling argument. I'm fine leaving the
strict worker validation off of this configuration to potentially be
added later. If a KIP was raised to perform strict validation of the
worker config, it would include the version config, and can address
backwards compatibility for both configs together.
> RE exposing the version property in the /connector-plugins/<plugin>/config
> endpoint, the behavior is inconsistent across plugin types.
Yes it is, and that started in a bugfix, once people started using
this endpoint: https://issues.apache.org/jira/browse/KAFKA-14843 . I
wasn't planning on converging the behavior in this KIP, as I
considered it out-of-scope, and was going to follow the current
behavior. To summarize:
GET /connector-plugins/<connector>/config will emit:
* connector.class (already implemented)
* connector.version (new)
* key.converter (already implemented)
* key.converter.version (new)
* value.converter (already implemented)
* value.converter.version (new)
* header.converter (already implemented)
* header.converter.version (new)
But will NOT emit:
* transforms.<alias>.type
* transforms.<alias>.version
* transforms.<alias>.predicate
* predicates.<alias>.type
* predicates.<alias>.version
* predicates.<alias>.negate
GET /connector-plugins/<converter>/config will NOT emit:
* "" (there's not even a ".class" prefix!)
* version
GET /connector-plugins/<transform>/config will NOT emit:
* type
* version
* predicate
GET /connector-plugins/<predicate>/config will NOT emit:
* type
* version
* negate
Do you want the converter, transform, and predicate endpoints changed?
Do you want just "version", or do you want all of the prefixed configs
including "type", "predicate" and "negate"? How would you want to
handle the converters?
And when I say the configs are "not part of the plugin config itself"
I mean that saying that GET
/connector-plugins/Flatten$Key/config?version=3.8.0 has a "version"
config that must be "3.8.0" is a little bit nonsense, as the version
is already specified.
> IMO
> it's worth including this information somewhere directly accessible without
> having to provide a full connector config. FWIW I'd be fine with GET
> /connector-plugins/<plugin>/versions as a first-class endpoint
You don't have to provide a configuration to call GET
/connector-plugins?connectorsOnly=false , is that endpoint not close
enough to what you have in mind? See also the Rejected Alternative
"Adding new REST API endpoints"
If you're calling /connector-plugins/<transform>/config, you know the
name of a plugin right? That either comes from out-of-band knowledge,
validating a connector config, or calling GET
/connector-plugins?connectorsOnly=false.
* If you have out-of-band knowledge of plugin classes, perhaps you
have out-of-band knowledge of versions too.
* If you've just validated a connector config, there should be an
accompanying "version" field there with an accurate default value and
recommenders.
* If you've called GET /connector-plugins?connectorsOnly=false, that
endpoint includes version information.
Thanks,
Greg
On Wed, May 22, 2024 at 11:05 AM Chris Egerton <ch...@aiven.io.invalid> wrote:
>
> Hi Greg,
>
> Hope you had a nice weekend! Gonna try to keep things concise.
>
> Concluded points:
>
> RE version recommenders, I agree it's likely that programmatic UIs will
> already be able to handle dynamic configuration definitions, and the detail
> about SMTs is a great point. I still anticipate some awkwardness with
> connector versions, though: if the latest version supports some new
> properties, then a user switches to an earlier version, a UI may respond by
> wiping values for these properties. I guess we can bite the bullet, though.
>
> RE double-dinging during preflight validation for invalid versions, I like
> the analogy with login credentials. I'm convinced that the proposal in the
> KIP is best 👍
>
> Continued points:
>
> RE failing on worker startup, sorry, I should be clearer: there is no _new_
> justification for it that doesn't also apply to existing behavior. We
> shouldn't diverge from existing behavior solely for this new case. An
> alternative could be to change existing behavior to fail fast on any
> invalid default converter configuration instead of just for invalid
> versions, but I'd vote to just stick to existing behavior and not
> complicate things, especially since no other part of the KIP requires this
> change.
>
> RE exposing the version property in the /connector-plugins/<plugin>/config
> endpoint, the behavior is inconsistent across plugin types. Hitting the
> endpoint for the FileStreamSinkConnector on version 3.7.0 yields a response
> that includes, among other things, the "topics", "topics.regex", and
> "errors.tolerance" properties. I see that we don't do this everywhere (the
> examples you cite for SMT and converter properties are accurate), but IMO
> it's worth including this information somewhere directly accessible without
> having to provide a full connector config. FWIW I'd be fine with GET
> /connector-plugins/<plugin>/versions as a first-class endpoint either
> instead of or in addition to adding recommended values for all plugin
> versions.
>
> Thanks for your continued work on this KIP, and with the progress we're
> making I'm optimistic about its chances of appearing in 4.0.0.
>
> Cheers,
>
> Chris
>
> On Wed, May 15, 2024 at 1:22 PM Greg Harris <gr...@aiven.io.invalid>
> wrote:
>
> > Hey Chris,
> >
> > Thanks for your quick follow up.
> >
> > > But this risk is already present with
> > > existing error cases, and I don't see anything that justifies changing
> > > existing behavior with an invalid converter class, or diverging from it
> > in
> > > the case of invalid converter versions.
> >
> > The justification is to fail-fast, and prevent REST API users from
> > receiving errors from bad configs that they didn't write, or maybe
> > don't even know apply to them.
> > Up until recently errors in these configurations surfaced as failures
> > to create the connector, or failures to start, and you made them
> > fail-fast during validation. I think this change is in the same
> > spirit, pulling the validation further forward and not letting errors
> > lie dormant.
> >
> > And to call back to your original concern about interrupting
> > connectors that explicitly provide these configurations and don't use
> > the worker configs: I expect that operators with a majority of these
> > sorts of clients aren't going to be setting the worker .version
> > properties, because it would have no effect on the majority of their
> > connectors. They would be able to rely on backwards-compatibility and
> > continue to ignore the class properties.
> >
> > > CLI
> > > and programmatic UI developers will want to develop their own tooling
> > > layers.
> >
> > This is a very compelling argument. Snehashis do you want to figure
> > out a REST API design for this use-case?
> >
> > > Regarding the GET /connector-plugins/<plugin>/config endpoint, I was
> > > thinking about the response for non-connector plugins, e.g.,
> > > GET /connector-plugins/RegexRouter/config. Would a "version" property
> > > appear with recommended values?
> >
> > I intended for the ".version" property to be like other framework
> > configs (".class", ".type", ".predicate", ".negate") where they are
> > inside the plugin namespace, but not part of the plugin config itself.
> > Perhaps we can deviate from those configs because it would aid in
> > discovering other valid `GET
> > /connector-plugins/<plugin>/config?version=` calls, without calling
> > `GET /connector-plugins?connectorsOnly=false`.
> > I don't really feel strongly either way.
> >
> > > if a user changes the
> > > connector version in, e.g., a dropdown menu, then the UI either has to
> > > re-fetch the ConfigDef for the new version, or risk operating on stale
> > > information
> >
> > This is a really interesting situation, thanks for finding that! This
> > is already a footgun with transformations and predicates; Once you
> > fill out a transformation class (which includes a recommender) you
> > will then see all of the transformations configurations.
> > I think this means that UI developers should have already developed
> > the infrastructure for handling dynamic recommenders, or else they've
> > had this bug since KIP-66 in 2017. It may require some manual
> > attention to roll-out support for the ".version" properties
> > specifically, but I don't think that should prevent us from providing
> > this information in a natural place.
> >
> > > but will it be possible to configure
> > > a connector to use two instances of the same transform or predicate, but
> > > with different versions for each?
> >
> > Yes, and this is included in the example in the KIP,
> > `transforms.flatten-latest` and `transforms.flatten-old`. This will
> > work in the natural way, with the two instances having a different
> > version if configured for that. If a user wants to pin the same
> > version of a plugin for every instance, they will need to provide the
> > version config multiple times and keep them in-sync themselves.
> >
> > > I would have expected the error
> > > to only be attributed to the version property, and for the class property
> > > to be reported as valid.
> >
> > I considered this, and then went back and changed it. To quote someone
> > else, this is to catch "misspelling cat as dog". For example a user
> > types DogConverter and meant to type CatConverter, and then types a
> > version which is valid for CatConverter, conceptually the error is in
> > the class name and not the version.
> > That's a very contrived scenario, but I think similar arguments are
> > used for attributing validation errors to endpoints/urls/hostnames
> > when the associated credentials are unable to log into the remote
> > system. Did the user provide the correct testing credentials, but
> > accidentally typed the production endpoint? Even if the production
> > endpoint looks valid (it's a real hostname that is reachable) the
> > conceptual error is still in the hostname and should have the error
> > attributed to it to draw the user's attention.
> > If that's not convincing, I think the alternative of only attributing
> > version errors to the version property is also acceptable.
> >
> > Thanks,
> > Greg
> >
> > On Wed, May 15, 2024 at 8:06 AM Chris Egerton <ch...@aiven.io.invalid>
> > wrote:
> > >
> > > Hi Greg,
> > >
> > > Thanks for your responses! Continuations of existing discussions:
> > >
> > > Regarding crashing the worker on startup--yes, there is also a risk to
> > > allowing it to join the cluster. But this risk is already present with
> > > existing error cases, and I don't see anything that justifies changing
> > > existing behavior with an invalid converter class, or diverging from it
> > in
> > > the case of invalid converter versions. I think we should keep this
> > simple
> > > and not do anything different for worker startup than we do already.
> > >
> > > As far as REST API vs. metrics go--I think you're right that the original
> > > version metrics were added as a "monitoring" detail. However, this was
> > back
> > > when plugin versions were managed solely by cluster administrators. With
> > > this KIP, connector users will be able to manage plugin versions, and CLI
> > > and programmatic UI developers will want to develop their own tooling
> > > layers. I think focusing on the REST API as the primary interface for
> > this
> > > KIP would be best for these users.
> > >
> > > (All that said, I don't object to the metrics that are proposed in the
> > KIP;
> > > I just think they make more sense in addition to new REST API
> > > functionality, as opposed to instead of it.)
> > >
> > > Regarding the GET /connector-plugins/<plugin>/config endpoint, I was
> > > thinking about the response for non-connector plugins, e.g.,
> > > GET /connector-plugins/RegexRouter/config. Would a "version" property
> > > appear with recommended values?
> > >
> > >
> > > And new thoughts:
> > >
> > > 1) Regarding the recommended values for "connector.version", this might
> > be
> > > confusing since there could be differences between the ConfigDefs for the
> > > latest version of the connector and prior versions. It also makes the
> > flow
> > > a bit awkward for programmatic UI developers: if a user changes the
> > > connector version in, e.g., a dropdown menu, then the UI either has to
> > > re-fetch the ConfigDef for the new version, or risk operating on stale
> > > information. I'm starting to doubt that exposing the range of available
> > > versions via recommended values is the best way to proceed, instead of a
> > > more explicit approach like GET /connector-plugins/<plugin>/versions, or
> > > the "Adding new REST API endpoints" rejected alternative.
> > >
> > > 2) I know that this is a bit heinous, but will it be possible to
> > configure
> > > a connector to use two instances of the same transform or predicate, but
> > > with different versions for each? (I don't think this is worth
> > significant
> > > design/implementation effort, so if it would inflate either of those,
> > > please don't feel obligated to support it.)
> > >
> > > 3) Just out of curiosity, why double-ding during config validation if the
> > > version for a plugin class can't be found? I would have expected the
> > error
> > > to only be attributed to the version property, and for the class property
> > > to be reported as valid.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Tue, May 14, 2024 at 1:11 PM Greg Harris <greg.harris@aiven.io.invalid
> > >
> > > wrote:
> > >
> > > > Hey Chris,
> > > >
> > > > Thanks for your comments. I'm glad you like the motivations, Snehashis
> > > > wrote that part!
> > > >
> > > > > the configuration syntax for the most basic use case of
> > > > > specifying a single desired version is pretty counterintuitive.
> > > >
> > > > I agree, and the "soft" requirement scheme is something I wasn't
> > > > explicitly looking for, but would be inherited from the library. I'm
> > > > fine with eliminating the soft requirement semantics, and having
> > > > "1.0.0" behave the same as "[1.0.0]". I'm less inclined to include
> > > > "range" in the property name, or have two properties.
> > > >
> > > > > Failing startup is drastic and has the potential to disrupt the
> > > > > availability of connectors that would otherwise be able to run
> > healthily
> > > > > because they were explicitly configured to use valid converters
> > instead
> > > > of
> > > > > the worker defaults.
> > > >
> > > > I think this argument cuts both ways. If someone reconfigures a worker
> > > > and adds an invalid ".version" string to the worker (or changes the
> > > > plugin.path to make it invalid), it would be permitted to enter the
> > > > group, and accept work assignments. If those work assignments used
> > > > these configurations, a set of tasks could transition to FAILED and
> > > > not be able to recover, because they would be restarted again on the
> > > > same worker.
> > > >
> > > > > Why are metrics utilized to report information about plugin versions
> > > > > utilized by connectors at runtime instead of publishing this info in
> > the
> > > > > REST API
> > > >
> > > > I was following the existing pattern for exposing runtime versions for
> > > > connectors, and it did seem like a "monitoring" feature. If that
> > > > approach is flawed and should be phased out, I think it would be a
> > > > good idea to reconsider the REST API rejected alternative.
> > > > We would need some additional design work to spec out the REST API
> > > > interface, as I don't have anything in mind currently.
> > > >
> > > > > I'm unclear on whether or not it'll be possible to see this
> > information
> > > > via the
> > > > > GET /connector-plugins/<plugin>/config endpoint
> > > >
> > > > There is room in the API to add recommenders for "key.converter",
> > > > "value.converter", and "header.converter", but not for transforms and
> > > > predicates, as they include aliases that depend on an actual
> > > > configuration. We could explicitly say we're going to do that, or do
> > > > whatever is convenient during the implementation phase, or leave it
> > > > open to be improved later.
> > > > There will not be any recommenders for ".version" properties in the
> > > > `/config` endpoint, because those recommenders are dynamic and depend
> > > > on an actual configuration.
> > > >
> > > > 5) There are two relevant lines in the KIP: "If a .version property
> > > > contains a hard requirement, select the latest installed version which
> > > > satisfies the requirement." and "This configuration is re-evaluated
> > > > each time the connector or task are assigned to a new worker". I would
> > > > call this "eager" upgrade behavior, rather than a "sticky" or "lazy"
> > > > upgrade behavior.
> > > >
> > > > 6) Updated!
> > > >
> > > > Thanks,
> > > > Greg
> > > >
> > > > On Tue, May 14, 2024 at 9:14 AM Chris Egerton <chrise@aiven.io.invalid
> > >
> > > > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Thanks Greg for updating the KIP, and thanks Snehashis for starting
> > the
> > > > > work on this originally.
> > > > >
> > > > > The motivation section makes a pretty convincing case for this kind
> > of
> > > > > feature. My thoughts are mostly about specific details:
> > > > >
> > > > > 1) I like the support for version ranges (the example demonstrating
> > how
> > > > to
> > > > > avoid KAFKA-10574 with the header converter was particularly
> > > > entertaining),
> > > > > but the configuration syntax for the most basic use case of
> > specifying a
> > > > > single desired version is pretty counterintuitive. People may get
> > bitten
> > > > or
> > > > > at least frustrated if they put connector.version=3.8.0 in a config
> > but
> > > > > then version 3.7.5 ends up running. I'd like it if we could either
> > > > > intentionally deviate from Maven ranges when a bare version is
> > present,
> > > > or
> > > > > separate things out into two properties: foo.version would be the
> > single
> > > > > accepted version for the foo plugin, and foo.version.range would use
> > > > Maven
> > > > > range syntax. Open to other options too, just providing a couple to
> > get
> > > > the
> > > > > ball rolling.
> > > > >
> > > > > 2) Although the current behavior for a worker with an invalid
> > > > > key/value/header converter class specified in its config file is a
> > little
> > > > > strange (I was surprised to learn that it wouldn't fail on startup),
> > I
> > > > > don't see a good reason to deviate from this when an invalid version
> > is
> > > > > specified. Failing startup is drastic and has the potential to
> > disrupt
> > > > the
> > > > > availability of connectors that would otherwise be able to run
> > healthily
> > > > > because they were explicitly configured to use valid converters
> > instead
> > > > of
> > > > > the worker defaults.
> > > > >
> > > > > 3) Why are metrics utilized to report information about plugin
> > versions
> > > > > utilized by connectors at runtime instead of publishing this info in
> > the
> > > > > REST API? I saw that this was mentioned as a rejected alternative,
> > but I
> > > > > didn't get a sense of why. It seems like the REST API would be
> > easier to
> > > > > access and more intuitive for most users than new metrics.
> > > > >
> > > > > 4) In the "Validation" section it's stated that "Users can use these
> > > > > recommenders to discover the valid plugin classes and versions,
> > without
> > > > > requiring an earlier call to GET
> > > > /connector-plugins?connectorsOnly=false."
> > > > > I really like the creativity and simplicity of reusing the
> > recommender
> > > > > mechanism to expose available versions for plugins via the REST API.
> > I'm
> > > > > unclear on whether or not it'll be possible to see this information
> > via
> > > > the
> > > > > GET /connector-plugins/<plugin>/config endpoint, though. It'd be
> > great if
> > > > > this were supported, since we learned in KIP-769 [1] that people
> > really
> > > > > want to be able to see configuration options for connectors and their
> > > > > plugins via some kind of GET endpoint without having to provide a
> > > > complete
> > > > > connector config for validation.
> > > > >
> > > > > 5) In the Maven version range docs, it's stated that "Maven picks the
> > > > > highest version of each project that satisfies all the hard
> > requirements
> > > > of
> > > > > the dependencies on that project." I'm guessing this behavior will be
> > > > > retained for Connect; i.e., the highest-possible version of each
> > plugin
> > > > > that satisfies the user-specified version constraints will be run?
> > (An
> > > > > alternative approach could be to have some kind of "sticky" logic
> > that
> > > > only
> > > > > restarts connectors/tasks when their currently-used version becomes
> > > > > incompatible with the configured constraints.)
> > > > >
> > > > > 6) (Nit) It'd be nice to add a link to the TestPlugins class or
> > somewhere
> > > > > in its neighborhood to the testing plan; unfamiliar readers probably
> > > > won't
> > > > > get much out of what's there right now.
> > > > >
> > > > > [1] -
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Mon, May 13, 2024 at 2:01 PM Snehashis <sn...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hi Greg,
> > > > > >
> > > > > > That is much appreciated. No complaints on the additional scope, I
> > will
> > > > > > make some time out to work on this once we have approval.
> > > > > >
> > > > > > Thanks
> > > > > > Snehashis
> > > > > >
> > > > > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> > > > <gr...@aiven.io.invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hey Snehashis,
> > > > > > >
> > > > > > > I'm glad to hear you're still interested in this KIP!
> > > > > > > I'm happy to let you drive this, and I apologize for increasing
> > the
> > > > > > > scope of work so drastically. To make up for that, I'll
> > volunteer to
> > > > > > > be the primary PR reviewer to help get this done quickly once
> > the KIP
> > > > > > > is approved.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Greg
> > > > > > >
> > > > > > >
> > > > > > > On Fri, May 10, 2024 at 3:51 AM Snehashis <
> > snehashisp1995@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Greg,
> > > > > > > >
> > > > > > > > Thanks for the follow up to my original KIP, I am in favour of
> > the
> > > > > > > > additions made to expand its scope, the addition of range
> > versions
> > > > > > > > specifically make a lot of sense.
> > > > > > > >
> > > > > > > > Apologies if I have not publicly worked on this KIP for a long
> > > > time.
> > > > > > The
> > > > > > > > original work was done when the move to service loading was in
> > > > > > discussion
> > > > > > > > and I wanted to loop back to this only after that work was
> > > > completed.
> > > > > > > Post
> > > > > > > > its conclusion, I have not been able to take this up due to
> > other
> > > > > > > > priorities. If it's okay with you, I would still like to get
> > this
> > > > > > > > implemented myself, including the additional scope.
> > > > > > > >
> > > > > > > > Thanks and regards
> > > > > > > > Snehashis
> > > > > > > >
> > > > > > > > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> > > > > > > <gr...@aiven.io.invalid>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I'd like to reboot the discussion on KIP-891:
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > > > >
> > > > > > > > > I've made some changes, most notably:
> > > > > > > > >
> > > > > > > > > 1. Specifying versions for all plugins in Connector configs
> > > > > > > > > (converters, header converters, transforms, and predicates)
> > not
> > > > just
> > > > > > > > > connectors & tasks
> > > > > > > > > 2. Specifying a range of versions instead of an exact match
> > > > > > > > > 3. New metrics to observe what versions are in-use
> > > > > > > > >
> > > > > > > > > Thanks to Snehashis for the original KIP idea!
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Greg
> > > > > > > > >
> > > > > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <
> > > > greg.harris@aiven.io>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Snehashis,
> > > > > > > > > >
> > > > > > > > > > Thank you for the KIP! This is something I've wanted for a
> > long
> > > > > > time.
> > > > > > > > > >
> > > > > > > > > > I know the discussion has gone cold, are you still
> > interested
> > > > in
> > > > > > > > > > pursuing this feature? I'll make time to review the KIP if
> > you
> > > > are
> > > > > > > > > > still accepting comments.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Greg
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis <
> > > > > > snehashisp1995@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the points Sagar.
> > > > > > > > > > >
> > > > > > > > > > > > 1) Should we update the GET /connectors endpoint to
> > > > include the
> > > > > > > > > version of
> > > > > > > > > > > > the plugin that is running? It could be useful to
> > figure
> > > > out
> > > > > > the
> > > > > > > > > version
> > > > > > > > > > > of
> > > > > > > > > > > > the plugin or I am assuming it gets returned by the
> > > > expand=info
> > > > > > > call?
> > > > > > > > > > >
> > > > > > > > > > > I think this is good to have and possible future
> > > > enhancement. The
> > > > > > > > > version
> > > > > > > > > > > info will be present in the config of the connector if
> > the
> > > > user
> > > > > > has
> > > > > > > > > > > specified the version. Otherwise it is the latest version
> > > > which
> > > > > > the
> > > > > > > > > user
> > > > > > > > > > > can find out from the connector-plugin endpoint. The
> > > > information
> > > > > > > can be
> > > > > > > > > > > introduced to the response of the GET /connectors
> > endpoint
> > > > > > itself,
> > > > > > > > > however
> > > > > > > > > > > the most ideal way of doing this would be to get the
> > > > currently
> > > > > > > running
> > > > > > > > > > > instance of the connector and get the version directly
> > from
> > > > > > there.
> > > > > > > > > This is
> > > > > > > > > > > slightly tricky as the connector could be running in a
> > > > different
> > > > > > > node.
> > > > > > > > > > > One way to do this would be to persist the version
> > > > information in
> > > > > > > the
> > > > > > > > > > > status backing store during instantiation of the
> > connector.
> > > > It
> > > > > > > requires
> > > > > > > > > > > some more thought and since the version is part of the
> > > > configs if
> > > > > > > > > provided
> > > > > > > > > > > and evident otherwise, I have not included it in this
> > KIP.
> > > > > > > > > > >
> > > > > > > > > > > > 2) I am not aware of this and hence asking, can 2
> > > > connectors
> > > > > > with
> > > > > > > > > > > different
> > > > > > > > > > > > versions have the same name? Does the plugin isolation
> > > > allow
> > > > > > > this?
> > > > > > > > > This
> > > > > > > > > > > > could have a bearing when using the lifecycle
> > endpoints for
> > > > > > > > > connectors
> > > > > > > > > > > like
> > > > > > > > > > > > DELETE etc.
> > > > > > > > > > >
> > > > > > > > > > > All connectors in a cluster need to have uniquire
> > connector
> > > > names
> > > > > > > > > > > regardless of what version of the plugin the connector is
> > > > running
> > > > > > > > > > > underneath. This is something enforced by the connect
> > runtime
> > > > > > > itself.
> > > > > > > > > All
> > > > > > > > > > > connect CRUD operations are keyed on the connector name
> > so
> > > > there
> > > > > > > will
> > > > > > > > > not
> > > > > > > > > > > be an issue.
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > > Snehashis
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <
> > > > sagarmeansocean@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hey Snehashsih,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the KIP. It looks like a very useful
> > feature.
> > > > Couple
> > > > > > > of
> > > > > > > > > > > > small-ish points, let me know what you think:
> > > > > > > > > > > >
> > > > > > > > > > > > 1) Should we update the GET /connectors endpoint to
> > > > include the
> > > > > > > > > version of
> > > > > > > > > > > > the plugin that is running? It could be useful to
> > figure
> > > > out
> > > > > > the
> > > > > > > > > version of
> > > > > > > > > > > > the plugin or I am assuming it gets returned by the
> > > > expand=info
> > > > > > > call?
> > > > > > > > > > > > 2) I am not aware of this and hence asking, can 2
> > > > connectors
> > > > > > with
> > > > > > > > > different
> > > > > > > > > > > > versions have the same name? Does the plugin isolation
> > > > allow
> > > > > > > this?
> > > > > > > > > This
> > > > > > > > > > > > could have a bearing when using the lifecycle
> > endpoints for
> > > > > > > > > connectors like
> > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks!
> > > > > > > > > > > > Sagar.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin
> > > > > > > <apankaj@confluent.io.invalid
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > >
> > > > > > > > > > > > > > IIUC (please correct me if I am wrong here), what
> > you
> > > > > > > highlighted
> > > > > > > > > > > > above,
> > > > > > > > > > > > > is
> > > > > > > > > > > > > a versioning scheme for a connector config for the
> > same
> > > > > > > connector
> > > > > > > > > (and
> > > > > > > > > > > > not
> > > > > > > > > > > > > different versions of a connector plugin).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Sorry for not being more precise in my wording - I
> > meant
> > > > > > > > > registering
> > > > > > > > > > > > > versions of schema for connector config.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Let's take the example of a fictional connector which
> > > > uses a
> > > > > > > > > fictional
> > > > > > > > > > > > AWS
> > > > > > > > > > > > > service.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Fictional Connector Config schema version:2.0
> > > > > > > > > > > > > ---
> > > > > > > > > > > > > {
> > > > > > > > > > > > > "$schema": "
> > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > },
> > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > },
> > > > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > },
> > > > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > }
> > > > > > > > > > > > > },
> > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > "name",
> > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > > > ]
> > > > > > > > > > > > > }
> > > > > > > > > > > > >
> > > > > > > > > > > > > Fictional Connector config schema version:3.0
> > > > > > > > > > > > > ---
> > > > > > > > > > > > > {
> > > > > > > > > > > > > "$schema": "
> > http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > },
> > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > },
> > > > > > > > > > > > > "iam_role": {
> > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > }
> > > > > > > > > > > > > },
> > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > "name",
> > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > "iam_role"
> > > > > > > > > > > > > ]
> > > > > > > > > > > > > }
> > > > > > > > > > > > >
> > > > > > > > > > > > > The connector which supports Fictional config schema
> > 2.0
> > > > > > will
> > > > > > > > > validate
> > > > > > > > > > > > the
> > > > > > > > > > > > > access key and secret key.
> > > > > > > > > > > > > Whereas a connector which supports config with schema
> > > > version
> > > > > > > 3.0
> > > > > > > > > will
> > > > > > > > > > > > only
> > > > > > > > > > > > > validate the IAM role.
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is the alternative which I wanted to suggest.
> > Each
> > > > > > plugin
> > > > > > > will
> > > > > > > > > > > > > register the schema versions of connector config
> > which it
> > > > > > > supports.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The plugin paths may be optionally different i.e we
> > > > don't
> > > > > > > have to
> > > > > > > > > > > > > mandatorily add a new plugin path to support a new
> > schema
> > > > > > > version.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <
> > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. Can you elaborate on the rejected
> > alternatives ?
> > > > > > Suppose
> > > > > > > > > connector
> > > > > > > > > > > > > > > config is versioned and has a schema. Then a
> > single
> > > > > > plugin
> > > > > > > > > (whose
> > > > > > > > > > > > > > > dependencies have not changed) can handle
> > multiple
> > > > config
> > > > > > > > > versions
> > > > > > > > > > > > for
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > IIUC (please correct me if I am wrong here), what
> > you
> > > > > > > highlighted
> > > > > > > > > > > > above,
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > a versioning scheme for a connector config for the
> > same
> > > > > > > > > connector (and
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > different versions of a connector plugin). That is
> > a
> > > > > > somewhat
> > > > > > > > > > > > tangential
> > > > > > > > > > > > > > problem. While it is definitely a useful feature to
> > > > have,
> > > > > > > like a
> > > > > > > > > log to
> > > > > > > > > > > > > > check what changes were made over time to the
> > config
> > > > which
> > > > > > > might
> > > > > > > > > make
> > > > > > > > > > > > it
> > > > > > > > > > > > > > easier to do rollbacks, it is not the focus here.
> > Here
> > > > by
> > > > > > > > > version we
> > > > > > > > > > > > mean
> > > > > > > > > > > > > > to say what underlying version of the plugin
> > should the
> > > > > > given
> > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > of the connector use. Perhaps it is better to
> > change
> > > > the
> > > > > > > name of
> > > > > > > > > the
> > > > > > > > > > > > > > parameter from connector.version to
> > > > > > connector.plugin.version
> > > > > > > or
> > > > > > > > > > > > > > plugin.version if it was confusing. wdyt?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. Any plans to support assisted migration e.g
> > if a
> > > > user
> > > > > > > > > invokes
> > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > connector/config?migrate=latest", the latest
> > version
> > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > transform the existing config to the newer
> > version.
> > > > This
> > > > > > > would
> > > > > > > > > > > > require
> > > > > > > > > > > > > > > adding a method like "boolean migrate(Version
> > > > > > > fromVersion)" to
> > > > > > > > > the
> > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is an enhancement we can think of doing in
> > future.
> > > > > > > Users can
> > > > > > > > > > > > simply
> > > > > > > > > > > > > do
> > > > > > > > > > > > > > a PUT call with the updated config which has the
> > > > updated
> > > > > > > version
> > > > > > > > > > > > number.
> > > > > > > > > > > > > > The assisted mode could be handy as the user does
> > not
> > > > need
> > > > > > to
> > > > > > > > > know the
> > > > > > > > > > > > > > config but beyond this it does not seem to justify
> > its
> > > > > > > existence.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin
> > > > > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This is a really useful feature and thanks for
> > > > initiating
> > > > > > > this
> > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I had the following questions -
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. Can you elaborate on the rejected
> > alternatives ?
> > > > > > Suppose
> > > > > > > > > connector
> > > > > > > > > > > > > > > config is versioned and has a schema. Then a
> > single
> > > > > > plugin
> > > > > > > > > (whose
> > > > > > > > > > > > > > > dependencies have not changed) can handle
> > multiple
> > > > config
> > > > > > > > > versions
> > > > > > > > > > > > for
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. Any plans to support assisted migration e.g
> > if a
> > > > user
> > > > > > > > > invokes
> > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > connector/config?migrate=latest", the latest
> > version
> > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > transform the existing config to the newer
> > version.
> > > > This
> > > > > > > would
> > > > > > > > > > > > require
> > > > > > > > > > > > > > > adding a method like "boolean migrate(Version
> > > > > > > fromVersion)" to
> > > > > > > > > the
> > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <
> > > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'd like to start a discussion thread on
> > KIP-891:
> > > > > > Running
> > > > > > > > > multiple
> > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The KIP aims to add the ability for the connect
> > > > runtime
> > > > > > > to
> > > > > > > > > run
> > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > versions of a connector.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Please take a look and let me know what you
> > think.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Chris Egerton <ch...@aiven.io.INVALID>.
Hi Greg,
Hope you had a nice weekend! Gonna try to keep things concise.
Concluded points:
RE version recommenders, I agree it's likely that programmatic UIs will
already be able to handle dynamic configuration definitions, and the detail
about SMTs is a great point. I still anticipate some awkwardness with
connector versions, though: if the latest version supports some new
properties, then a user switches to an earlier version, a UI may respond by
wiping values for these properties. I guess we can bite the bullet, though.
RE double-dinging during preflight validation for invalid versions, I like
the analogy with login credentials. I'm convinced that the proposal in the
KIP is best 👍
Continued points:
RE failing on worker startup, sorry, I should be clearer: there is no _new_
justification for it that doesn't also apply to existing behavior. We
shouldn't diverge from existing behavior solely for this new case. An
alternative could be to change existing behavior to fail fast on any
invalid default converter configuration instead of just for invalid
versions, but I'd vote to just stick to existing behavior and not
complicate things, especially since no other part of the KIP requires this
change.
RE exposing the version property in the /connector-plugins/<plugin>/config
endpoint, the behavior is inconsistent across plugin types. Hitting the
endpoint for the FileStreamSinkConnector on version 3.7.0 yields a response
that includes, among other things, the "topics", "topics.regex", and
"errors.tolerance" properties. I see that we don't do this everywhere (the
examples you cite for SMT and converter properties are accurate), but IMO
it's worth including this information somewhere directly accessible without
having to provide a full connector config. FWIW I'd be fine with GET
/connector-plugins/<plugin>/versions as a first-class endpoint either
instead of or in addition to adding recommended values for all plugin
versions.
Thanks for your continued work on this KIP, and with the progress we're
making I'm optimistic about its chances of appearing in 4.0.0.
Cheers,
Chris
On Wed, May 15, 2024 at 1:22 PM Greg Harris <gr...@aiven.io.invalid>
wrote:
> Hey Chris,
>
> Thanks for your quick follow up.
>
> > But this risk is already present with
> > existing error cases, and I don't see anything that justifies changing
> > existing behavior with an invalid converter class, or diverging from it
> in
> > the case of invalid converter versions.
>
> The justification is to fail-fast, and prevent REST API users from
> receiving errors from bad configs that they didn't write, or maybe
> don't even know apply to them.
> Up until recently errors in these configurations surfaced as failures
> to create the connector, or failures to start, and you made them
> fail-fast during validation. I think this change is in the same
> spirit, pulling the validation further forward and not letting errors
> lie dormant.
>
> And to call back to your original concern about interrupting
> connectors that explicitly provide these configurations and don't use
> the worker configs: I expect that operators with a majority of these
> sorts of clients aren't going to be setting the worker .version
> properties, because it would have no effect on the majority of their
> connectors. They would be able to rely on backwards-compatibility and
> continue to ignore the class properties.
>
> > CLI
> > and programmatic UI developers will want to develop their own tooling
> > layers.
>
> This is a very compelling argument. Snehashis do you want to figure
> out a REST API design for this use-case?
>
> > Regarding the GET /connector-plugins/<plugin>/config endpoint, I was
> > thinking about the response for non-connector plugins, e.g.,
> > GET /connector-plugins/RegexRouter/config. Would a "version" property
> > appear with recommended values?
>
> I intended for the ".version" property to be like other framework
> configs (".class", ".type", ".predicate", ".negate") where they are
> inside the plugin namespace, but not part of the plugin config itself.
> Perhaps we can deviate from those configs because it would aid in
> discovering other valid `GET
> /connector-plugins/<plugin>/config?version=` calls, without calling
> `GET /connector-plugins?connectorsOnly=false`.
> I don't really feel strongly either way.
>
> > if a user changes the
> > connector version in, e.g., a dropdown menu, then the UI either has to
> > re-fetch the ConfigDef for the new version, or risk operating on stale
> > information
>
> This is a really interesting situation, thanks for finding that! This
> is already a footgun with transformations and predicates; Once you
> fill out a transformation class (which includes a recommender) you
> will then see all of the transformations configurations.
> I think this means that UI developers should have already developed
> the infrastructure for handling dynamic recommenders, or else they've
> had this bug since KIP-66 in 2017. It may require some manual
> attention to roll-out support for the ".version" properties
> specifically, but I don't think that should prevent us from providing
> this information in a natural place.
>
> > but will it be possible to configure
> > a connector to use two instances of the same transform or predicate, but
> > with different versions for each?
>
> Yes, and this is included in the example in the KIP,
> `transforms.flatten-latest` and `transforms.flatten-old`. This will
> work in the natural way, with the two instances having a different
> version if configured for that. If a user wants to pin the same
> version of a plugin for every instance, they will need to provide the
> version config multiple times and keep them in-sync themselves.
>
> > I would have expected the error
> > to only be attributed to the version property, and for the class property
> > to be reported as valid.
>
> I considered this, and then went back and changed it. To quote someone
> else, this is to catch "misspelling cat as dog". For example a user
> types DogConverter and meant to type CatConverter, and then types a
> version which is valid for CatConverter, conceptually the error is in
> the class name and not the version.
> That's a very contrived scenario, but I think similar arguments are
> used for attributing validation errors to endpoints/urls/hostnames
> when the associated credentials are unable to log into the remote
> system. Did the user provide the correct testing credentials, but
> accidentally typed the production endpoint? Even if the production
> endpoint looks valid (it's a real hostname that is reachable) the
> conceptual error is still in the hostname and should have the error
> attributed to it to draw the user's attention.
> If that's not convincing, I think the alternative of only attributing
> version errors to the version property is also acceptable.
>
> Thanks,
> Greg
>
> On Wed, May 15, 2024 at 8:06 AM Chris Egerton <ch...@aiven.io.invalid>
> wrote:
> >
> > Hi Greg,
> >
> > Thanks for your responses! Continuations of existing discussions:
> >
> > Regarding crashing the worker on startup--yes, there is also a risk to
> > allowing it to join the cluster. But this risk is already present with
> > existing error cases, and I don't see anything that justifies changing
> > existing behavior with an invalid converter class, or diverging from it
> in
> > the case of invalid converter versions. I think we should keep this
> simple
> > and not do anything different for worker startup than we do already.
> >
> > As far as REST API vs. metrics go--I think you're right that the original
> > version metrics were added as a "monitoring" detail. However, this was
> back
> > when plugin versions were managed solely by cluster administrators. With
> > this KIP, connector users will be able to manage plugin versions, and CLI
> > and programmatic UI developers will want to develop their own tooling
> > layers. I think focusing on the REST API as the primary interface for
> this
> > KIP would be best for these users.
> >
> > (All that said, I don't object to the metrics that are proposed in the
> KIP;
> > I just think they make more sense in addition to new REST API
> > functionality, as opposed to instead of it.)
> >
> > Regarding the GET /connector-plugins/<plugin>/config endpoint, I was
> > thinking about the response for non-connector plugins, e.g.,
> > GET /connector-plugins/RegexRouter/config. Would a "version" property
> > appear with recommended values?
> >
> >
> > And new thoughts:
> >
> > 1) Regarding the recommended values for "connector.version", this might
> be
> > confusing since there could be differences between the ConfigDefs for the
> > latest version of the connector and prior versions. It also makes the
> flow
> > a bit awkward for programmatic UI developers: if a user changes the
> > connector version in, e.g., a dropdown menu, then the UI either has to
> > re-fetch the ConfigDef for the new version, or risk operating on stale
> > information. I'm starting to doubt that exposing the range of available
> > versions via recommended values is the best way to proceed, instead of a
> > more explicit approach like GET /connector-plugins/<plugin>/versions, or
> > the "Adding new REST API endpoints" rejected alternative.
> >
> > 2) I know that this is a bit heinous, but will it be possible to
> configure
> > a connector to use two instances of the same transform or predicate, but
> > with different versions for each? (I don't think this is worth
> significant
> > design/implementation effort, so if it would inflate either of those,
> > please don't feel obligated to support it.)
> >
> > 3) Just out of curiosity, why double-ding during config validation if the
> > version for a plugin class can't be found? I would have expected the
> error
> > to only be attributed to the version property, and for the class property
> > to be reported as valid.
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, May 14, 2024 at 1:11 PM Greg Harris <greg.harris@aiven.io.invalid
> >
> > wrote:
> >
> > > Hey Chris,
> > >
> > > Thanks for your comments. I'm glad you like the motivations, Snehashis
> > > wrote that part!
> > >
> > > > the configuration syntax for the most basic use case of
> > > > specifying a single desired version is pretty counterintuitive.
> > >
> > > I agree, and the "soft" requirement scheme is something I wasn't
> > > explicitly looking for, but would be inherited from the library. I'm
> > > fine with eliminating the soft requirement semantics, and having
> > > "1.0.0" behave the same as "[1.0.0]". I'm less inclined to include
> > > "range" in the property name, or have two properties.
> > >
> > > > Failing startup is drastic and has the potential to disrupt the
> > > > availability of connectors that would otherwise be able to run
> healthily
> > > > because they were explicitly configured to use valid converters
> instead
> > > of
> > > > the worker defaults.
> > >
> > > I think this argument cuts both ways. If someone reconfigures a worker
> > > and adds an invalid ".version" string to the worker (or changes the
> > > plugin.path to make it invalid), it would be permitted to enter the
> > > group, and accept work assignments. If those work assignments used
> > > these configurations, a set of tasks could transition to FAILED and
> > > not be able to recover, because they would be restarted again on the
> > > same worker.
> > >
> > > > Why are metrics utilized to report information about plugin versions
> > > > utilized by connectors at runtime instead of publishing this info in
> the
> > > > REST API
> > >
> > > I was following the existing pattern for exposing runtime versions for
> > > connectors, and it did seem like a "monitoring" feature. If that
> > > approach is flawed and should be phased out, I think it would be a
> > > good idea to reconsider the REST API rejected alternative.
> > > We would need some additional design work to spec out the REST API
> > > interface, as I don't have anything in mind currently.
> > >
> > > > I'm unclear on whether or not it'll be possible to see this
> information
> > > via the
> > > > GET /connector-plugins/<plugin>/config endpoint
> > >
> > > There is room in the API to add recommenders for "key.converter",
> > > "value.converter", and "header.converter", but not for transforms and
> > > predicates, as they include aliases that depend on an actual
> > > configuration. We could explicitly say we're going to do that, or do
> > > whatever is convenient during the implementation phase, or leave it
> > > open to be improved later.
> > > There will not be any recommenders for ".version" properties in the
> > > `/config` endpoint, because those recommenders are dynamic and depend
> > > on an actual configuration.
> > >
> > > 5) There are two relevant lines in the KIP: "If a .version property
> > > contains a hard requirement, select the latest installed version which
> > > satisfies the requirement." and "This configuration is re-evaluated
> > > each time the connector or task are assigned to a new worker". I would
> > > call this "eager" upgrade behavior, rather than a "sticky" or "lazy"
> > > upgrade behavior.
> > >
> > > 6) Updated!
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Tue, May 14, 2024 at 9:14 AM Chris Egerton <chrise@aiven.io.invalid
> >
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Thanks Greg for updating the KIP, and thanks Snehashis for starting
> the
> > > > work on this originally.
> > > >
> > > > The motivation section makes a pretty convincing case for this kind
> of
> > > > feature. My thoughts are mostly about specific details:
> > > >
> > > > 1) I like the support for version ranges (the example demonstrating
> how
> > > to
> > > > avoid KAFKA-10574 with the header converter was particularly
> > > entertaining),
> > > > but the configuration syntax for the most basic use case of
> specifying a
> > > > single desired version is pretty counterintuitive. People may get
> bitten
> > > or
> > > > at least frustrated if they put connector.version=3.8.0 in a config
> but
> > > > then version 3.7.5 ends up running. I'd like it if we could either
> > > > intentionally deviate from Maven ranges when a bare version is
> present,
> > > or
> > > > separate things out into two properties: foo.version would be the
> single
> > > > accepted version for the foo plugin, and foo.version.range would use
> > > Maven
> > > > range syntax. Open to other options too, just providing a couple to
> get
> > > the
> > > > ball rolling.
> > > >
> > > > 2) Although the current behavior for a worker with an invalid
> > > > key/value/header converter class specified in its config file is a
> little
> > > > strange (I was surprised to learn that it wouldn't fail on startup),
> I
> > > > don't see a good reason to deviate from this when an invalid version
> is
> > > > specified. Failing startup is drastic and has the potential to
> disrupt
> > > the
> > > > availability of connectors that would otherwise be able to run
> healthily
> > > > because they were explicitly configured to use valid converters
> instead
> > > of
> > > > the worker defaults.
> > > >
> > > > 3) Why are metrics utilized to report information about plugin
> versions
> > > > utilized by connectors at runtime instead of publishing this info in
> the
> > > > REST API? I saw that this was mentioned as a rejected alternative,
> but I
> > > > didn't get a sense of why. It seems like the REST API would be
> easier to
> > > > access and more intuitive for most users than new metrics.
> > > >
> > > > 4) In the "Validation" section it's stated that "Users can use these
> > > > recommenders to discover the valid plugin classes and versions,
> without
> > > > requiring an earlier call to GET
> > > /connector-plugins?connectorsOnly=false."
> > > > I really like the creativity and simplicity of reusing the
> recommender
> > > > mechanism to expose available versions for plugins via the REST API.
> I'm
> > > > unclear on whether or not it'll be possible to see this information
> via
> > > the
> > > > GET /connector-plugins/<plugin>/config endpoint, though. It'd be
> great if
> > > > this were supported, since we learned in KIP-769 [1] that people
> really
> > > > want to be able to see configuration options for connectors and their
> > > > plugins via some kind of GET endpoint without having to provide a
> > > complete
> > > > connector config for validation.
> > > >
> > > > 5) In the Maven version range docs, it's stated that "Maven picks the
> > > > highest version of each project that satisfies all the hard
> requirements
> > > of
> > > > the dependencies on that project." I'm guessing this behavior will be
> > > > retained for Connect; i.e., the highest-possible version of each
> plugin
> > > > that satisfies the user-specified version constraints will be run?
> (An
> > > > alternative approach could be to have some kind of "sticky" logic
> that
> > > only
> > > > restarts connectors/tasks when their currently-used version becomes
> > > > incompatible with the configured constraints.)
> > > >
> > > > 6) (Nit) It'd be nice to add a link to the TestPlugins class or
> somewhere
> > > > in its neighborhood to the testing plan; unfamiliar readers probably
> > > won't
> > > > get much out of what's there right now.
> > > >
> > > > [1] -
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Mon, May 13, 2024 at 2:01 PM Snehashis <sn...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Greg,
> > > > >
> > > > > That is much appreciated. No complaints on the additional scope, I
> will
> > > > > make some time out to work on this once we have approval.
> > > > >
> > > > > Thanks
> > > > > Snehashis
> > > > >
> > > > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> > > <gr...@aiven.io.invalid>
> > > > > wrote:
> > > > >
> > > > > > Hey Snehashis,
> > > > > >
> > > > > > I'm glad to hear you're still interested in this KIP!
> > > > > > I'm happy to let you drive this, and I apologize for increasing
> the
> > > > > > scope of work so drastically. To make up for that, I'll
> volunteer to
> > > > > > be the primary PR reviewer to help get this done quickly once
> the KIP
> > > > > > is approved.
> > > > > >
> > > > > > Thanks,
> > > > > > Greg
> > > > > >
> > > > > >
> > > > > > On Fri, May 10, 2024 at 3:51 AM Snehashis <
> snehashisp1995@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Greg,
> > > > > > >
> > > > > > > Thanks for the follow up to my original KIP, I am in favour of
> the
> > > > > > > additions made to expand its scope, the addition of range
> versions
> > > > > > > specifically make a lot of sense.
> > > > > > >
> > > > > > > Apologies if I have not publicly worked on this KIP for a long
> > > time.
> > > > > The
> > > > > > > original work was done when the move to service loading was in
> > > > > discussion
> > > > > > > and I wanted to loop back to this only after that work was
> > > completed.
> > > > > > Post
> > > > > > > its conclusion, I have not been able to take this up due to
> other
> > > > > > > priorities. If it's okay with you, I would still like to get
> this
> > > > > > > implemented myself, including the additional scope.
> > > > > > >
> > > > > > > Thanks and regards
> > > > > > > Snehashis
> > > > > > >
> > > > > > > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> > > > > > <gr...@aiven.io.invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I'd like to reboot the discussion on KIP-891:
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > > >
> > > > > > > > I've made some changes, most notably:
> > > > > > > >
> > > > > > > > 1. Specifying versions for all plugins in Connector configs
> > > > > > > > (converters, header converters, transforms, and predicates)
> not
> > > just
> > > > > > > > connectors & tasks
> > > > > > > > 2. Specifying a range of versions instead of an exact match
> > > > > > > > 3. New metrics to observe what versions are in-use
> > > > > > > >
> > > > > > > > Thanks to Snehashis for the original KIP idea!
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Greg
> > > > > > > >
> > > > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <
> > > greg.harris@aiven.io>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Snehashis,
> > > > > > > > >
> > > > > > > > > Thank you for the KIP! This is something I've wanted for a
> long
> > > > > time.
> > > > > > > > >
> > > > > > > > > I know the discussion has gone cold, are you still
> interested
> > > in
> > > > > > > > > pursuing this feature? I'll make time to review the KIP if
> you
> > > are
> > > > > > > > > still accepting comments.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Greg
> > > > > > > > >
> > > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis <
> > > > > snehashisp1995@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Thanks for the points Sagar.
> > > > > > > > > >
> > > > > > > > > > > 1) Should we update the GET /connectors endpoint to
> > > include the
> > > > > > > > version of
> > > > > > > > > > > the plugin that is running? It could be useful to
> figure
> > > out
> > > > > the
> > > > > > > > version
> > > > > > > > > > of
> > > > > > > > > > > the plugin or I am assuming it gets returned by the
> > > expand=info
> > > > > > call?
> > > > > > > > > >
> > > > > > > > > > I think this is good to have and possible future
> > > enhancement. The
> > > > > > > > version
> > > > > > > > > > info will be present in the config of the connector if
> the
> > > user
> > > > > has
> > > > > > > > > > specified the version. Otherwise it is the latest version
> > > which
> > > > > the
> > > > > > > > user
> > > > > > > > > > can find out from the connector-plugin endpoint. The
> > > information
> > > > > > can be
> > > > > > > > > > introduced to the response of the GET /connectors
> endpoint
> > > > > itself,
> > > > > > > > however
> > > > > > > > > > the most ideal way of doing this would be to get the
> > > currently
> > > > > > running
> > > > > > > > > > instance of the connector and get the version directly
> from
> > > > > there.
> > > > > > > > This is
> > > > > > > > > > slightly tricky as the connector could be running in a
> > > different
> > > > > > node.
> > > > > > > > > > One way to do this would be to persist the version
> > > information in
> > > > > > the
> > > > > > > > > > status backing store during instantiation of the
> connector.
> > > It
> > > > > > requires
> > > > > > > > > > some more thought and since the version is part of the
> > > configs if
> > > > > > > > provided
> > > > > > > > > > and evident otherwise, I have not included it in this
> KIP.
> > > > > > > > > >
> > > > > > > > > > > 2) I am not aware of this and hence asking, can 2
> > > connectors
> > > > > with
> > > > > > > > > > different
> > > > > > > > > > > versions have the same name? Does the plugin isolation
> > > allow
> > > > > > this?
> > > > > > > > This
> > > > > > > > > > > could have a bearing when using the lifecycle
> endpoints for
> > > > > > > > connectors
> > > > > > > > > > like
> > > > > > > > > > > DELETE etc.
> > > > > > > > > >
> > > > > > > > > > All connectors in a cluster need to have uniquire
> connector
> > > names
> > > > > > > > > > regardless of what version of the plugin the connector is
> > > running
> > > > > > > > > > underneath. This is something enforced by the connect
> runtime
> > > > > > itself.
> > > > > > > > All
> > > > > > > > > > connect CRUD operations are keyed on the connector name
> so
> > > there
> > > > > > will
> > > > > > > > not
> > > > > > > > > > be an issue.
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > > Snehashis
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <
> > > sagarmeansocean@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey Snehashsih,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the KIP. It looks like a very useful
> feature.
> > > Couple
> > > > > > of
> > > > > > > > > > > small-ish points, let me know what you think:
> > > > > > > > > > >
> > > > > > > > > > > 1) Should we update the GET /connectors endpoint to
> > > include the
> > > > > > > > version of
> > > > > > > > > > > the plugin that is running? It could be useful to
> figure
> > > out
> > > > > the
> > > > > > > > version of
> > > > > > > > > > > the plugin or I am assuming it gets returned by the
> > > expand=info
> > > > > > call?
> > > > > > > > > > > 2) I am not aware of this and hence asking, can 2
> > > connectors
> > > > > with
> > > > > > > > different
> > > > > > > > > > > versions have the same name? Does the plugin isolation
> > > allow
> > > > > > this?
> > > > > > > > This
> > > > > > > > > > > could have a bearing when using the lifecycle
> endpoints for
> > > > > > > > connectors like
> > > > > > > > > > > DELETE etc.
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > > Sagar.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin
> > > > > > <apankaj@confluent.io.invalid
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > >
> > > > > > > > > > > > > IIUC (please correct me if I am wrong here), what
> you
> > > > > > highlighted
> > > > > > > > > > > above,
> > > > > > > > > > > > is
> > > > > > > > > > > > a versioning scheme for a connector config for the
> same
> > > > > > connector
> > > > > > > > (and
> > > > > > > > > > > not
> > > > > > > > > > > > different versions of a connector plugin).
> > > > > > > > > > > >
> > > > > > > > > > > > Sorry for not being more precise in my wording - I
> meant
> > > > > > > > registering
> > > > > > > > > > > > versions of schema for connector config.
> > > > > > > > > > > >
> > > > > > > > > > > > Let's take the example of a fictional connector which
> > > uses a
> > > > > > > > fictional
> > > > > > > > > > > AWS
> > > > > > > > > > > > service.
> > > > > > > > > > > >
> > > > > > > > > > > > Fictional Connector Config schema version:2.0
> > > > > > > > > > > > ---
> > > > > > > > > > > > {
> > > > > > > > > > > > "$schema": "
> http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > "properties": {
> > > > > > > > > > > > "name": {
> > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > },
> > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > },
> > > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > },
> > > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > }
> > > > > > > > > > > > },
> > > > > > > > > > > > "required": [
> > > > > > > > > > > > "name",
> > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > > ]
> > > > > > > > > > > > }
> > > > > > > > > > > >
> > > > > > > > > > > > Fictional Connector config schema version:3.0
> > > > > > > > > > > > ---
> > > > > > > > > > > > {
> > > > > > > > > > > > "$schema": "
> http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > "properties": {
> > > > > > > > > > > > "name": {
> > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > },
> > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > },
> > > > > > > > > > > > "iam_role": {
> > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > }
> > > > > > > > > > > > },
> > > > > > > > > > > > "required": [
> > > > > > > > > > > > "name",
> > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > "iam_role"
> > > > > > > > > > > > ]
> > > > > > > > > > > > }
> > > > > > > > > > > >
> > > > > > > > > > > > The connector which supports Fictional config schema
> 2.0
> > > > > will
> > > > > > > > validate
> > > > > > > > > > > the
> > > > > > > > > > > > access key and secret key.
> > > > > > > > > > > > Whereas a connector which supports config with schema
> > > version
> > > > > > 3.0
> > > > > > > > will
> > > > > > > > > > > only
> > > > > > > > > > > > validate the IAM role.
> > > > > > > > > > > >
> > > > > > > > > > > > This is the alternative which I wanted to suggest.
> Each
> > > > > plugin
> > > > > > will
> > > > > > > > > > > > register the schema versions of connector config
> which it
> > > > > > supports.
> > > > > > > > > > > >
> > > > > > > > > > > > The plugin paths may be optionally different i.e we
> > > don't
> > > > > > have to
> > > > > > > > > > > > mandatorily add a new plugin path to support a new
> schema
> > > > > > version.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Ashwin
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <
> > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. Can you elaborate on the rejected
> alternatives ?
> > > > > Suppose
> > > > > > > > connector
> > > > > > > > > > > > > > config is versioned and has a schema. Then a
> single
> > > > > plugin
> > > > > > > > (whose
> > > > > > > > > > > > > > dependencies have not changed) can handle
> multiple
> > > config
> > > > > > > > versions
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > >
> > > > > > > > > > > > > IIUC (please correct me if I am wrong here), what
> you
> > > > > > highlighted
> > > > > > > > > > > above,
> > > > > > > > > > > > is
> > > > > > > > > > > > > a versioning scheme for a connector config for the
> same
> > > > > > > > connector (and
> > > > > > > > > > > > not
> > > > > > > > > > > > > different versions of a connector plugin). That is
> a
> > > > > somewhat
> > > > > > > > > > > tangential
> > > > > > > > > > > > > problem. While it is definitely a useful feature to
> > > have,
> > > > > > like a
> > > > > > > > log to
> > > > > > > > > > > > > check what changes were made over time to the
> config
> > > which
> > > > > > might
> > > > > > > > make
> > > > > > > > > > > it
> > > > > > > > > > > > > easier to do rollbacks, it is not the focus here.
> Here
> > > by
> > > > > > > > version we
> > > > > > > > > > > mean
> > > > > > > > > > > > > to say what underlying version of the plugin
> should the
> > > > > given
> > > > > > > > > > > > configuration
> > > > > > > > > > > > > of the connector use. Perhaps it is better to
> change
> > > the
> > > > > > name of
> > > > > > > > the
> > > > > > > > > > > > > parameter from connector.version to
> > > > > connector.plugin.version
> > > > > > or
> > > > > > > > > > > > > plugin.version if it was confusing. wdyt?
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. Any plans to support assisted migration e.g
> if a
> > > user
> > > > > > > > invokes
> > > > > > > > > > > "POST
> > > > > > > > > > > > > > connector/config?migrate=latest", the latest
> version
> > > > > > > > __attempts__ to
> > > > > > > > > > > > > > transform the existing config to the newer
> version.
> > > This
> > > > > > would
> > > > > > > > > > > require
> > > > > > > > > > > > > > adding a method like "boolean migrate(Version
> > > > > > fromVersion)" to
> > > > > > > > the
> > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is an enhancement we can think of doing in
> future.
> > > > > > Users can
> > > > > > > > > > > simply
> > > > > > > > > > > > do
> > > > > > > > > > > > > a PUT call with the updated config which has the
> > > updated
> > > > > > version
> > > > > > > > > > > number.
> > > > > > > > > > > > > The assisted mode could be handy as the user does
> not
> > > need
> > > > > to
> > > > > > > > know the
> > > > > > > > > > > > > config but beyond this it does not seem to justify
> its
> > > > > > existence.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards
> > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin
> > > > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is a really useful feature and thanks for
> > > initiating
> > > > > > this
> > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I had the following questions -
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. Can you elaborate on the rejected
> alternatives ?
> > > > > Suppose
> > > > > > > > connector
> > > > > > > > > > > > > > config is versioned and has a schema. Then a
> single
> > > > > plugin
> > > > > > > > (whose
> > > > > > > > > > > > > > dependencies have not changed) can handle
> multiple
> > > config
> > > > > > > > versions
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. Any plans to support assisted migration e.g
> if a
> > > user
> > > > > > > > invokes
> > > > > > > > > > > "POST
> > > > > > > > > > > > > > connector/config?migrate=latest", the latest
> version
> > > > > > > > __attempts__ to
> > > > > > > > > > > > > > transform the existing config to the newer
> version.
> > > This
> > > > > > would
> > > > > > > > > > > require
> > > > > > > > > > > > > > adding a method like "boolean migrate(Version
> > > > > > fromVersion)" to
> > > > > > > > the
> > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <
> > > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'd like to start a discussion thread on
> KIP-891:
> > > > > Running
> > > > > > > > multiple
> > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The KIP aims to add the ability for the connect
> > > runtime
> > > > > > to
> > > > > > > > run
> > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > versions of a connector.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Please take a look and let me know what you
> think.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Hey Chris,
Thanks for your quick follow up.
> But this risk is already present with
> existing error cases, and I don't see anything that justifies changing
> existing behavior with an invalid converter class, or diverging from it in
> the case of invalid converter versions.
The justification is to fail-fast, and prevent REST API users from
receiving errors from bad configs that they didn't write, or maybe
don't even know apply to them.
Up until recently errors in these configurations surfaced as failures
to create the connector, or failures to start, and you made them
fail-fast during validation. I think this change is in the same
spirit, pulling the validation further forward and not letting errors
lie dormant.
And to call back to your original concern about interrupting
connectors that explicitly provide these configurations and don't use
the worker configs: I expect that operators with a majority of these
sorts of clients aren't going to be setting the worker .version
properties, because it would have no effect on the majority of their
connectors. They would be able to rely on backwards-compatibility and
continue to ignore the class properties.
> CLI
> and programmatic UI developers will want to develop their own tooling
> layers.
This is a very compelling argument. Snehashis do you want to figure
out a REST API design for this use-case?
> Regarding the GET /connector-plugins/<plugin>/config endpoint, I was
> thinking about the response for non-connector plugins, e.g.,
> GET /connector-plugins/RegexRouter/config. Would a "version" property
> appear with recommended values?
I intended for the ".version" property to be like other framework
configs (".class", ".type", ".predicate", ".negate") where they are
inside the plugin namespace, but not part of the plugin config itself.
Perhaps we can deviate from those configs because it would aid in
discovering other valid `GET
/connector-plugins/<plugin>/config?version=` calls, without calling
`GET /connector-plugins?connectorsOnly=false`.
I don't really feel strongly either way.
> if a user changes the
> connector version in, e.g., a dropdown menu, then the UI either has to
> re-fetch the ConfigDef for the new version, or risk operating on stale
> information
This is a really interesting situation, thanks for finding that! This
is already a footgun with transformations and predicates; Once you
fill out a transformation class (which includes a recommender) you
will then see all of the transformations configurations.
I think this means that UI developers should have already developed
the infrastructure for handling dynamic recommenders, or else they've
had this bug since KIP-66 in 2017. It may require some manual
attention to roll-out support for the ".version" properties
specifically, but I don't think that should prevent us from providing
this information in a natural place.
> but will it be possible to configure
> a connector to use two instances of the same transform or predicate, but
> with different versions for each?
Yes, and this is included in the example in the KIP,
`transforms.flatten-latest` and `transforms.flatten-old`. This will
work in the natural way, with the two instances having a different
version if configured for that. If a user wants to pin the same
version of a plugin for every instance, they will need to provide the
version config multiple times and keep them in-sync themselves.
> I would have expected the error
> to only be attributed to the version property, and for the class property
> to be reported as valid.
I considered this, and then went back and changed it. To quote someone
else, this is to catch "misspelling cat as dog". For example a user
types DogConverter and meant to type CatConverter, and then types a
version which is valid for CatConverter, conceptually the error is in
the class name and not the version.
That's a very contrived scenario, but I think similar arguments are
used for attributing validation errors to endpoints/urls/hostnames
when the associated credentials are unable to log into the remote
system. Did the user provide the correct testing credentials, but
accidentally typed the production endpoint? Even if the production
endpoint looks valid (it's a real hostname that is reachable) the
conceptual error is still in the hostname and should have the error
attributed to it to draw the user's attention.
If that's not convincing, I think the alternative of only attributing
version errors to the version property is also acceptable.
Thanks,
Greg
On Wed, May 15, 2024 at 8:06 AM Chris Egerton <ch...@aiven.io.invalid> wrote:
>
> Hi Greg,
>
> Thanks for your responses! Continuations of existing discussions:
>
> Regarding crashing the worker on startup--yes, there is also a risk to
> allowing it to join the cluster. But this risk is already present with
> existing error cases, and I don't see anything that justifies changing
> existing behavior with an invalid converter class, or diverging from it in
> the case of invalid converter versions. I think we should keep this simple
> and not do anything different for worker startup than we do already.
>
> As far as REST API vs. metrics go--I think you're right that the original
> version metrics were added as a "monitoring" detail. However, this was back
> when plugin versions were managed solely by cluster administrators. With
> this KIP, connector users will be able to manage plugin versions, and CLI
> and programmatic UI developers will want to develop their own tooling
> layers. I think focusing on the REST API as the primary interface for this
> KIP would be best for these users.
>
> (All that said, I don't object to the metrics that are proposed in the KIP;
> I just think they make more sense in addition to new REST API
> functionality, as opposed to instead of it.)
>
> Regarding the GET /connector-plugins/<plugin>/config endpoint, I was
> thinking about the response for non-connector plugins, e.g.,
> GET /connector-plugins/RegexRouter/config. Would a "version" property
> appear with recommended values?
>
>
> And new thoughts:
>
> 1) Regarding the recommended values for "connector.version", this might be
> confusing since there could be differences between the ConfigDefs for the
> latest version of the connector and prior versions. It also makes the flow
> a bit awkward for programmatic UI developers: if a user changes the
> connector version in, e.g., a dropdown menu, then the UI either has to
> re-fetch the ConfigDef for the new version, or risk operating on stale
> information. I'm starting to doubt that exposing the range of available
> versions via recommended values is the best way to proceed, instead of a
> more explicit approach like GET /connector-plugins/<plugin>/versions, or
> the "Adding new REST API endpoints" rejected alternative.
>
> 2) I know that this is a bit heinous, but will it be possible to configure
> a connector to use two instances of the same transform or predicate, but
> with different versions for each? (I don't think this is worth significant
> design/implementation effort, so if it would inflate either of those,
> please don't feel obligated to support it.)
>
> 3) Just out of curiosity, why double-ding during config validation if the
> version for a plugin class can't be found? I would have expected the error
> to only be attributed to the version property, and for the class property
> to be reported as valid.
>
> Cheers,
>
> Chris
>
> On Tue, May 14, 2024 at 1:11 PM Greg Harris <gr...@aiven.io.invalid>
> wrote:
>
> > Hey Chris,
> >
> > Thanks for your comments. I'm glad you like the motivations, Snehashis
> > wrote that part!
> >
> > > the configuration syntax for the most basic use case of
> > > specifying a single desired version is pretty counterintuitive.
> >
> > I agree, and the "soft" requirement scheme is something I wasn't
> > explicitly looking for, but would be inherited from the library. I'm
> > fine with eliminating the soft requirement semantics, and having
> > "1.0.0" behave the same as "[1.0.0]". I'm less inclined to include
> > "range" in the property name, or have two properties.
> >
> > > Failing startup is drastic and has the potential to disrupt the
> > > availability of connectors that would otherwise be able to run healthily
> > > because they were explicitly configured to use valid converters instead
> > of
> > > the worker defaults.
> >
> > I think this argument cuts both ways. If someone reconfigures a worker
> > and adds an invalid ".version" string to the worker (or changes the
> > plugin.path to make it invalid), it would be permitted to enter the
> > group, and accept work assignments. If those work assignments used
> > these configurations, a set of tasks could transition to FAILED and
> > not be able to recover, because they would be restarted again on the
> > same worker.
> >
> > > Why are metrics utilized to report information about plugin versions
> > > utilized by connectors at runtime instead of publishing this info in the
> > > REST API
> >
> > I was following the existing pattern for exposing runtime versions for
> > connectors, and it did seem like a "monitoring" feature. If that
> > approach is flawed and should be phased out, I think it would be a
> > good idea to reconsider the REST API rejected alternative.
> > We would need some additional design work to spec out the REST API
> > interface, as I don't have anything in mind currently.
> >
> > > I'm unclear on whether or not it'll be possible to see this information
> > via the
> > > GET /connector-plugins/<plugin>/config endpoint
> >
> > There is room in the API to add recommenders for "key.converter",
> > "value.converter", and "header.converter", but not for transforms and
> > predicates, as they include aliases that depend on an actual
> > configuration. We could explicitly say we're going to do that, or do
> > whatever is convenient during the implementation phase, or leave it
> > open to be improved later.
> > There will not be any recommenders for ".version" properties in the
> > `/config` endpoint, because those recommenders are dynamic and depend
> > on an actual configuration.
> >
> > 5) There are two relevant lines in the KIP: "If a .version property
> > contains a hard requirement, select the latest installed version which
> > satisfies the requirement." and "This configuration is re-evaluated
> > each time the connector or task are assigned to a new worker". I would
> > call this "eager" upgrade behavior, rather than a "sticky" or "lazy"
> > upgrade behavior.
> >
> > 6) Updated!
> >
> > Thanks,
> > Greg
> >
> > On Tue, May 14, 2024 at 9:14 AM Chris Egerton <ch...@aiven.io.invalid>
> > wrote:
> > >
> > > Hi all,
> > >
> > > Thanks Greg for updating the KIP, and thanks Snehashis for starting the
> > > work on this originally.
> > >
> > > The motivation section makes a pretty convincing case for this kind of
> > > feature. My thoughts are mostly about specific details:
> > >
> > > 1) I like the support for version ranges (the example demonstrating how
> > to
> > > avoid KAFKA-10574 with the header converter was particularly
> > entertaining),
> > > but the configuration syntax for the most basic use case of specifying a
> > > single desired version is pretty counterintuitive. People may get bitten
> > or
> > > at least frustrated if they put connector.version=3.8.0 in a config but
> > > then version 3.7.5 ends up running. I'd like it if we could either
> > > intentionally deviate from Maven ranges when a bare version is present,
> > or
> > > separate things out into two properties: foo.version would be the single
> > > accepted version for the foo plugin, and foo.version.range would use
> > Maven
> > > range syntax. Open to other options too, just providing a couple to get
> > the
> > > ball rolling.
> > >
> > > 2) Although the current behavior for a worker with an invalid
> > > key/value/header converter class specified in its config file is a little
> > > strange (I was surprised to learn that it wouldn't fail on startup), I
> > > don't see a good reason to deviate from this when an invalid version is
> > > specified. Failing startup is drastic and has the potential to disrupt
> > the
> > > availability of connectors that would otherwise be able to run healthily
> > > because they were explicitly configured to use valid converters instead
> > of
> > > the worker defaults.
> > >
> > > 3) Why are metrics utilized to report information about plugin versions
> > > utilized by connectors at runtime instead of publishing this info in the
> > > REST API? I saw that this was mentioned as a rejected alternative, but I
> > > didn't get a sense of why. It seems like the REST API would be easier to
> > > access and more intuitive for most users than new metrics.
> > >
> > > 4) In the "Validation" section it's stated that "Users can use these
> > > recommenders to discover the valid plugin classes and versions, without
> > > requiring an earlier call to GET
> > /connector-plugins?connectorsOnly=false."
> > > I really like the creativity and simplicity of reusing the recommender
> > > mechanism to expose available versions for plugins via the REST API. I'm
> > > unclear on whether or not it'll be possible to see this information via
> > the
> > > GET /connector-plugins/<plugin>/config endpoint, though. It'd be great if
> > > this were supported, since we learned in KIP-769 [1] that people really
> > > want to be able to see configuration options for connectors and their
> > > plugins via some kind of GET endpoint without having to provide a
> > complete
> > > connector config for validation.
> > >
> > > 5) In the Maven version range docs, it's stated that "Maven picks the
> > > highest version of each project that satisfies all the hard requirements
> > of
> > > the dependencies on that project." I'm guessing this behavior will be
> > > retained for Connect; i.e., the highest-possible version of each plugin
> > > that satisfies the user-specified version constraints will be run? (An
> > > alternative approach could be to have some kind of "sticky" logic that
> > only
> > > restarts connectors/tasks when their currently-used version becomes
> > > incompatible with the configured constraints.)
> > >
> > > 6) (Nit) It'd be nice to add a link to the TestPlugins class or somewhere
> > > in its neighborhood to the testing plan; unfamiliar readers probably
> > won't
> > > get much out of what's there right now.
> > >
> > > [1] -
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Mon, May 13, 2024 at 2:01 PM Snehashis <sn...@gmail.com>
> > wrote:
> > >
> > > > Hi Greg,
> > > >
> > > > That is much appreciated. No complaints on the additional scope, I will
> > > > make some time out to work on this once we have approval.
> > > >
> > > > Thanks
> > > > Snehashis
> > > >
> > > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> > <gr...@aiven.io.invalid>
> > > > wrote:
> > > >
> > > > > Hey Snehashis,
> > > > >
> > > > > I'm glad to hear you're still interested in this KIP!
> > > > > I'm happy to let you drive this, and I apologize for increasing the
> > > > > scope of work so drastically. To make up for that, I'll volunteer to
> > > > > be the primary PR reviewer to help get this done quickly once the KIP
> > > > > is approved.
> > > > >
> > > > > Thanks,
> > > > > Greg
> > > > >
> > > > >
> > > > > On Fri, May 10, 2024 at 3:51 AM Snehashis <sn...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi Greg,
> > > > > >
> > > > > > Thanks for the follow up to my original KIP, I am in favour of the
> > > > > > additions made to expand its scope, the addition of range versions
> > > > > > specifically make a lot of sense.
> > > > > >
> > > > > > Apologies if I have not publicly worked on this KIP for a long
> > time.
> > > > The
> > > > > > original work was done when the move to service loading was in
> > > > discussion
> > > > > > and I wanted to loop back to this only after that work was
> > completed.
> > > > > Post
> > > > > > its conclusion, I have not been able to take this up due to other
> > > > > > priorities. If it's okay with you, I would still like to get this
> > > > > > implemented myself, including the additional scope.
> > > > > >
> > > > > > Thanks and regards
> > > > > > Snehashis
> > > > > >
> > > > > > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> > > > > <gr...@aiven.io.invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'd like to reboot the discussion on KIP-891:
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > >
> > > > > > > I've made some changes, most notably:
> > > > > > >
> > > > > > > 1. Specifying versions for all plugins in Connector configs
> > > > > > > (converters, header converters, transforms, and predicates) not
> > just
> > > > > > > connectors & tasks
> > > > > > > 2. Specifying a range of versions instead of an exact match
> > > > > > > 3. New metrics to observe what versions are in-use
> > > > > > >
> > > > > > > Thanks to Snehashis for the original KIP idea!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Greg
> > > > > > >
> > > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <
> > greg.harris@aiven.io>
> > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Snehashis,
> > > > > > > >
> > > > > > > > Thank you for the KIP! This is something I've wanted for a long
> > > > time.
> > > > > > > >
> > > > > > > > I know the discussion has gone cold, are you still interested
> > in
> > > > > > > > pursuing this feature? I'll make time to review the KIP if you
> > are
> > > > > > > > still accepting comments.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Greg
> > > > > > > >
> > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis <
> > > > snehashisp1995@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Thanks for the points Sagar.
> > > > > > > > >
> > > > > > > > > > 1) Should we update the GET /connectors endpoint to
> > include the
> > > > > > > version of
> > > > > > > > > > the plugin that is running? It could be useful to figure
> > out
> > > > the
> > > > > > > version
> > > > > > > > > of
> > > > > > > > > > the plugin or I am assuming it gets returned by the
> > expand=info
> > > > > call?
> > > > > > > > >
> > > > > > > > > I think this is good to have and possible future
> > enhancement. The
> > > > > > > version
> > > > > > > > > info will be present in the config of the connector if the
> > user
> > > > has
> > > > > > > > > specified the version. Otherwise it is the latest version
> > which
> > > > the
> > > > > > > user
> > > > > > > > > can find out from the connector-plugin endpoint. The
> > information
> > > > > can be
> > > > > > > > > introduced to the response of the GET /connectors endpoint
> > > > itself,
> > > > > > > however
> > > > > > > > > the most ideal way of doing this would be to get the
> > currently
> > > > > running
> > > > > > > > > instance of the connector and get the version directly from
> > > > there.
> > > > > > > This is
> > > > > > > > > slightly tricky as the connector could be running in a
> > different
> > > > > node.
> > > > > > > > > One way to do this would be to persist the version
> > information in
> > > > > the
> > > > > > > > > status backing store during instantiation of the connector.
> > It
> > > > > requires
> > > > > > > > > some more thought and since the version is part of the
> > configs if
> > > > > > > provided
> > > > > > > > > and evident otherwise, I have not included it in this KIP.
> > > > > > > > >
> > > > > > > > > > 2) I am not aware of this and hence asking, can 2
> > connectors
> > > > with
> > > > > > > > > different
> > > > > > > > > > versions have the same name? Does the plugin isolation
> > allow
> > > > > this?
> > > > > > > This
> > > > > > > > > > could have a bearing when using the lifecycle endpoints for
> > > > > > > connectors
> > > > > > > > > like
> > > > > > > > > > DELETE etc.
> > > > > > > > >
> > > > > > > > > All connectors in a cluster need to have uniquire connector
> > names
> > > > > > > > > regardless of what version of the plugin the connector is
> > running
> > > > > > > > > underneath. This is something enforced by the connect runtime
> > > > > itself.
> > > > > > > All
> > > > > > > > > connect CRUD operations are keyed on the connector name so
> > there
> > > > > will
> > > > > > > not
> > > > > > > > > be an issue.
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > Snehashis
> > > > > > > > >
> > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <
> > sagarmeansocean@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hey Snehashsih,
> > > > > > > > > >
> > > > > > > > > > Thanks for the KIP. It looks like a very useful feature.
> > Couple
> > > > > of
> > > > > > > > > > small-ish points, let me know what you think:
> > > > > > > > > >
> > > > > > > > > > 1) Should we update the GET /connectors endpoint to
> > include the
> > > > > > > version of
> > > > > > > > > > the plugin that is running? It could be useful to figure
> > out
> > > > the
> > > > > > > version of
> > > > > > > > > > the plugin or I am assuming it gets returned by the
> > expand=info
> > > > > call?
> > > > > > > > > > 2) I am not aware of this and hence asking, can 2
> > connectors
> > > > with
> > > > > > > different
> > > > > > > > > > versions have the same name? Does the plugin isolation
> > allow
> > > > > this?
> > > > > > > This
> > > > > > > > > > could have a bearing when using the lifecycle endpoints for
> > > > > > > connectors like
> > > > > > > > > > DELETE etc.
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > > Sagar.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin
> > > > > <apankaj@confluent.io.invalid
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > >
> > > > > > > > > > > > IIUC (please correct me if I am wrong here), what you
> > > > > highlighted
> > > > > > > > > > above,
> > > > > > > > > > > is
> > > > > > > > > > > a versioning scheme for a connector config for the same
> > > > > connector
> > > > > > > (and
> > > > > > > > > > not
> > > > > > > > > > > different versions of a connector plugin).
> > > > > > > > > > >
> > > > > > > > > > > Sorry for not being more precise in my wording - I meant
> > > > > > > registering
> > > > > > > > > > > versions of schema for connector config.
> > > > > > > > > > >
> > > > > > > > > > > Let's take the example of a fictional connector which
> > uses a
> > > > > > > fictional
> > > > > > > > > > AWS
> > > > > > > > > > > service.
> > > > > > > > > > >
> > > > > > > > > > > Fictional Connector Config schema version:2.0
> > > > > > > > > > > ---
> > > > > > > > > > > {
> > > > > > > > > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > "type": "object",
> > > > > > > > > > > "properties": {
> > > > > > > > > > > "name": {
> > > > > > > > > > > "type": "string"
> > > > > > > > > > > },
> > > > > > > > > > > "schema_version": {
> > > > > > > > > > > "type": "string"
> > > > > > > > > > > },
> > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > "type": "string"
> > > > > > > > > > > },
> > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > "type": "string"
> > > > > > > > > > > }
> > > > > > > > > > > },
> > > > > > > > > > > "required": [
> > > > > > > > > > > "name",
> > > > > > > > > > > "schema_version",
> > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > ]
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > Fictional Connector config schema version:3.0
> > > > > > > > > > > ---
> > > > > > > > > > > {
> > > > > > > > > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > > > > > > > > "type": "object",
> > > > > > > > > > > "properties": {
> > > > > > > > > > > "name": {
> > > > > > > > > > > "type": "string"
> > > > > > > > > > > },
> > > > > > > > > > > "schema_version": {
> > > > > > > > > > > "type": "string"
> > > > > > > > > > > },
> > > > > > > > > > > "iam_role": {
> > > > > > > > > > > "type": "string"
> > > > > > > > > > > }
> > > > > > > > > > > },
> > > > > > > > > > > "required": [
> > > > > > > > > > > "name",
> > > > > > > > > > > "schema_version",
> > > > > > > > > > > "iam_role"
> > > > > > > > > > > ]
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > The connector which supports Fictional config schema 2.0
> > > > will
> > > > > > > validate
> > > > > > > > > > the
> > > > > > > > > > > access key and secret key.
> > > > > > > > > > > Whereas a connector which supports config with schema
> > version
> > > > > 3.0
> > > > > > > will
> > > > > > > > > > only
> > > > > > > > > > > validate the IAM role.
> > > > > > > > > > >
> > > > > > > > > > > This is the alternative which I wanted to suggest. Each
> > > > plugin
> > > > > will
> > > > > > > > > > > register the schema versions of connector config which it
> > > > > supports.
> > > > > > > > > > >
> > > > > > > > > > > The plugin paths may be optionally different i.e we
> > don't
> > > > > have to
> > > > > > > > > > > mandatorily add a new plugin path to support a new schema
> > > > > version.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Ashwin
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <
> > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > > > > >
> > > > > > > > > > > > > 1. Can you elaborate on the rejected alternatives ?
> > > > Suppose
> > > > > > > connector
> > > > > > > > > > > > > config is versioned and has a schema. Then a single
> > > > plugin
> > > > > > > (whose
> > > > > > > > > > > > > dependencies have not changed) can handle multiple
> > config
> > > > > > > versions
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > same connector class.
> > > > > > > > > > > >
> > > > > > > > > > > > IIUC (please correct me if I am wrong here), what you
> > > > > highlighted
> > > > > > > > > > above,
> > > > > > > > > > > is
> > > > > > > > > > > > a versioning scheme for a connector config for the same
> > > > > > > connector (and
> > > > > > > > > > > not
> > > > > > > > > > > > different versions of a connector plugin). That is a
> > > > somewhat
> > > > > > > > > > tangential
> > > > > > > > > > > > problem. While it is definitely a useful feature to
> > have,
> > > > > like a
> > > > > > > log to
> > > > > > > > > > > > check what changes were made over time to the config
> > which
> > > > > might
> > > > > > > make
> > > > > > > > > > it
> > > > > > > > > > > > easier to do rollbacks, it is not the focus here. Here
> > by
> > > > > > > version we
> > > > > > > > > > mean
> > > > > > > > > > > > to say what underlying version of the plugin should the
> > > > given
> > > > > > > > > > > configuration
> > > > > > > > > > > > of the connector use. Perhaps it is better to change
> > the
> > > > > name of
> > > > > > > the
> > > > > > > > > > > > parameter from connector.version to
> > > > connector.plugin.version
> > > > > or
> > > > > > > > > > > > plugin.version if it was confusing. wdyt?
> > > > > > > > > > > >
> > > > > > > > > > > > > 2. Any plans to support assisted migration e.g if a
> > user
> > > > > > > invokes
> > > > > > > > > > "POST
> > > > > > > > > > > > > connector/config?migrate=latest", the latest version
> > > > > > > __attempts__ to
> > > > > > > > > > > > > transform the existing config to the newer version.
> > This
> > > > > would
> > > > > > > > > > require
> > > > > > > > > > > > > adding a method like "boolean migrate(Version
> > > > > fromVersion)" to
> > > > > > > the
> > > > > > > > > > > > > connector interface.
> > > > > > > > > > > >
> > > > > > > > > > > > This is an enhancement we can think of doing in future.
> > > > > Users can
> > > > > > > > > > simply
> > > > > > > > > > > do
> > > > > > > > > > > > a PUT call with the updated config which has the
> > updated
> > > > > version
> > > > > > > > > > number.
> > > > > > > > > > > > The assisted mode could be handy as the user does not
> > need
> > > > to
> > > > > > > know the
> > > > > > > > > > > > config but beyond this it does not seem to justify its
> > > > > existence.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards
> > > > > > > > > > > > Snehashis
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin
> > > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is a really useful feature and thanks for
> > initiating
> > > > > this
> > > > > > > > > > > > discussion.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I had the following questions -
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Can you elaborate on the rejected alternatives ?
> > > > Suppose
> > > > > > > connector
> > > > > > > > > > > > > config is versioned and has a schema. Then a single
> > > > plugin
> > > > > > > (whose
> > > > > > > > > > > > > dependencies have not changed) can handle multiple
> > config
> > > > > > > versions
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. Any plans to support assisted migration e.g if a
> > user
> > > > > > > invokes
> > > > > > > > > > "POST
> > > > > > > > > > > > > connector/config?migrate=latest", the latest version
> > > > > > > __attempts__ to
> > > > > > > > > > > > > transform the existing config to the newer version.
> > This
> > > > > would
> > > > > > > > > > require
> > > > > > > > > > > > > adding a method like "boolean migrate(Version
> > > > > fromVersion)" to
> > > > > > > the
> > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <
> > > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'd like to start a discussion thread on KIP-891:
> > > > Running
> > > > > > > multiple
> > > > > > > > > > > > > versions
> > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The KIP aims to add the ability for the connect
> > runtime
> > > > > to
> > > > > > > run
> > > > > > > > > > > multiple
> > > > > > > > > > > > > > versions of a connector.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please take a look and let me know what you think.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > > >
> > > >
> >
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Chris Egerton <ch...@aiven.io.INVALID>.
Hi Greg,
Thanks for your responses! Continuations of existing discussions:
Regarding crashing the worker on startup--yes, there is also a risk to
allowing it to join the cluster. But this risk is already present with
existing error cases, and I don't see anything that justifies changing
existing behavior with an invalid converter class, or diverging from it in
the case of invalid converter versions. I think we should keep this simple
and not do anything different for worker startup than we do already.
As far as REST API vs. metrics go--I think you're right that the original
version metrics were added as a "monitoring" detail. However, this was back
when plugin versions were managed solely by cluster administrators. With
this KIP, connector users will be able to manage plugin versions, and CLI
and programmatic UI developers will want to develop their own tooling
layers. I think focusing on the REST API as the primary interface for this
KIP would be best for these users.
(All that said, I don't object to the metrics that are proposed in the KIP;
I just think they make more sense in addition to new REST API
functionality, as opposed to instead of it.)
Regarding the GET /connector-plugins/<plugin>/config endpoint, I was
thinking about the response for non-connector plugins, e.g.,
GET /connector-plugins/RegexRouter/config. Would a "version" property
appear with recommended values?
And new thoughts:
1) Regarding the recommended values for "connector.version", this might be
confusing since there could be differences between the ConfigDefs for the
latest version of the connector and prior versions. It also makes the flow
a bit awkward for programmatic UI developers: if a user changes the
connector version in, e.g., a dropdown menu, then the UI either has to
re-fetch the ConfigDef for the new version, or risk operating on stale
information. I'm starting to doubt that exposing the range of available
versions via recommended values is the best way to proceed, instead of a
more explicit approach like GET /connector-plugins/<plugin>/versions, or
the "Adding new REST API endpoints" rejected alternative.
2) I know that this is a bit heinous, but will it be possible to configure
a connector to use two instances of the same transform or predicate, but
with different versions for each? (I don't think this is worth significant
design/implementation effort, so if it would inflate either of those,
please don't feel obligated to support it.)
3) Just out of curiosity, why double-ding during config validation if the
version for a plugin class can't be found? I would have expected the error
to only be attributed to the version property, and for the class property
to be reported as valid.
Cheers,
Chris
On Tue, May 14, 2024 at 1:11 PM Greg Harris <gr...@aiven.io.invalid>
wrote:
> Hey Chris,
>
> Thanks for your comments. I'm glad you like the motivations, Snehashis
> wrote that part!
>
> > the configuration syntax for the most basic use case of
> > specifying a single desired version is pretty counterintuitive.
>
> I agree, and the "soft" requirement scheme is something I wasn't
> explicitly looking for, but would be inherited from the library. I'm
> fine with eliminating the soft requirement semantics, and having
> "1.0.0" behave the same as "[1.0.0]". I'm less inclined to include
> "range" in the property name, or have two properties.
>
> > Failing startup is drastic and has the potential to disrupt the
> > availability of connectors that would otherwise be able to run healthily
> > because they were explicitly configured to use valid converters instead
> of
> > the worker defaults.
>
> I think this argument cuts both ways. If someone reconfigures a worker
> and adds an invalid ".version" string to the worker (or changes the
> plugin.path to make it invalid), it would be permitted to enter the
> group, and accept work assignments. If those work assignments used
> these configurations, a set of tasks could transition to FAILED and
> not be able to recover, because they would be restarted again on the
> same worker.
>
> > Why are metrics utilized to report information about plugin versions
> > utilized by connectors at runtime instead of publishing this info in the
> > REST API
>
> I was following the existing pattern for exposing runtime versions for
> connectors, and it did seem like a "monitoring" feature. If that
> approach is flawed and should be phased out, I think it would be a
> good idea to reconsider the REST API rejected alternative.
> We would need some additional design work to spec out the REST API
> interface, as I don't have anything in mind currently.
>
> > I'm unclear on whether or not it'll be possible to see this information
> via the
> > GET /connector-plugins/<plugin>/config endpoint
>
> There is room in the API to add recommenders for "key.converter",
> "value.converter", and "header.converter", but not for transforms and
> predicates, as they include aliases that depend on an actual
> configuration. We could explicitly say we're going to do that, or do
> whatever is convenient during the implementation phase, or leave it
> open to be improved later.
> There will not be any recommenders for ".version" properties in the
> `/config` endpoint, because those recommenders are dynamic and depend
> on an actual configuration.
>
> 5) There are two relevant lines in the KIP: "If a .version property
> contains a hard requirement, select the latest installed version which
> satisfies the requirement." and "This configuration is re-evaluated
> each time the connector or task are assigned to a new worker". I would
> call this "eager" upgrade behavior, rather than a "sticky" or "lazy"
> upgrade behavior.
>
> 6) Updated!
>
> Thanks,
> Greg
>
> On Tue, May 14, 2024 at 9:14 AM Chris Egerton <ch...@aiven.io.invalid>
> wrote:
> >
> > Hi all,
> >
> > Thanks Greg for updating the KIP, and thanks Snehashis for starting the
> > work on this originally.
> >
> > The motivation section makes a pretty convincing case for this kind of
> > feature. My thoughts are mostly about specific details:
> >
> > 1) I like the support for version ranges (the example demonstrating how
> to
> > avoid KAFKA-10574 with the header converter was particularly
> entertaining),
> > but the configuration syntax for the most basic use case of specifying a
> > single desired version is pretty counterintuitive. People may get bitten
> or
> > at least frustrated if they put connector.version=3.8.0 in a config but
> > then version 3.7.5 ends up running. I'd like it if we could either
> > intentionally deviate from Maven ranges when a bare version is present,
> or
> > separate things out into two properties: foo.version would be the single
> > accepted version for the foo plugin, and foo.version.range would use
> Maven
> > range syntax. Open to other options too, just providing a couple to get
> the
> > ball rolling.
> >
> > 2) Although the current behavior for a worker with an invalid
> > key/value/header converter class specified in its config file is a little
> > strange (I was surprised to learn that it wouldn't fail on startup), I
> > don't see a good reason to deviate from this when an invalid version is
> > specified. Failing startup is drastic and has the potential to disrupt
> the
> > availability of connectors that would otherwise be able to run healthily
> > because they were explicitly configured to use valid converters instead
> of
> > the worker defaults.
> >
> > 3) Why are metrics utilized to report information about plugin versions
> > utilized by connectors at runtime instead of publishing this info in the
> > REST API? I saw that this was mentioned as a rejected alternative, but I
> > didn't get a sense of why. It seems like the REST API would be easier to
> > access and more intuitive for most users than new metrics.
> >
> > 4) In the "Validation" section it's stated that "Users can use these
> > recommenders to discover the valid plugin classes and versions, without
> > requiring an earlier call to GET
> /connector-plugins?connectorsOnly=false."
> > I really like the creativity and simplicity of reusing the recommender
> > mechanism to expose available versions for plugins via the REST API. I'm
> > unclear on whether or not it'll be possible to see this information via
> the
> > GET /connector-plugins/<plugin>/config endpoint, though. It'd be great if
> > this were supported, since we learned in KIP-769 [1] that people really
> > want to be able to see configuration options for connectors and their
> > plugins via some kind of GET endpoint without having to provide a
> complete
> > connector config for validation.
> >
> > 5) In the Maven version range docs, it's stated that "Maven picks the
> > highest version of each project that satisfies all the hard requirements
> of
> > the dependencies on that project." I'm guessing this behavior will be
> > retained for Connect; i.e., the highest-possible version of each plugin
> > that satisfies the user-specified version constraints will be run? (An
> > alternative approach could be to have some kind of "sticky" logic that
> only
> > restarts connectors/tasks when their currently-used version becomes
> > incompatible with the configured constraints.)
> >
> > 6) (Nit) It'd be nice to add a link to the TestPlugins class or somewhere
> > in its neighborhood to the testing plan; unfamiliar readers probably
> won't
> > get much out of what's there right now.
> >
> > [1] -
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> >
> > Cheers,
> >
> > Chris
> >
> > On Mon, May 13, 2024 at 2:01 PM Snehashis <sn...@gmail.com>
> wrote:
> >
> > > Hi Greg,
> > >
> > > That is much appreciated. No complaints on the additional scope, I will
> > > make some time out to work on this once we have approval.
> > >
> > > Thanks
> > > Snehashis
> > >
> > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> <gr...@aiven.io.invalid>
> > > wrote:
> > >
> > > > Hey Snehashis,
> > > >
> > > > I'm glad to hear you're still interested in this KIP!
> > > > I'm happy to let you drive this, and I apologize for increasing the
> > > > scope of work so drastically. To make up for that, I'll volunteer to
> > > > be the primary PR reviewer to help get this done quickly once the KIP
> > > > is approved.
> > > >
> > > > Thanks,
> > > > Greg
> > > >
> > > >
> > > > On Fri, May 10, 2024 at 3:51 AM Snehashis <sn...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hi Greg,
> > > > >
> > > > > Thanks for the follow up to my original KIP, I am in favour of the
> > > > > additions made to expand its scope, the addition of range versions
> > > > > specifically make a lot of sense.
> > > > >
> > > > > Apologies if I have not publicly worked on this KIP for a long
> time.
> > > The
> > > > > original work was done when the move to service loading was in
> > > discussion
> > > > > and I wanted to loop back to this only after that work was
> completed.
> > > > Post
> > > > > its conclusion, I have not been able to take this up due to other
> > > > > priorities. If it's okay with you, I would still like to get this
> > > > > implemented myself, including the additional scope.
> > > > >
> > > > > Thanks and regards
> > > > > Snehashis
> > > > >
> > > > > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> > > > <gr...@aiven.io.invalid>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to reboot the discussion on KIP-891:
> > > > > >
> > > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > >
> > > > > > I've made some changes, most notably:
> > > > > >
> > > > > > 1. Specifying versions for all plugins in Connector configs
> > > > > > (converters, header converters, transforms, and predicates) not
> just
> > > > > > connectors & tasks
> > > > > > 2. Specifying a range of versions instead of an exact match
> > > > > > 3. New metrics to observe what versions are in-use
> > > > > >
> > > > > > Thanks to Snehashis for the original KIP idea!
> > > > > >
> > > > > > Thanks,
> > > > > > Greg
> > > > > >
> > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <
> greg.harris@aiven.io>
> > > > wrote:
> > > > > > >
> > > > > > > Hi Snehashis,
> > > > > > >
> > > > > > > Thank you for the KIP! This is something I've wanted for a long
> > > time.
> > > > > > >
> > > > > > > I know the discussion has gone cold, are you still interested
> in
> > > > > > > pursuing this feature? I'll make time to review the KIP if you
> are
> > > > > > > still accepting comments.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Greg
> > > > > > >
> > > > > > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis <
> > > snehashisp1995@gmail.com
> > > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > Thanks for the points Sagar.
> > > > > > > >
> > > > > > > > > 1) Should we update the GET /connectors endpoint to
> include the
> > > > > > version of
> > > > > > > > > the plugin that is running? It could be useful to figure
> out
> > > the
> > > > > > version
> > > > > > > > of
> > > > > > > > > the plugin or I am assuming it gets returned by the
> expand=info
> > > > call?
> > > > > > > >
> > > > > > > > I think this is good to have and possible future
> enhancement. The
> > > > > > version
> > > > > > > > info will be present in the config of the connector if the
> user
> > > has
> > > > > > > > specified the version. Otherwise it is the latest version
> which
> > > the
> > > > > > user
> > > > > > > > can find out from the connector-plugin endpoint. The
> information
> > > > can be
> > > > > > > > introduced to the response of the GET /connectors endpoint
> > > itself,
> > > > > > however
> > > > > > > > the most ideal way of doing this would be to get the
> currently
> > > > running
> > > > > > > > instance of the connector and get the version directly from
> > > there.
> > > > > > This is
> > > > > > > > slightly tricky as the connector could be running in a
> different
> > > > node.
> > > > > > > > One way to do this would be to persist the version
> information in
> > > > the
> > > > > > > > status backing store during instantiation of the connector.
> It
> > > > requires
> > > > > > > > some more thought and since the version is part of the
> configs if
> > > > > > provided
> > > > > > > > and evident otherwise, I have not included it in this KIP.
> > > > > > > >
> > > > > > > > > 2) I am not aware of this and hence asking, can 2
> connectors
> > > with
> > > > > > > > different
> > > > > > > > > versions have the same name? Does the plugin isolation
> allow
> > > > this?
> > > > > > This
> > > > > > > > > could have a bearing when using the lifecycle endpoints for
> > > > > > connectors
> > > > > > > > like
> > > > > > > > > DELETE etc.
> > > > > > > >
> > > > > > > > All connectors in a cluster need to have uniquire connector
> names
> > > > > > > > regardless of what version of the plugin the connector is
> running
> > > > > > > > underneath. This is something enforced by the connect runtime
> > > > itself.
> > > > > > All
> > > > > > > > connect CRUD operations are keyed on the connector name so
> there
> > > > will
> > > > > > not
> > > > > > > > be an issue.
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Snehashis
> > > > > > > >
> > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <
> sagarmeansocean@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Snehashsih,
> > > > > > > > >
> > > > > > > > > Thanks for the KIP. It looks like a very useful feature.
> Couple
> > > > of
> > > > > > > > > small-ish points, let me know what you think:
> > > > > > > > >
> > > > > > > > > 1) Should we update the GET /connectors endpoint to
> include the
> > > > > > version of
> > > > > > > > > the plugin that is running? It could be useful to figure
> out
> > > the
> > > > > > version of
> > > > > > > > > the plugin or I am assuming it gets returned by the
> expand=info
> > > > call?
> > > > > > > > > 2) I am not aware of this and hence asking, can 2
> connectors
> > > with
> > > > > > different
> > > > > > > > > versions have the same name? Does the plugin isolation
> allow
> > > > this?
> > > > > > This
> > > > > > > > > could have a bearing when using the lifecycle endpoints for
> > > > > > connectors like
> > > > > > > > > DELETE etc.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > > Sagar.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin
> > > > <apankaj@confluent.io.invalid
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Snehasis,
> > > > > > > > > >
> > > > > > > > > > > IIUC (please correct me if I am wrong here), what you
> > > > highlighted
> > > > > > > > > above,
> > > > > > > > > > is
> > > > > > > > > > a versioning scheme for a connector config for the same
> > > > connector
> > > > > > (and
> > > > > > > > > not
> > > > > > > > > > different versions of a connector plugin).
> > > > > > > > > >
> > > > > > > > > > Sorry for not being more precise in my wording - I meant
> > > > > > registering
> > > > > > > > > > versions of schema for connector config.
> > > > > > > > > >
> > > > > > > > > > Let's take the example of a fictional connector which
> uses a
> > > > > > fictional
> > > > > > > > > AWS
> > > > > > > > > > service.
> > > > > > > > > >
> > > > > > > > > > Fictional Connector Config schema version:2.0
> > > > > > > > > > ---
> > > > > > > > > > {
> > > > > > > > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > > > > > > > "type": "object",
> > > > > > > > > > "properties": {
> > > > > > > > > > "name": {
> > > > > > > > > > "type": "string"
> > > > > > > > > > },
> > > > > > > > > > "schema_version": {
> > > > > > > > > > "type": "string"
> > > > > > > > > > },
> > > > > > > > > > "aws_access_key": {
> > > > > > > > > > "type": "string"
> > > > > > > > > > },
> > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > "type": "string"
> > > > > > > > > > }
> > > > > > > > > > },
> > > > > > > > > > "required": [
> > > > > > > > > > "name",
> > > > > > > > > > "schema_version",
> > > > > > > > > > "aws_access_key",
> > > > > > > > > > "aws_secret_key"
> > > > > > > > > > ]
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > Fictional Connector config schema version:3.0
> > > > > > > > > > ---
> > > > > > > > > > {
> > > > > > > > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > > > > > > > "type": "object",
> > > > > > > > > > "properties": {
> > > > > > > > > > "name": {
> > > > > > > > > > "type": "string"
> > > > > > > > > > },
> > > > > > > > > > "schema_version": {
> > > > > > > > > > "type": "string"
> > > > > > > > > > },
> > > > > > > > > > "iam_role": {
> > > > > > > > > > "type": "string"
> > > > > > > > > > }
> > > > > > > > > > },
> > > > > > > > > > "required": [
> > > > > > > > > > "name",
> > > > > > > > > > "schema_version",
> > > > > > > > > > "iam_role"
> > > > > > > > > > ]
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > The connector which supports Fictional config schema 2.0
> > > will
> > > > > > validate
> > > > > > > > > the
> > > > > > > > > > access key and secret key.
> > > > > > > > > > Whereas a connector which supports config with schema
> version
> > > > 3.0
> > > > > > will
> > > > > > > > > only
> > > > > > > > > > validate the IAM role.
> > > > > > > > > >
> > > > > > > > > > This is the alternative which I wanted to suggest. Each
> > > plugin
> > > > will
> > > > > > > > > > register the schema versions of connector config which it
> > > > supports.
> > > > > > > > > >
> > > > > > > > > > The plugin paths may be optionally different i.e we
> don't
> > > > have to
> > > > > > > > > > mandatorily add a new plugin path to support a new schema
> > > > version.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Ashwin
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <
> > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > > > >
> > > > > > > > > > > > 1. Can you elaborate on the rejected alternatives ?
> > > Suppose
> > > > > > connector
> > > > > > > > > > > > config is versioned and has a schema. Then a single
> > > plugin
> > > > > > (whose
> > > > > > > > > > > > dependencies have not changed) can handle multiple
> config
> > > > > > versions
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > same connector class.
> > > > > > > > > > >
> > > > > > > > > > > IIUC (please correct me if I am wrong here), what you
> > > > highlighted
> > > > > > > > > above,
> > > > > > > > > > is
> > > > > > > > > > > a versioning scheme for a connector config for the same
> > > > > > connector (and
> > > > > > > > > > not
> > > > > > > > > > > different versions of a connector plugin). That is a
> > > somewhat
> > > > > > > > > tangential
> > > > > > > > > > > problem. While it is definitely a useful feature to
> have,
> > > > like a
> > > > > > log to
> > > > > > > > > > > check what changes were made over time to the config
> which
> > > > might
> > > > > > make
> > > > > > > > > it
> > > > > > > > > > > easier to do rollbacks, it is not the focus here. Here
> by
> > > > > > version we
> > > > > > > > > mean
> > > > > > > > > > > to say what underlying version of the plugin should the
> > > given
> > > > > > > > > > configuration
> > > > > > > > > > > of the connector use. Perhaps it is better to change
> the
> > > > name of
> > > > > > the
> > > > > > > > > > > parameter from connector.version to
> > > connector.plugin.version
> > > > or
> > > > > > > > > > > plugin.version if it was confusing. wdyt?
> > > > > > > > > > >
> > > > > > > > > > > > 2. Any plans to support assisted migration e.g if a
> user
> > > > > > invokes
> > > > > > > > > "POST
> > > > > > > > > > > > connector/config?migrate=latest", the latest version
> > > > > > __attempts__ to
> > > > > > > > > > > > transform the existing config to the newer version.
> This
> > > > would
> > > > > > > > > require
> > > > > > > > > > > > adding a method like "boolean migrate(Version
> > > > fromVersion)" to
> > > > > > the
> > > > > > > > > > > > connector interface.
> > > > > > > > > > >
> > > > > > > > > > > This is an enhancement we can think of doing in future.
> > > > Users can
> > > > > > > > > simply
> > > > > > > > > > do
> > > > > > > > > > > a PUT call with the updated config which has the
> updated
> > > > version
> > > > > > > > > number.
> > > > > > > > > > > The assisted mode could be handy as the user does not
> need
> > > to
> > > > > > know the
> > > > > > > > > > > config but beyond this it does not seem to justify its
> > > > existence.
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > > Snehashis
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin
> > > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > >
> > > > > > > > > > > > This is a really useful feature and thanks for
> initiating
> > > > this
> > > > > > > > > > > discussion.
> > > > > > > > > > > >
> > > > > > > > > > > > I had the following questions -
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Can you elaborate on the rejected alternatives ?
> > > Suppose
> > > > > > connector
> > > > > > > > > > > > config is versioned and has a schema. Then a single
> > > plugin
> > > > > > (whose
> > > > > > > > > > > > dependencies have not changed) can handle multiple
> config
> > > > > > versions
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > same connector class.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. Any plans to support assisted migration e.g if a
> user
> > > > > > invokes
> > > > > > > > > "POST
> > > > > > > > > > > > connector/config?migrate=latest", the latest version
> > > > > > __attempts__ to
> > > > > > > > > > > > transform the existing config to the newer version.
> This
> > > > would
> > > > > > > > > require
> > > > > > > > > > > > adding a method like "boolean migrate(Version
> > > > fromVersion)" to
> > > > > > the
> > > > > > > > > > > > connector interface.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Ashwin
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <
> > > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'd like to start a discussion thread on KIP-891:
> > > Running
> > > > > > multiple
> > > > > > > > > > > > versions
> > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The KIP aims to add the ability for the connect
> runtime
> > > > to
> > > > > > run
> > > > > > > > > > multiple
> > > > > > > > > > > > > versions of a connector.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please take a look and let me know what you think.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > >
> > >
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Hey Chris,
Thanks for your comments. I'm glad you like the motivations, Snehashis
wrote that part!
> the configuration syntax for the most basic use case of
> specifying a single desired version is pretty counterintuitive.
I agree, and the "soft" requirement scheme is something I wasn't
explicitly looking for, but would be inherited from the library. I'm
fine with eliminating the soft requirement semantics, and having
"1.0.0" behave the same as "[1.0.0]". I'm less inclined to include
"range" in the property name, or have two properties.
> Failing startup is drastic and has the potential to disrupt the
> availability of connectors that would otherwise be able to run healthily
> because they were explicitly configured to use valid converters instead of
> the worker defaults.
I think this argument cuts both ways. If someone reconfigures a worker
and adds an invalid ".version" string to the worker (or changes the
plugin.path to make it invalid), it would be permitted to enter the
group, and accept work assignments. If those work assignments used
these configurations, a set of tasks could transition to FAILED and
not be able to recover, because they would be restarted again on the
same worker.
> Why are metrics utilized to report information about plugin versions
> utilized by connectors at runtime instead of publishing this info in the
> REST API
I was following the existing pattern for exposing runtime versions for
connectors, and it did seem like a "monitoring" feature. If that
approach is flawed and should be phased out, I think it would be a
good idea to reconsider the REST API rejected alternative.
We would need some additional design work to spec out the REST API
interface, as I don't have anything in mind currently.
> I'm unclear on whether or not it'll be possible to see this information via the
> GET /connector-plugins/<plugin>/config endpoint
There is room in the API to add recommenders for "key.converter",
"value.converter", and "header.converter", but not for transforms and
predicates, as they include aliases that depend on an actual
configuration. We could explicitly say we're going to do that, or do
whatever is convenient during the implementation phase, or leave it
open to be improved later.
There will not be any recommenders for ".version" properties in the
`/config` endpoint, because those recommenders are dynamic and depend
on an actual configuration.
5) There are two relevant lines in the KIP: "If a .version property
contains a hard requirement, select the latest installed version which
satisfies the requirement." and "This configuration is re-evaluated
each time the connector or task are assigned to a new worker". I would
call this "eager" upgrade behavior, rather than a "sticky" or "lazy"
upgrade behavior.
6) Updated!
Thanks,
Greg
On Tue, May 14, 2024 at 9:14 AM Chris Egerton <ch...@aiven.io.invalid> wrote:
>
> Hi all,
>
> Thanks Greg for updating the KIP, and thanks Snehashis for starting the
> work on this originally.
>
> The motivation section makes a pretty convincing case for this kind of
> feature. My thoughts are mostly about specific details:
>
> 1) I like the support for version ranges (the example demonstrating how to
> avoid KAFKA-10574 with the header converter was particularly entertaining),
> but the configuration syntax for the most basic use case of specifying a
> single desired version is pretty counterintuitive. People may get bitten or
> at least frustrated if they put connector.version=3.8.0 in a config but
> then version 3.7.5 ends up running. I'd like it if we could either
> intentionally deviate from Maven ranges when a bare version is present, or
> separate things out into two properties: foo.version would be the single
> accepted version for the foo plugin, and foo.version.range would use Maven
> range syntax. Open to other options too, just providing a couple to get the
> ball rolling.
>
> 2) Although the current behavior for a worker with an invalid
> key/value/header converter class specified in its config file is a little
> strange (I was surprised to learn that it wouldn't fail on startup), I
> don't see a good reason to deviate from this when an invalid version is
> specified. Failing startup is drastic and has the potential to disrupt the
> availability of connectors that would otherwise be able to run healthily
> because they were explicitly configured to use valid converters instead of
> the worker defaults.
>
> 3) Why are metrics utilized to report information about plugin versions
> utilized by connectors at runtime instead of publishing this info in the
> REST API? I saw that this was mentioned as a rejected alternative, but I
> didn't get a sense of why. It seems like the REST API would be easier to
> access and more intuitive for most users than new metrics.
>
> 4) In the "Validation" section it's stated that "Users can use these
> recommenders to discover the valid plugin classes and versions, without
> requiring an earlier call to GET /connector-plugins?connectorsOnly=false."
> I really like the creativity and simplicity of reusing the recommender
> mechanism to expose available versions for plugins via the REST API. I'm
> unclear on whether or not it'll be possible to see this information via the
> GET /connector-plugins/<plugin>/config endpoint, though. It'd be great if
> this were supported, since we learned in KIP-769 [1] that people really
> want to be able to see configuration options for connectors and their
> plugins via some kind of GET endpoint without having to provide a complete
> connector config for validation.
>
> 5) In the Maven version range docs, it's stated that "Maven picks the
> highest version of each project that satisfies all the hard requirements of
> the dependencies on that project." I'm guessing this behavior will be
> retained for Connect; i.e., the highest-possible version of each plugin
> that satisfies the user-specified version constraints will be run? (An
> alternative approach could be to have some kind of "sticky" logic that only
> restarts connectors/tasks when their currently-used version becomes
> incompatible with the configured constraints.)
>
> 6) (Nit) It'd be nice to add a link to the TestPlugins class or somewhere
> in its neighborhood to the testing plan; unfamiliar readers probably won't
> get much out of what's there right now.
>
> [1] -
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
>
> Cheers,
>
> Chris
>
> On Mon, May 13, 2024 at 2:01 PM Snehashis <sn...@gmail.com> wrote:
>
> > Hi Greg,
> >
> > That is much appreciated. No complaints on the additional scope, I will
> > make some time out to work on this once we have approval.
> >
> > Thanks
> > Snehashis
> >
> > On Fri, May 10, 2024 at 9:28 PM Greg Harris <gr...@aiven.io.invalid>
> > wrote:
> >
> > > Hey Snehashis,
> > >
> > > I'm glad to hear you're still interested in this KIP!
> > > I'm happy to let you drive this, and I apologize for increasing the
> > > scope of work so drastically. To make up for that, I'll volunteer to
> > > be the primary PR reviewer to help get this done quickly once the KIP
> > > is approved.
> > >
> > > Thanks,
> > > Greg
> > >
> > >
> > > On Fri, May 10, 2024 at 3:51 AM Snehashis <sn...@gmail.com>
> > > wrote:
> > > >
> > > > Hi Greg,
> > > >
> > > > Thanks for the follow up to my original KIP, I am in favour of the
> > > > additions made to expand its scope, the addition of range versions
> > > > specifically make a lot of sense.
> > > >
> > > > Apologies if I have not publicly worked on this KIP for a long time.
> > The
> > > > original work was done when the move to service loading was in
> > discussion
> > > > and I wanted to loop back to this only after that work was completed.
> > > Post
> > > > its conclusion, I have not been able to take this up due to other
> > > > priorities. If it's okay with you, I would still like to get this
> > > > implemented myself, including the additional scope.
> > > >
> > > > Thanks and regards
> > > > Snehashis
> > > >
> > > > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> > > <gr...@aiven.io.invalid>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to reboot the discussion on KIP-891:
> > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > >
> > > > > I've made some changes, most notably:
> > > > >
> > > > > 1. Specifying versions for all plugins in Connector configs
> > > > > (converters, header converters, transforms, and predicates) not just
> > > > > connectors & tasks
> > > > > 2. Specifying a range of versions instead of an exact match
> > > > > 3. New metrics to observe what versions are in-use
> > > > >
> > > > > Thanks to Snehashis for the original KIP idea!
> > > > >
> > > > > Thanks,
> > > > > Greg
> > > > >
> > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <gr...@aiven.io>
> > > wrote:
> > > > > >
> > > > > > Hi Snehashis,
> > > > > >
> > > > > > Thank you for the KIP! This is something I've wanted for a long
> > time.
> > > > > >
> > > > > > I know the discussion has gone cold, are you still interested in
> > > > > > pursuing this feature? I'll make time to review the KIP if you are
> > > > > > still accepting comments.
> > > > > >
> > > > > > Thanks,
> > > > > > Greg
> > > > > >
> > > > > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis <
> > snehashisp1995@gmail.com
> > > >
> > > > > wrote:
> > > > > > >
> > > > > > > Thanks for the points Sagar.
> > > > > > >
> > > > > > > > 1) Should we update the GET /connectors endpoint to include the
> > > > > version of
> > > > > > > > the plugin that is running? It could be useful to figure out
> > the
> > > > > version
> > > > > > > of
> > > > > > > > the plugin or I am assuming it gets returned by the expand=info
> > > call?
> > > > > > >
> > > > > > > I think this is good to have and possible future enhancement. The
> > > > > version
> > > > > > > info will be present in the config of the connector if the user
> > has
> > > > > > > specified the version. Otherwise it is the latest version which
> > the
> > > > > user
> > > > > > > can find out from the connector-plugin endpoint. The information
> > > can be
> > > > > > > introduced to the response of the GET /connectors endpoint
> > itself,
> > > > > however
> > > > > > > the most ideal way of doing this would be to get the currently
> > > running
> > > > > > > instance of the connector and get the version directly from
> > there.
> > > > > This is
> > > > > > > slightly tricky as the connector could be running in a different
> > > node.
> > > > > > > One way to do this would be to persist the version information in
> > > the
> > > > > > > status backing store during instantiation of the connector. It
> > > requires
> > > > > > > some more thought and since the version is part of the configs if
> > > > > provided
> > > > > > > and evident otherwise, I have not included it in this KIP.
> > > > > > >
> > > > > > > > 2) I am not aware of this and hence asking, can 2 connectors
> > with
> > > > > > > different
> > > > > > > > versions have the same name? Does the plugin isolation allow
> > > this?
> > > > > This
> > > > > > > > could have a bearing when using the lifecycle endpoints for
> > > > > connectors
> > > > > > > like
> > > > > > > > DELETE etc.
> > > > > > >
> > > > > > > All connectors in a cluster need to have uniquire connector names
> > > > > > > regardless of what version of the plugin the connector is running
> > > > > > > underneath. This is something enforced by the connect runtime
> > > itself.
> > > > > All
> > > > > > > connect CRUD operations are keyed on the connector name so there
> > > will
> > > > > not
> > > > > > > be an issue.
> > > > > > >
> > > > > > > Regards
> > > > > > > Snehashis
> > > > > > >
> > > > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <sagarmeansocean@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hey Snehashsih,
> > > > > > > >
> > > > > > > > Thanks for the KIP. It looks like a very useful feature. Couple
> > > of
> > > > > > > > small-ish points, let me know what you think:
> > > > > > > >
> > > > > > > > 1) Should we update the GET /connectors endpoint to include the
> > > > > version of
> > > > > > > > the plugin that is running? It could be useful to figure out
> > the
> > > > > version of
> > > > > > > > the plugin or I am assuming it gets returned by the expand=info
> > > call?
> > > > > > > > 2) I am not aware of this and hence asking, can 2 connectors
> > with
> > > > > different
> > > > > > > > versions have the same name? Does the plugin isolation allow
> > > this?
> > > > > This
> > > > > > > > could have a bearing when using the lifecycle endpoints for
> > > > > connectors like
> > > > > > > > DELETE etc.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > Sagar.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin
> > > <apankaj@confluent.io.invalid
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Snehasis,
> > > > > > > > >
> > > > > > > > > > IIUC (please correct me if I am wrong here), what you
> > > highlighted
> > > > > > > > above,
> > > > > > > > > is
> > > > > > > > > a versioning scheme for a connector config for the same
> > > connector
> > > > > (and
> > > > > > > > not
> > > > > > > > > different versions of a connector plugin).
> > > > > > > > >
> > > > > > > > > Sorry for not being more precise in my wording - I meant
> > > > > registering
> > > > > > > > > versions of schema for connector config.
> > > > > > > > >
> > > > > > > > > Let's take the example of a fictional connector which uses a
> > > > > fictional
> > > > > > > > AWS
> > > > > > > > > service.
> > > > > > > > >
> > > > > > > > > Fictional Connector Config schema version:2.0
> > > > > > > > > ---
> > > > > > > > > {
> > > > > > > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > > > > > > "type": "object",
> > > > > > > > > "properties": {
> > > > > > > > > "name": {
> > > > > > > > > "type": "string"
> > > > > > > > > },
> > > > > > > > > "schema_version": {
> > > > > > > > > "type": "string"
> > > > > > > > > },
> > > > > > > > > "aws_access_key": {
> > > > > > > > > "type": "string"
> > > > > > > > > },
> > > > > > > > > "aws_secret_key": {
> > > > > > > > > "type": "string"
> > > > > > > > > }
> > > > > > > > > },
> > > > > > > > > "required": [
> > > > > > > > > "name",
> > > > > > > > > "schema_version",
> > > > > > > > > "aws_access_key",
> > > > > > > > > "aws_secret_key"
> > > > > > > > > ]
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > Fictional Connector config schema version:3.0
> > > > > > > > > ---
> > > > > > > > > {
> > > > > > > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > > > > > > "type": "object",
> > > > > > > > > "properties": {
> > > > > > > > > "name": {
> > > > > > > > > "type": "string"
> > > > > > > > > },
> > > > > > > > > "schema_version": {
> > > > > > > > > "type": "string"
> > > > > > > > > },
> > > > > > > > > "iam_role": {
> > > > > > > > > "type": "string"
> > > > > > > > > }
> > > > > > > > > },
> > > > > > > > > "required": [
> > > > > > > > > "name",
> > > > > > > > > "schema_version",
> > > > > > > > > "iam_role"
> > > > > > > > > ]
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > The connector which supports Fictional config schema 2.0
> > will
> > > > > validate
> > > > > > > > the
> > > > > > > > > access key and secret key.
> > > > > > > > > Whereas a connector which supports config with schema version
> > > 3.0
> > > > > will
> > > > > > > > only
> > > > > > > > > validate the IAM role.
> > > > > > > > >
> > > > > > > > > This is the alternative which I wanted to suggest. Each
> > plugin
> > > will
> > > > > > > > > register the schema versions of connector config which it
> > > supports.
> > > > > > > > >
> > > > > > > > > The plugin paths may be optionally different i.e we don't
> > > have to
> > > > > > > > > mandatorily add a new plugin path to support a new schema
> > > version.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Ashwin
> > > > > > > > >
> > > > > > > > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <
> > > > > snehashisp1995@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > > >
> > > > > > > > > > > 1. Can you elaborate on the rejected alternatives ?
> > Suppose
> > > > > connector
> > > > > > > > > > > config is versioned and has a schema. Then a single
> > plugin
> > > > > (whose
> > > > > > > > > > > dependencies have not changed) can handle multiple config
> > > > > versions
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > same connector class.
> > > > > > > > > >
> > > > > > > > > > IIUC (please correct me if I am wrong here), what you
> > > highlighted
> > > > > > > > above,
> > > > > > > > > is
> > > > > > > > > > a versioning scheme for a connector config for the same
> > > > > connector (and
> > > > > > > > > not
> > > > > > > > > > different versions of a connector plugin). That is a
> > somewhat
> > > > > > > > tangential
> > > > > > > > > > problem. While it is definitely a useful feature to have,
> > > like a
> > > > > log to
> > > > > > > > > > check what changes were made over time to the config which
> > > might
> > > > > make
> > > > > > > > it
> > > > > > > > > > easier to do rollbacks, it is not the focus here. Here by
> > > > > version we
> > > > > > > > mean
> > > > > > > > > > to say what underlying version of the plugin should the
> > given
> > > > > > > > > configuration
> > > > > > > > > > of the connector use. Perhaps it is better to change the
> > > name of
> > > > > the
> > > > > > > > > > parameter from connector.version to
> > connector.plugin.version
> > > or
> > > > > > > > > > plugin.version if it was confusing. wdyt?
> > > > > > > > > >
> > > > > > > > > > > 2. Any plans to support assisted migration e.g if a user
> > > > > invokes
> > > > > > > > "POST
> > > > > > > > > > > connector/config?migrate=latest", the latest version
> > > > > __attempts__ to
> > > > > > > > > > > transform the existing config to the newer version. This
> > > would
> > > > > > > > require
> > > > > > > > > > > adding a method like "boolean migrate(Version
> > > fromVersion)" to
> > > > > the
> > > > > > > > > > > connector interface.
> > > > > > > > > >
> > > > > > > > > > This is an enhancement we can think of doing in future.
> > > Users can
> > > > > > > > simply
> > > > > > > > > do
> > > > > > > > > > a PUT call with the updated config which has the updated
> > > version
> > > > > > > > number.
> > > > > > > > > > The assisted mode could be handy as the user does not need
> > to
> > > > > know the
> > > > > > > > > > config but beyond this it does not seem to justify its
> > > existence.
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > > Snehashis
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin
> > > > > <ap...@confluent.io.invalid>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > >
> > > > > > > > > > > This is a really useful feature and thanks for initiating
> > > this
> > > > > > > > > > discussion.
> > > > > > > > > > >
> > > > > > > > > > > I had the following questions -
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 1. Can you elaborate on the rejected alternatives ?
> > Suppose
> > > > > connector
> > > > > > > > > > > config is versioned and has a schema. Then a single
> > plugin
> > > > > (whose
> > > > > > > > > > > dependencies have not changed) can handle multiple config
> > > > > versions
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > same connector class.
> > > > > > > > > > >
> > > > > > > > > > > 2. Any plans to support assisted migration e.g if a user
> > > > > invokes
> > > > > > > > "POST
> > > > > > > > > > > connector/config?migrate=latest", the latest version
> > > > > __attempts__ to
> > > > > > > > > > > transform the existing config to the newer version. This
> > > would
> > > > > > > > require
> > > > > > > > > > > adding a method like "boolean migrate(Version
> > > fromVersion)" to
> > > > > the
> > > > > > > > > > > connector interface.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Ashwin
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <
> > > > > snehashisp1995@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I'd like to start a discussion thread on KIP-891:
> > Running
> > > > > multiple
> > > > > > > > > > > versions
> > > > > > > > > > > > of a connector.
> > > > > > > > > > > >
> > > > > > > > > > > > The KIP aims to add the ability for the connect runtime
> > > to
> > > > > run
> > > > > > > > > multiple
> > > > > > > > > > > > versions of a connector.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > >
> > > > > > > > > > > > Please take a look and let me know what you think.
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you
> > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > >
> > >
> >
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Chris Egerton <ch...@aiven.io.INVALID>.
Hi all,
Thanks Greg for updating the KIP, and thanks Snehashis for starting the
work on this originally.
The motivation section makes a pretty convincing case for this kind of
feature. My thoughts are mostly about specific details:
1) I like the support for version ranges (the example demonstrating how to
avoid KAFKA-10574 with the header converter was particularly entertaining),
but the configuration syntax for the most basic use case of specifying a
single desired version is pretty counterintuitive. People may get bitten or
at least frustrated if they put connector.version=3.8.0 in a config but
then version 3.7.5 ends up running. I'd like it if we could either
intentionally deviate from Maven ranges when a bare version is present, or
separate things out into two properties: foo.version would be the single
accepted version for the foo plugin, and foo.version.range would use Maven
range syntax. Open to other options too, just providing a couple to get the
ball rolling.
2) Although the current behavior for a worker with an invalid
key/value/header converter class specified in its config file is a little
strange (I was surprised to learn that it wouldn't fail on startup), I
don't see a good reason to deviate from this when an invalid version is
specified. Failing startup is drastic and has the potential to disrupt the
availability of connectors that would otherwise be able to run healthily
because they were explicitly configured to use valid converters instead of
the worker defaults.
3) Why are metrics utilized to report information about plugin versions
utilized by connectors at runtime instead of publishing this info in the
REST API? I saw that this was mentioned as a rejected alternative, but I
didn't get a sense of why. It seems like the REST API would be easier to
access and more intuitive for most users than new metrics.
4) In the "Validation" section it's stated that "Users can use these
recommenders to discover the valid plugin classes and versions, without
requiring an earlier call to GET /connector-plugins?connectorsOnly=false."
I really like the creativity and simplicity of reusing the recommender
mechanism to expose available versions for plugins via the REST API. I'm
unclear on whether or not it'll be possible to see this information via the
GET /connector-plugins/<plugin>/config endpoint, though. It'd be great if
this were supported, since we learned in KIP-769 [1] that people really
want to be able to see configuration options for connectors and their
plugins via some kind of GET endpoint without having to provide a complete
connector config for validation.
5) In the Maven version range docs, it's stated that "Maven picks the
highest version of each project that satisfies all the hard requirements of
the dependencies on that project." I'm guessing this behavior will be
retained for Connect; i.e., the highest-possible version of each plugin
that satisfies the user-specified version constraints will be run? (An
alternative approach could be to have some kind of "sticky" logic that only
restarts connectors/tasks when their currently-used version becomes
incompatible with the configured constraints.)
6) (Nit) It'd be nice to add a link to the TestPlugins class or somewhere
in its neighborhood to the testing plan; unfamiliar readers probably won't
get much out of what's there right now.
[1] -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
Cheers,
Chris
On Mon, May 13, 2024 at 2:01 PM Snehashis <sn...@gmail.com> wrote:
> Hi Greg,
>
> That is much appreciated. No complaints on the additional scope, I will
> make some time out to work on this once we have approval.
>
> Thanks
> Snehashis
>
> On Fri, May 10, 2024 at 9:28 PM Greg Harris <gr...@aiven.io.invalid>
> wrote:
>
> > Hey Snehashis,
> >
> > I'm glad to hear you're still interested in this KIP!
> > I'm happy to let you drive this, and I apologize for increasing the
> > scope of work so drastically. To make up for that, I'll volunteer to
> > be the primary PR reviewer to help get this done quickly once the KIP
> > is approved.
> >
> > Thanks,
> > Greg
> >
> >
> > On Fri, May 10, 2024 at 3:51 AM Snehashis <sn...@gmail.com>
> > wrote:
> > >
> > > Hi Greg,
> > >
> > > Thanks for the follow up to my original KIP, I am in favour of the
> > > additions made to expand its scope, the addition of range versions
> > > specifically make a lot of sense.
> > >
> > > Apologies if I have not publicly worked on this KIP for a long time.
> The
> > > original work was done when the move to service loading was in
> discussion
> > > and I wanted to loop back to this only after that work was completed.
> > Post
> > > its conclusion, I have not been able to take this up due to other
> > > priorities. If it's okay with you, I would still like to get this
> > > implemented myself, including the additional scope.
> > >
> > > Thanks and regards
> > > Snehashis
> > >
> > > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> > <gr...@aiven.io.invalid>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to reboot the discussion on KIP-891:
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > >
> > > > I've made some changes, most notably:
> > > >
> > > > 1. Specifying versions for all plugins in Connector configs
> > > > (converters, header converters, transforms, and predicates) not just
> > > > connectors & tasks
> > > > 2. Specifying a range of versions instead of an exact match
> > > > 3. New metrics to observe what versions are in-use
> > > >
> > > > Thanks to Snehashis for the original KIP idea!
> > > >
> > > > Thanks,
> > > > Greg
> > > >
> > > > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <gr...@aiven.io>
> > wrote:
> > > > >
> > > > > Hi Snehashis,
> > > > >
> > > > > Thank you for the KIP! This is something I've wanted for a long
> time.
> > > > >
> > > > > I know the discussion has gone cold, are you still interested in
> > > > > pursuing this feature? I'll make time to review the KIP if you are
> > > > > still accepting comments.
> > > > >
> > > > > Thanks,
> > > > > Greg
> > > > >
> > > > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis <
> snehashisp1995@gmail.com
> > >
> > > > wrote:
> > > > > >
> > > > > > Thanks for the points Sagar.
> > > > > >
> > > > > > > 1) Should we update the GET /connectors endpoint to include the
> > > > version of
> > > > > > > the plugin that is running? It could be useful to figure out
> the
> > > > version
> > > > > > of
> > > > > > > the plugin or I am assuming it gets returned by the expand=info
> > call?
> > > > > >
> > > > > > I think this is good to have and possible future enhancement. The
> > > > version
> > > > > > info will be present in the config of the connector if the user
> has
> > > > > > specified the version. Otherwise it is the latest version which
> the
> > > > user
> > > > > > can find out from the connector-plugin endpoint. The information
> > can be
> > > > > > introduced to the response of the GET /connectors endpoint
> itself,
> > > > however
> > > > > > the most ideal way of doing this would be to get the currently
> > running
> > > > > > instance of the connector and get the version directly from
> there.
> > > > This is
> > > > > > slightly tricky as the connector could be running in a different
> > node.
> > > > > > One way to do this would be to persist the version information in
> > the
> > > > > > status backing store during instantiation of the connector. It
> > requires
> > > > > > some more thought and since the version is part of the configs if
> > > > provided
> > > > > > and evident otherwise, I have not included it in this KIP.
> > > > > >
> > > > > > > 2) I am not aware of this and hence asking, can 2 connectors
> with
> > > > > > different
> > > > > > > versions have the same name? Does the plugin isolation allow
> > this?
> > > > This
> > > > > > > could have a bearing when using the lifecycle endpoints for
> > > > connectors
> > > > > > like
> > > > > > > DELETE etc.
> > > > > >
> > > > > > All connectors in a cluster need to have uniquire connector names
> > > > > > regardless of what version of the plugin the connector is running
> > > > > > underneath. This is something enforced by the connect runtime
> > itself.
> > > > All
> > > > > > connect CRUD operations are keyed on the connector name so there
> > will
> > > > not
> > > > > > be an issue.
> > > > > >
> > > > > > Regards
> > > > > > Snehashis
> > > > > >
> > > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <sagarmeansocean@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Hey Snehashsih,
> > > > > > >
> > > > > > > Thanks for the KIP. It looks like a very useful feature. Couple
> > of
> > > > > > > small-ish points, let me know what you think:
> > > > > > >
> > > > > > > 1) Should we update the GET /connectors endpoint to include the
> > > > version of
> > > > > > > the plugin that is running? It could be useful to figure out
> the
> > > > version of
> > > > > > > the plugin or I am assuming it gets returned by the expand=info
> > call?
> > > > > > > 2) I am not aware of this and hence asking, can 2 connectors
> with
> > > > different
> > > > > > > versions have the same name? Does the plugin isolation allow
> > this?
> > > > This
> > > > > > > could have a bearing when using the lifecycle endpoints for
> > > > connectors like
> > > > > > > DELETE etc.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Sagar.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin
> > <apankaj@confluent.io.invalid
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Snehasis,
> > > > > > > >
> > > > > > > > > IIUC (please correct me if I am wrong here), what you
> > highlighted
> > > > > > > above,
> > > > > > > > is
> > > > > > > > a versioning scheme for a connector config for the same
> > connector
> > > > (and
> > > > > > > not
> > > > > > > > different versions of a connector plugin).
> > > > > > > >
> > > > > > > > Sorry for not being more precise in my wording - I meant
> > > > registering
> > > > > > > > versions of schema for connector config.
> > > > > > > >
> > > > > > > > Let's take the example of a fictional connector which uses a
> > > > fictional
> > > > > > > AWS
> > > > > > > > service.
> > > > > > > >
> > > > > > > > Fictional Connector Config schema version:2.0
> > > > > > > > ---
> > > > > > > > {
> > > > > > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > > > > > "type": "object",
> > > > > > > > "properties": {
> > > > > > > > "name": {
> > > > > > > > "type": "string"
> > > > > > > > },
> > > > > > > > "schema_version": {
> > > > > > > > "type": "string"
> > > > > > > > },
> > > > > > > > "aws_access_key": {
> > > > > > > > "type": "string"
> > > > > > > > },
> > > > > > > > "aws_secret_key": {
> > > > > > > > "type": "string"
> > > > > > > > }
> > > > > > > > },
> > > > > > > > "required": [
> > > > > > > > "name",
> > > > > > > > "schema_version",
> > > > > > > > "aws_access_key",
> > > > > > > > "aws_secret_key"
> > > > > > > > ]
> > > > > > > > }
> > > > > > > >
> > > > > > > > Fictional Connector config schema version:3.0
> > > > > > > > ---
> > > > > > > > {
> > > > > > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > > > > > "type": "object",
> > > > > > > > "properties": {
> > > > > > > > "name": {
> > > > > > > > "type": "string"
> > > > > > > > },
> > > > > > > > "schema_version": {
> > > > > > > > "type": "string"
> > > > > > > > },
> > > > > > > > "iam_role": {
> > > > > > > > "type": "string"
> > > > > > > > }
> > > > > > > > },
> > > > > > > > "required": [
> > > > > > > > "name",
> > > > > > > > "schema_version",
> > > > > > > > "iam_role"
> > > > > > > > ]
> > > > > > > > }
> > > > > > > >
> > > > > > > > The connector which supports Fictional config schema 2.0
> will
> > > > validate
> > > > > > > the
> > > > > > > > access key and secret key.
> > > > > > > > Whereas a connector which supports config with schema version
> > 3.0
> > > > will
> > > > > > > only
> > > > > > > > validate the IAM role.
> > > > > > > >
> > > > > > > > This is the alternative which I wanted to suggest. Each
> plugin
> > will
> > > > > > > > register the schema versions of connector config which it
> > supports.
> > > > > > > >
> > > > > > > > The plugin paths may be optionally different i.e we don't
> > have to
> > > > > > > > mandatorily add a new plugin path to support a new schema
> > version.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Ashwin
> > > > > > > >
> > > > > > > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <
> > > > snehashisp1995@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for the input Ashwin.
> > > > > > > > >
> > > > > > > > > > 1. Can you elaborate on the rejected alternatives ?
> Suppose
> > > > connector
> > > > > > > > > > config is versioned and has a schema. Then a single
> plugin
> > > > (whose
> > > > > > > > > > dependencies have not changed) can handle multiple config
> > > > versions
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > same connector class.
> > > > > > > > >
> > > > > > > > > IIUC (please correct me if I am wrong here), what you
> > highlighted
> > > > > > > above,
> > > > > > > > is
> > > > > > > > > a versioning scheme for a connector config for the same
> > > > connector (and
> > > > > > > > not
> > > > > > > > > different versions of a connector plugin). That is a
> somewhat
> > > > > > > tangential
> > > > > > > > > problem. While it is definitely a useful feature to have,
> > like a
> > > > log to
> > > > > > > > > check what changes were made over time to the config which
> > might
> > > > make
> > > > > > > it
> > > > > > > > > easier to do rollbacks, it is not the focus here. Here by
> > > > version we
> > > > > > > mean
> > > > > > > > > to say what underlying version of the plugin should the
> given
> > > > > > > > configuration
> > > > > > > > > of the connector use. Perhaps it is better to change the
> > name of
> > > > the
> > > > > > > > > parameter from connector.version to
> connector.plugin.version
> > or
> > > > > > > > > plugin.version if it was confusing. wdyt?
> > > > > > > > >
> > > > > > > > > > 2. Any plans to support assisted migration e.g if a user
> > > > invokes
> > > > > > > "POST
> > > > > > > > > > connector/config?migrate=latest", the latest version
> > > > __attempts__ to
> > > > > > > > > > transform the existing config to the newer version. This
> > would
> > > > > > > require
> > > > > > > > > > adding a method like "boolean migrate(Version
> > fromVersion)" to
> > > > the
> > > > > > > > > > connector interface.
> > > > > > > > >
> > > > > > > > > This is an enhancement we can think of doing in future.
> > Users can
> > > > > > > simply
> > > > > > > > do
> > > > > > > > > a PUT call with the updated config which has the updated
> > version
> > > > > > > number.
> > > > > > > > > The assisted mode could be handy as the user does not need
> to
> > > > know the
> > > > > > > > > config but beyond this it does not seem to justify its
> > existence.
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > Snehashis
> > > > > > > > >
> > > > > > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin
> > > > <ap...@confluent.io.invalid>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Snehasis,
> > > > > > > > > >
> > > > > > > > > > This is a really useful feature and thanks for initiating
> > this
> > > > > > > > > discussion.
> > > > > > > > > >
> > > > > > > > > > I had the following questions -
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 1. Can you elaborate on the rejected alternatives ?
> Suppose
> > > > connector
> > > > > > > > > > config is versioned and has a schema. Then a single
> plugin
> > > > (whose
> > > > > > > > > > dependencies have not changed) can handle multiple config
> > > > versions
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > same connector class.
> > > > > > > > > >
> > > > > > > > > > 2. Any plans to support assisted migration e.g if a user
> > > > invokes
> > > > > > > "POST
> > > > > > > > > > connector/config?migrate=latest", the latest version
> > > > __attempts__ to
> > > > > > > > > > transform the existing config to the newer version. This
> > would
> > > > > > > require
> > > > > > > > > > adding a method like "boolean migrate(Version
> > fromVersion)" to
> > > > the
> > > > > > > > > > connector interface.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Ashwin
> > > > > > > > > >
> > > > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <
> > > > snehashisp1995@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I'd like to start a discussion thread on KIP-891:
> Running
> > > > multiple
> > > > > > > > > > versions
> > > > > > > > > > > of a connector.
> > > > > > > > > > >
> > > > > > > > > > > The KIP aims to add the ability for the connect runtime
> > to
> > > > run
> > > > > > > > multiple
> > > > > > > > > > > versions of a connector.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > >
> > > > > > > > > > > Please take a look and let me know what you think.
> > > > > > > > > > >
> > > > > > > > > > > Thank you
> > > > > > > > > > > Snehashis Pal
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Snehashis <sn...@gmail.com>.
Hi Greg,
That is much appreciated. No complaints on the additional scope, I will
make some time out to work on this once we have approval.
Thanks
Snehashis
On Fri, May 10, 2024 at 9:28 PM Greg Harris <gr...@aiven.io.invalid>
wrote:
> Hey Snehashis,
>
> I'm glad to hear you're still interested in this KIP!
> I'm happy to let you drive this, and I apologize for increasing the
> scope of work so drastically. To make up for that, I'll volunteer to
> be the primary PR reviewer to help get this done quickly once the KIP
> is approved.
>
> Thanks,
> Greg
>
>
> On Fri, May 10, 2024 at 3:51 AM Snehashis <sn...@gmail.com>
> wrote:
> >
> > Hi Greg,
> >
> > Thanks for the follow up to my original KIP, I am in favour of the
> > additions made to expand its scope, the addition of range versions
> > specifically make a lot of sense.
> >
> > Apologies if I have not publicly worked on this KIP for a long time. The
> > original work was done when the move to service loading was in discussion
> > and I wanted to loop back to this only after that work was completed.
> Post
> > its conclusion, I have not been able to take this up due to other
> > priorities. If it's okay with you, I would still like to get this
> > implemented myself, including the additional scope.
> >
> > Thanks and regards
> > Snehashis
> >
> > On Fri, May 10, 2024 at 12:45 AM Greg Harris
> <gr...@aiven.io.invalid>
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to reboot the discussion on KIP-891:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > >
> > > I've made some changes, most notably:
> > >
> > > 1. Specifying versions for all plugins in Connector configs
> > > (converters, header converters, transforms, and predicates) not just
> > > connectors & tasks
> > > 2. Specifying a range of versions instead of an exact match
> > > 3. New metrics to observe what versions are in-use
> > >
> > > Thanks to Snehashis for the original KIP idea!
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <gr...@aiven.io>
> wrote:
> > > >
> > > > Hi Snehashis,
> > > >
> > > > Thank you for the KIP! This is something I've wanted for a long time.
> > > >
> > > > I know the discussion has gone cold, are you still interested in
> > > > pursuing this feature? I'll make time to review the KIP if you are
> > > > still accepting comments.
> > > >
> > > > Thanks,
> > > > Greg
> > > >
> > > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis <snehashisp1995@gmail.com
> >
> > > wrote:
> > > > >
> > > > > Thanks for the points Sagar.
> > > > >
> > > > > > 1) Should we update the GET /connectors endpoint to include the
> > > version of
> > > > > > the plugin that is running? It could be useful to figure out the
> > > version
> > > > > of
> > > > > > the plugin or I am assuming it gets returned by the expand=info
> call?
> > > > >
> > > > > I think this is good to have and possible future enhancement. The
> > > version
> > > > > info will be present in the config of the connector if the user has
> > > > > specified the version. Otherwise it is the latest version which the
> > > user
> > > > > can find out from the connector-plugin endpoint. The information
> can be
> > > > > introduced to the response of the GET /connectors endpoint itself,
> > > however
> > > > > the most ideal way of doing this would be to get the currently
> running
> > > > > instance of the connector and get the version directly from there.
> > > This is
> > > > > slightly tricky as the connector could be running in a different
> node.
> > > > > One way to do this would be to persist the version information in
> the
> > > > > status backing store during instantiation of the connector. It
> requires
> > > > > some more thought and since the version is part of the configs if
> > > provided
> > > > > and evident otherwise, I have not included it in this KIP.
> > > > >
> > > > > > 2) I am not aware of this and hence asking, can 2 connectors with
> > > > > different
> > > > > > versions have the same name? Does the plugin isolation allow
> this?
> > > This
> > > > > > could have a bearing when using the lifecycle endpoints for
> > > connectors
> > > > > like
> > > > > > DELETE etc.
> > > > >
> > > > > All connectors in a cluster need to have uniquire connector names
> > > > > regardless of what version of the plugin the connector is running
> > > > > underneath. This is something enforced by the connect runtime
> itself.
> > > All
> > > > > connect CRUD operations are keyed on the connector name so there
> will
> > > not
> > > > > be an issue.
> > > > >
> > > > > Regards
> > > > > Snehashis
> > > > >
> > > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <sa...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hey Snehashsih,
> > > > > >
> > > > > > Thanks for the KIP. It looks like a very useful feature. Couple
> of
> > > > > > small-ish points, let me know what you think:
> > > > > >
> > > > > > 1) Should we update the GET /connectors endpoint to include the
> > > version of
> > > > > > the plugin that is running? It could be useful to figure out the
> > > version of
> > > > > > the plugin or I am assuming it gets returned by the expand=info
> call?
> > > > > > 2) I am not aware of this and hence asking, can 2 connectors with
> > > different
> > > > > > versions have the same name? Does the plugin isolation allow
> this?
> > > This
> > > > > > could have a bearing when using the lifecycle endpoints for
> > > connectors like
> > > > > > DELETE etc.
> > > > > >
> > > > > > Thanks!
> > > > > > Sagar.
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin
> <apankaj@confluent.io.invalid
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Snehasis,
> > > > > > >
> > > > > > > > IIUC (please correct me if I am wrong here), what you
> highlighted
> > > > > > above,
> > > > > > > is
> > > > > > > a versioning scheme for a connector config for the same
> connector
> > > (and
> > > > > > not
> > > > > > > different versions of a connector plugin).
> > > > > > >
> > > > > > > Sorry for not being more precise in my wording - I meant
> > > registering
> > > > > > > versions of schema for connector config.
> > > > > > >
> > > > > > > Let's take the example of a fictional connector which uses a
> > > fictional
> > > > > > AWS
> > > > > > > service.
> > > > > > >
> > > > > > > Fictional Connector Config schema version:2.0
> > > > > > > ---
> > > > > > > {
> > > > > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > > > > "type": "object",
> > > > > > > "properties": {
> > > > > > > "name": {
> > > > > > > "type": "string"
> > > > > > > },
> > > > > > > "schema_version": {
> > > > > > > "type": "string"
> > > > > > > },
> > > > > > > "aws_access_key": {
> > > > > > > "type": "string"
> > > > > > > },
> > > > > > > "aws_secret_key": {
> > > > > > > "type": "string"
> > > > > > > }
> > > > > > > },
> > > > > > > "required": [
> > > > > > > "name",
> > > > > > > "schema_version",
> > > > > > > "aws_access_key",
> > > > > > > "aws_secret_key"
> > > > > > > ]
> > > > > > > }
> > > > > > >
> > > > > > > Fictional Connector config schema version:3.0
> > > > > > > ---
> > > > > > > {
> > > > > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > > > > "type": "object",
> > > > > > > "properties": {
> > > > > > > "name": {
> > > > > > > "type": "string"
> > > > > > > },
> > > > > > > "schema_version": {
> > > > > > > "type": "string"
> > > > > > > },
> > > > > > > "iam_role": {
> > > > > > > "type": "string"
> > > > > > > }
> > > > > > > },
> > > > > > > "required": [
> > > > > > > "name",
> > > > > > > "schema_version",
> > > > > > > "iam_role"
> > > > > > > ]
> > > > > > > }
> > > > > > >
> > > > > > > The connector which supports Fictional config schema 2.0 will
> > > validate
> > > > > > the
> > > > > > > access key and secret key.
> > > > > > > Whereas a connector which supports config with schema version
> 3.0
> > > will
> > > > > > only
> > > > > > > validate the IAM role.
> > > > > > >
> > > > > > > This is the alternative which I wanted to suggest. Each plugin
> will
> > > > > > > register the schema versions of connector config which it
> supports.
> > > > > > >
> > > > > > > The plugin paths may be optionally different i.e we don't
> have to
> > > > > > > mandatorily add a new plugin path to support a new schema
> version.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Ashwin
> > > > > > >
> > > > > > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <
> > > snehashisp1995@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for the input Ashwin.
> > > > > > > >
> > > > > > > > > 1. Can you elaborate on the rejected alternatives ? Suppose
> > > connector
> > > > > > > > > config is versioned and has a schema. Then a single plugin
> > > (whose
> > > > > > > > > dependencies have not changed) can handle multiple config
> > > versions
> > > > > > for
> > > > > > > > the
> > > > > > > > > same connector class.
> > > > > > > >
> > > > > > > > IIUC (please correct me if I am wrong here), what you
> highlighted
> > > > > > above,
> > > > > > > is
> > > > > > > > a versioning scheme for a connector config for the same
> > > connector (and
> > > > > > > not
> > > > > > > > different versions of a connector plugin). That is a somewhat
> > > > > > tangential
> > > > > > > > problem. While it is definitely a useful feature to have,
> like a
> > > log to
> > > > > > > > check what changes were made over time to the config which
> might
> > > make
> > > > > > it
> > > > > > > > easier to do rollbacks, it is not the focus here. Here by
> > > version we
> > > > > > mean
> > > > > > > > to say what underlying version of the plugin should the given
> > > > > > > configuration
> > > > > > > > of the connector use. Perhaps it is better to change the
> name of
> > > the
> > > > > > > > parameter from connector.version to connector.plugin.version
> or
> > > > > > > > plugin.version if it was confusing. wdyt?
> > > > > > > >
> > > > > > > > > 2. Any plans to support assisted migration e.g if a user
> > > invokes
> > > > > > "POST
> > > > > > > > > connector/config?migrate=latest", the latest version
> > > __attempts__ to
> > > > > > > > > transform the existing config to the newer version. This
> would
> > > > > > require
> > > > > > > > > adding a method like "boolean migrate(Version
> fromVersion)" to
> > > the
> > > > > > > > > connector interface.
> > > > > > > >
> > > > > > > > This is an enhancement we can think of doing in future.
> Users can
> > > > > > simply
> > > > > > > do
> > > > > > > > a PUT call with the updated config which has the updated
> version
> > > > > > number.
> > > > > > > > The assisted mode could be handy as the user does not need to
> > > know the
> > > > > > > > config but beyond this it does not seem to justify its
> existence.
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Snehashis
> > > > > > > >
> > > > > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin
> > > <ap...@confluent.io.invalid>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Snehasis,
> > > > > > > > >
> > > > > > > > > This is a really useful feature and thanks for initiating
> this
> > > > > > > > discussion.
> > > > > > > > >
> > > > > > > > > I had the following questions -
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 1. Can you elaborate on the rejected alternatives ? Suppose
> > > connector
> > > > > > > > > config is versioned and has a schema. Then a single plugin
> > > (whose
> > > > > > > > > dependencies have not changed) can handle multiple config
> > > versions
> > > > > > for
> > > > > > > > the
> > > > > > > > > same connector class.
> > > > > > > > >
> > > > > > > > > 2. Any plans to support assisted migration e.g if a user
> > > invokes
> > > > > > "POST
> > > > > > > > > connector/config?migrate=latest", the latest version
> > > __attempts__ to
> > > > > > > > > transform the existing config to the newer version. This
> would
> > > > > > require
> > > > > > > > > adding a method like "boolean migrate(Version
> fromVersion)" to
> > > the
> > > > > > > > > connector interface.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Ashwin
> > > > > > > > >
> > > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <
> > > snehashisp1995@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I'd like to start a discussion thread on KIP-891: Running
> > > multiple
> > > > > > > > > versions
> > > > > > > > > > of a connector.
> > > > > > > > > >
> > > > > > > > > > The KIP aims to add the ability for the connect runtime
> to
> > > run
> > > > > > > multiple
> > > > > > > > > > versions of a connector.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > >
> > > > > > > > > > Please take a look and let me know what you think.
> > > > > > > > > >
> > > > > > > > > > Thank you
> > > > > > > > > > Snehashis Pal
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > >
>
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Hey Snehashis,
I'm glad to hear you're still interested in this KIP!
I'm happy to let you drive this, and I apologize for increasing the
scope of work so drastically. To make up for that, I'll volunteer to
be the primary PR reviewer to help get this done quickly once the KIP
is approved.
Thanks,
Greg
On Fri, May 10, 2024 at 3:51 AM Snehashis <sn...@gmail.com> wrote:
>
> Hi Greg,
>
> Thanks for the follow up to my original KIP, I am in favour of the
> additions made to expand its scope, the addition of range versions
> specifically make a lot of sense.
>
> Apologies if I have not publicly worked on this KIP for a long time. The
> original work was done when the move to service loading was in discussion
> and I wanted to loop back to this only after that work was completed. Post
> its conclusion, I have not been able to take this up due to other
> priorities. If it's okay with you, I would still like to get this
> implemented myself, including the additional scope.
>
> Thanks and regards
> Snehashis
>
> On Fri, May 10, 2024 at 12:45 AM Greg Harris <gr...@aiven.io.invalid>
> wrote:
>
> > Hi all,
> >
> > I'd like to reboot the discussion on KIP-891:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> >
> > I've made some changes, most notably:
> >
> > 1. Specifying versions for all plugins in Connector configs
> > (converters, header converters, transforms, and predicates) not just
> > connectors & tasks
> > 2. Specifying a range of versions instead of an exact match
> > 3. New metrics to observe what versions are in-use
> >
> > Thanks to Snehashis for the original KIP idea!
> >
> > Thanks,
> > Greg
> >
> > On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <gr...@aiven.io> wrote:
> > >
> > > Hi Snehashis,
> > >
> > > Thank you for the KIP! This is something I've wanted for a long time.
> > >
> > > I know the discussion has gone cold, are you still interested in
> > > pursuing this feature? I'll make time to review the KIP if you are
> > > still accepting comments.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Tue, Nov 22, 2022 at 12:29 PM Snehashis <sn...@gmail.com>
> > wrote:
> > > >
> > > > Thanks for the points Sagar.
> > > >
> > > > > 1) Should we update the GET /connectors endpoint to include the
> > version of
> > > > > the plugin that is running? It could be useful to figure out the
> > version
> > > > of
> > > > > the plugin or I am assuming it gets returned by the expand=info call?
> > > >
> > > > I think this is good to have and possible future enhancement. The
> > version
> > > > info will be present in the config of the connector if the user has
> > > > specified the version. Otherwise it is the latest version which the
> > user
> > > > can find out from the connector-plugin endpoint. The information can be
> > > > introduced to the response of the GET /connectors endpoint itself,
> > however
> > > > the most ideal way of doing this would be to get the currently running
> > > > instance of the connector and get the version directly from there.
> > This is
> > > > slightly tricky as the connector could be running in a different node.
> > > > One way to do this would be to persist the version information in the
> > > > status backing store during instantiation of the connector. It requires
> > > > some more thought and since the version is part of the configs if
> > provided
> > > > and evident otherwise, I have not included it in this KIP.
> > > >
> > > > > 2) I am not aware of this and hence asking, can 2 connectors with
> > > > different
> > > > > versions have the same name? Does the plugin isolation allow this?
> > This
> > > > > could have a bearing when using the lifecycle endpoints for
> > connectors
> > > > like
> > > > > DELETE etc.
> > > >
> > > > All connectors in a cluster need to have uniquire connector names
> > > > regardless of what version of the plugin the connector is running
> > > > underneath. This is something enforced by the connect runtime itself.
> > All
> > > > connect CRUD operations are keyed on the connector name so there will
> > not
> > > > be an issue.
> > > >
> > > > Regards
> > > > Snehashis
> > > >
> > > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <sa...@gmail.com>
> > wrote:
> > > >
> > > > > Hey Snehashsih,
> > > > >
> > > > > Thanks for the KIP. It looks like a very useful feature. Couple of
> > > > > small-ish points, let me know what you think:
> > > > >
> > > > > 1) Should we update the GET /connectors endpoint to include the
> > version of
> > > > > the plugin that is running? It could be useful to figure out the
> > version of
> > > > > the plugin or I am assuming it gets returned by the expand=info call?
> > > > > 2) I am not aware of this and hence asking, can 2 connectors with
> > different
> > > > > versions have the same name? Does the plugin isolation allow this?
> > This
> > > > > could have a bearing when using the lifecycle endpoints for
> > connectors like
> > > > > DELETE etc.
> > > > >
> > > > > Thanks!
> > > > > Sagar.
> > > > >
> > > > >
> > > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin <apankaj@confluent.io.invalid
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi Snehasis,
> > > > > >
> > > > > > > IIUC (please correct me if I am wrong here), what you highlighted
> > > > > above,
> > > > > > is
> > > > > > a versioning scheme for a connector config for the same connector
> > (and
> > > > > not
> > > > > > different versions of a connector plugin).
> > > > > >
> > > > > > Sorry for not being more precise in my wording - I meant
> > registering
> > > > > > versions of schema for connector config.
> > > > > >
> > > > > > Let's take the example of a fictional connector which uses a
> > fictional
> > > > > AWS
> > > > > > service.
> > > > > >
> > > > > > Fictional Connector Config schema version:2.0
> > > > > > ---
> > > > > > {
> > > > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > > > "type": "object",
> > > > > > "properties": {
> > > > > > "name": {
> > > > > > "type": "string"
> > > > > > },
> > > > > > "schema_version": {
> > > > > > "type": "string"
> > > > > > },
> > > > > > "aws_access_key": {
> > > > > > "type": "string"
> > > > > > },
> > > > > > "aws_secret_key": {
> > > > > > "type": "string"
> > > > > > }
> > > > > > },
> > > > > > "required": [
> > > > > > "name",
> > > > > > "schema_version",
> > > > > > "aws_access_key",
> > > > > > "aws_secret_key"
> > > > > > ]
> > > > > > }
> > > > > >
> > > > > > Fictional Connector config schema version:3.0
> > > > > > ---
> > > > > > {
> > > > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > > > "type": "object",
> > > > > > "properties": {
> > > > > > "name": {
> > > > > > "type": "string"
> > > > > > },
> > > > > > "schema_version": {
> > > > > > "type": "string"
> > > > > > },
> > > > > > "iam_role": {
> > > > > > "type": "string"
> > > > > > }
> > > > > > },
> > > > > > "required": [
> > > > > > "name",
> > > > > > "schema_version",
> > > > > > "iam_role"
> > > > > > ]
> > > > > > }
> > > > > >
> > > > > > The connector which supports Fictional config schema 2.0 will
> > validate
> > > > > the
> > > > > > access key and secret key.
> > > > > > Whereas a connector which supports config with schema version 3.0
> > will
> > > > > only
> > > > > > validate the IAM role.
> > > > > >
> > > > > > This is the alternative which I wanted to suggest. Each plugin will
> > > > > > register the schema versions of connector config which it supports.
> > > > > >
> > > > > > The plugin paths may be optionally different i.e we don't have to
> > > > > > mandatorily add a new plugin path to support a new schema version.
> > > > > >
> > > > > > Thanks,
> > > > > > Ashwin
> > > > > >
> > > > > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <
> > snehashisp1995@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for the input Ashwin.
> > > > > > >
> > > > > > > > 1. Can you elaborate on the rejected alternatives ? Suppose
> > connector
> > > > > > > > config is versioned and has a schema. Then a single plugin
> > (whose
> > > > > > > > dependencies have not changed) can handle multiple config
> > versions
> > > > > for
> > > > > > > the
> > > > > > > > same connector class.
> > > > > > >
> > > > > > > IIUC (please correct me if I am wrong here), what you highlighted
> > > > > above,
> > > > > > is
> > > > > > > a versioning scheme for a connector config for the same
> > connector (and
> > > > > > not
> > > > > > > different versions of a connector plugin). That is a somewhat
> > > > > tangential
> > > > > > > problem. While it is definitely a useful feature to have, like a
> > log to
> > > > > > > check what changes were made over time to the config which might
> > make
> > > > > it
> > > > > > > easier to do rollbacks, it is not the focus here. Here by
> > version we
> > > > > mean
> > > > > > > to say what underlying version of the plugin should the given
> > > > > > configuration
> > > > > > > of the connector use. Perhaps it is better to change the name of
> > the
> > > > > > > parameter from connector.version to connector.plugin.version or
> > > > > > > plugin.version if it was confusing. wdyt?
> > > > > > >
> > > > > > > > 2. Any plans to support assisted migration e.g if a user
> > invokes
> > > > > "POST
> > > > > > > > connector/config?migrate=latest", the latest version
> > __attempts__ to
> > > > > > > > transform the existing config to the newer version. This would
> > > > > require
> > > > > > > > adding a method like "boolean migrate(Version fromVersion)" to
> > the
> > > > > > > > connector interface.
> > > > > > >
> > > > > > > This is an enhancement we can think of doing in future. Users can
> > > > > simply
> > > > > > do
> > > > > > > a PUT call with the updated config which has the updated version
> > > > > number.
> > > > > > > The assisted mode could be handy as the user does not need to
> > know the
> > > > > > > config but beyond this it does not seem to justify its existence.
> > > > > > >
> > > > > > > Regards
> > > > > > > Snehashis
> > > > > > >
> > > > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin
> > <ap...@confluent.io.invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Snehasis,
> > > > > > > >
> > > > > > > > This is a really useful feature and thanks for initiating this
> > > > > > > discussion.
> > > > > > > >
> > > > > > > > I had the following questions -
> > > > > > > >
> > > > > > > >
> > > > > > > > 1. Can you elaborate on the rejected alternatives ? Suppose
> > connector
> > > > > > > > config is versioned and has a schema. Then a single plugin
> > (whose
> > > > > > > > dependencies have not changed) can handle multiple config
> > versions
> > > > > for
> > > > > > > the
> > > > > > > > same connector class.
> > > > > > > >
> > > > > > > > 2. Any plans to support assisted migration e.g if a user
> > invokes
> > > > > "POST
> > > > > > > > connector/config?migrate=latest", the latest version
> > __attempts__ to
> > > > > > > > transform the existing config to the newer version. This would
> > > > > require
> > > > > > > > adding a method like "boolean migrate(Version fromVersion)" to
> > the
> > > > > > > > connector interface.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Ashwin
> > > > > > > >
> > > > > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <
> > snehashisp1995@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I'd like to start a discussion thread on KIP-891: Running
> > multiple
> > > > > > > > versions
> > > > > > > > > of a connector.
> > > > > > > > >
> > > > > > > > > The KIP aims to add the ability for the connect runtime to
> > run
> > > > > > multiple
> > > > > > > > > versions of a connector.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > >
> > > > > > > > > Please take a look and let me know what you think.
> > > > > > > > >
> > > > > > > > > Thank you
> > > > > > > > > Snehashis Pal
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Snehashis <sn...@gmail.com>.
Hi Greg,
Thanks for the follow up to my original KIP, I am in favour of the
additions made to expand its scope, the addition of range versions
specifically make a lot of sense.
Apologies if I have not publicly worked on this KIP for a long time. The
original work was done when the move to service loading was in discussion
and I wanted to loop back to this only after that work was completed. Post
its conclusion, I have not been able to take this up due to other
priorities. If it's okay with you, I would still like to get this
implemented myself, including the additional scope.
Thanks and regards
Snehashis
On Fri, May 10, 2024 at 12:45 AM Greg Harris <gr...@aiven.io.invalid>
wrote:
> Hi all,
>
> I'd like to reboot the discussion on KIP-891:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
>
> I've made some changes, most notably:
>
> 1. Specifying versions for all plugins in Connector configs
> (converters, header converters, transforms, and predicates) not just
> connectors & tasks
> 2. Specifying a range of versions instead of an exact match
> 3. New metrics to observe what versions are in-use
>
> Thanks to Snehashis for the original KIP idea!
>
> Thanks,
> Greg
>
> On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <gr...@aiven.io> wrote:
> >
> > Hi Snehashis,
> >
> > Thank you for the KIP! This is something I've wanted for a long time.
> >
> > I know the discussion has gone cold, are you still interested in
> > pursuing this feature? I'll make time to review the KIP if you are
> > still accepting comments.
> >
> > Thanks,
> > Greg
> >
> > On Tue, Nov 22, 2022 at 12:29 PM Snehashis <sn...@gmail.com>
> wrote:
> > >
> > > Thanks for the points Sagar.
> > >
> > > > 1) Should we update the GET /connectors endpoint to include the
> version of
> > > > the plugin that is running? It could be useful to figure out the
> version
> > > of
> > > > the plugin or I am assuming it gets returned by the expand=info call?
> > >
> > > I think this is good to have and possible future enhancement. The
> version
> > > info will be present in the config of the connector if the user has
> > > specified the version. Otherwise it is the latest version which the
> user
> > > can find out from the connector-plugin endpoint. The information can be
> > > introduced to the response of the GET /connectors endpoint itself,
> however
> > > the most ideal way of doing this would be to get the currently running
> > > instance of the connector and get the version directly from there.
> This is
> > > slightly tricky as the connector could be running in a different node.
> > > One way to do this would be to persist the version information in the
> > > status backing store during instantiation of the connector. It requires
> > > some more thought and since the version is part of the configs if
> provided
> > > and evident otherwise, I have not included it in this KIP.
> > >
> > > > 2) I am not aware of this and hence asking, can 2 connectors with
> > > different
> > > > versions have the same name? Does the plugin isolation allow this?
> This
> > > > could have a bearing when using the lifecycle endpoints for
> connectors
> > > like
> > > > DELETE etc.
> > >
> > > All connectors in a cluster need to have uniquire connector names
> > > regardless of what version of the plugin the connector is running
> > > underneath. This is something enforced by the connect runtime itself.
> All
> > > connect CRUD operations are keyed on the connector name so there will
> not
> > > be an issue.
> > >
> > > Regards
> > > Snehashis
> > >
> > > On Tue, Nov 22, 2022 at 3:16 PM Sagar <sa...@gmail.com>
> wrote:
> > >
> > > > Hey Snehashsih,
> > > >
> > > > Thanks for the KIP. It looks like a very useful feature. Couple of
> > > > small-ish points, let me know what you think:
> > > >
> > > > 1) Should we update the GET /connectors endpoint to include the
> version of
> > > > the plugin that is running? It could be useful to figure out the
> version of
> > > > the plugin or I am assuming it gets returned by the expand=info call?
> > > > 2) I am not aware of this and hence asking, can 2 connectors with
> different
> > > > versions have the same name? Does the plugin isolation allow this?
> This
> > > > could have a bearing when using the lifecycle endpoints for
> connectors like
> > > > DELETE etc.
> > > >
> > > > Thanks!
> > > > Sagar.
> > > >
> > > >
> > > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin <apankaj@confluent.io.invalid
> >
> > > > wrote:
> > > >
> > > > > Hi Snehasis,
> > > > >
> > > > > > IIUC (please correct me if I am wrong here), what you highlighted
> > > > above,
> > > > > is
> > > > > a versioning scheme for a connector config for the same connector
> (and
> > > > not
> > > > > different versions of a connector plugin).
> > > > >
> > > > > Sorry for not being more precise in my wording - I meant
> registering
> > > > > versions of schema for connector config.
> > > > >
> > > > > Let's take the example of a fictional connector which uses a
> fictional
> > > > AWS
> > > > > service.
> > > > >
> > > > > Fictional Connector Config schema version:2.0
> > > > > ---
> > > > > {
> > > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > > "type": "object",
> > > > > "properties": {
> > > > > "name": {
> > > > > "type": "string"
> > > > > },
> > > > > "schema_version": {
> > > > > "type": "string"
> > > > > },
> > > > > "aws_access_key": {
> > > > > "type": "string"
> > > > > },
> > > > > "aws_secret_key": {
> > > > > "type": "string"
> > > > > }
> > > > > },
> > > > > "required": [
> > > > > "name",
> > > > > "schema_version",
> > > > > "aws_access_key",
> > > > > "aws_secret_key"
> > > > > ]
> > > > > }
> > > > >
> > > > > Fictional Connector config schema version:3.0
> > > > > ---
> > > > > {
> > > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > > "type": "object",
> > > > > "properties": {
> > > > > "name": {
> > > > > "type": "string"
> > > > > },
> > > > > "schema_version": {
> > > > > "type": "string"
> > > > > },
> > > > > "iam_role": {
> > > > > "type": "string"
> > > > > }
> > > > > },
> > > > > "required": [
> > > > > "name",
> > > > > "schema_version",
> > > > > "iam_role"
> > > > > ]
> > > > > }
> > > > >
> > > > > The connector which supports Fictional config schema 2.0 will
> validate
> > > > the
> > > > > access key and secret key.
> > > > > Whereas a connector which supports config with schema version 3.0
> will
> > > > only
> > > > > validate the IAM role.
> > > > >
> > > > > This is the alternative which I wanted to suggest. Each plugin will
> > > > > register the schema versions of connector config which it supports.
> > > > >
> > > > > The plugin paths may be optionally different i.e we don't have to
> > > > > mandatorily add a new plugin path to support a new schema version.
> > > > >
> > > > > Thanks,
> > > > > Ashwin
> > > > >
> > > > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <
> snehashisp1995@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks for the input Ashwin.
> > > > > >
> > > > > > > 1. Can you elaborate on the rejected alternatives ? Suppose
> connector
> > > > > > > config is versioned and has a schema. Then a single plugin
> (whose
> > > > > > > dependencies have not changed) can handle multiple config
> versions
> > > > for
> > > > > > the
> > > > > > > same connector class.
> > > > > >
> > > > > > IIUC (please correct me if I am wrong here), what you highlighted
> > > > above,
> > > > > is
> > > > > > a versioning scheme for a connector config for the same
> connector (and
> > > > > not
> > > > > > different versions of a connector plugin). That is a somewhat
> > > > tangential
> > > > > > problem. While it is definitely a useful feature to have, like a
> log to
> > > > > > check what changes were made over time to the config which might
> make
> > > > it
> > > > > > easier to do rollbacks, it is not the focus here. Here by
> version we
> > > > mean
> > > > > > to say what underlying version of the plugin should the given
> > > > > configuration
> > > > > > of the connector use. Perhaps it is better to change the name of
> the
> > > > > > parameter from connector.version to connector.plugin.version or
> > > > > > plugin.version if it was confusing. wdyt?
> > > > > >
> > > > > > > 2. Any plans to support assisted migration e.g if a user
> invokes
> > > > "POST
> > > > > > > connector/config?migrate=latest", the latest version
> __attempts__ to
> > > > > > > transform the existing config to the newer version. This would
> > > > require
> > > > > > > adding a method like "boolean migrate(Version fromVersion)" to
> the
> > > > > > > connector interface.
> > > > > >
> > > > > > This is an enhancement we can think of doing in future. Users can
> > > > simply
> > > > > do
> > > > > > a PUT call with the updated config which has the updated version
> > > > number.
> > > > > > The assisted mode could be handy as the user does not need to
> know the
> > > > > > config but beyond this it does not seem to justify its existence.
> > > > > >
> > > > > > Regards
> > > > > > Snehashis
> > > > > >
> > > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin
> <ap...@confluent.io.invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Snehasis,
> > > > > > >
> > > > > > > This is a really useful feature and thanks for initiating this
> > > > > > discussion.
> > > > > > >
> > > > > > > I had the following questions -
> > > > > > >
> > > > > > >
> > > > > > > 1. Can you elaborate on the rejected alternatives ? Suppose
> connector
> > > > > > > config is versioned and has a schema. Then a single plugin
> (whose
> > > > > > > dependencies have not changed) can handle multiple config
> versions
> > > > for
> > > > > > the
> > > > > > > same connector class.
> > > > > > >
> > > > > > > 2. Any plans to support assisted migration e.g if a user
> invokes
> > > > "POST
> > > > > > > connector/config?migrate=latest", the latest version
> __attempts__ to
> > > > > > > transform the existing config to the newer version. This would
> > > > require
> > > > > > > adding a method like "boolean migrate(Version fromVersion)" to
> the
> > > > > > > connector interface.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Ashwin
> > > > > > >
> > > > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <
> snehashisp1995@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I'd like to start a discussion thread on KIP-891: Running
> multiple
> > > > > > > versions
> > > > > > > > of a connector.
> > > > > > > >
> > > > > > > > The KIP aims to add the ability for the connect runtime to
> run
> > > > > multiple
> > > > > > > > versions of a connector.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > >
> > > > > > > > Please take a look and let me know what you think.
> > > > > > > >
> > > > > > > > Thank you
> > > > > > > > Snehashis Pal
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
>
[DISCUSS] KIP-891: Running multiple versions of Connector plugins
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Hi all,
I'd like to reboot the discussion on KIP-891:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
I've made some changes, most notably:
1. Specifying versions for all plugins in Connector configs
(converters, header converters, transforms, and predicates) not just
connectors & tasks
2. Specifying a range of versions instead of an exact match
3. New metrics to observe what versions are in-use
Thanks to Snehashis for the original KIP idea!
Thanks,
Greg
On Tue, Jan 2, 2024 at 11:49 AM Greg Harris <gr...@aiven.io> wrote:
>
> Hi Snehashis,
>
> Thank you for the KIP! This is something I've wanted for a long time.
>
> I know the discussion has gone cold, are you still interested in
> pursuing this feature? I'll make time to review the KIP if you are
> still accepting comments.
>
> Thanks,
> Greg
>
> On Tue, Nov 22, 2022 at 12:29 PM Snehashis <sn...@gmail.com> wrote:
> >
> > Thanks for the points Sagar.
> >
> > > 1) Should we update the GET /connectors endpoint to include the version of
> > > the plugin that is running? It could be useful to figure out the version
> > of
> > > the plugin or I am assuming it gets returned by the expand=info call?
> >
> > I think this is good to have and possible future enhancement. The version
> > info will be present in the config of the connector if the user has
> > specified the version. Otherwise it is the latest version which the user
> > can find out from the connector-plugin endpoint. The information can be
> > introduced to the response of the GET /connectors endpoint itself, however
> > the most ideal way of doing this would be to get the currently running
> > instance of the connector and get the version directly from there. This is
> > slightly tricky as the connector could be running in a different node.
> > One way to do this would be to persist the version information in the
> > status backing store during instantiation of the connector. It requires
> > some more thought and since the version is part of the configs if provided
> > and evident otherwise, I have not included it in this KIP.
> >
> > > 2) I am not aware of this and hence asking, can 2 connectors with
> > different
> > > versions have the same name? Does the plugin isolation allow this? This
> > > could have a bearing when using the lifecycle endpoints for connectors
> > like
> > > DELETE etc.
> >
> > All connectors in a cluster need to have uniquire connector names
> > regardless of what version of the plugin the connector is running
> > underneath. This is something enforced by the connect runtime itself. All
> > connect CRUD operations are keyed on the connector name so there will not
> > be an issue.
> >
> > Regards
> > Snehashis
> >
> > On Tue, Nov 22, 2022 at 3:16 PM Sagar <sa...@gmail.com> wrote:
> >
> > > Hey Snehashsih,
> > >
> > > Thanks for the KIP. It looks like a very useful feature. Couple of
> > > small-ish points, let me know what you think:
> > >
> > > 1) Should we update the GET /connectors endpoint to include the version of
> > > the plugin that is running? It could be useful to figure out the version of
> > > the plugin or I am assuming it gets returned by the expand=info call?
> > > 2) I am not aware of this and hence asking, can 2 connectors with different
> > > versions have the same name? Does the plugin isolation allow this? This
> > > could have a bearing when using the lifecycle endpoints for connectors like
> > > DELETE etc.
> > >
> > > Thanks!
> > > Sagar.
> > >
> > >
> > > On Tue, Nov 22, 2022 at 2:10 PM Ashwin <ap...@confluent.io.invalid>
> > > wrote:
> > >
> > > > Hi Snehasis,
> > > >
> > > > > IIUC (please correct me if I am wrong here), what you highlighted
> > > above,
> > > > is
> > > > a versioning scheme for a connector config for the same connector (and
> > > not
> > > > different versions of a connector plugin).
> > > >
> > > > Sorry for not being more precise in my wording - I meant registering
> > > > versions of schema for connector config.
> > > >
> > > > Let's take the example of a fictional connector which uses a fictional
> > > AWS
> > > > service.
> > > >
> > > > Fictional Connector Config schema version:2.0
> > > > ---
> > > > {
> > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > "type": "object",
> > > > "properties": {
> > > > "name": {
> > > > "type": "string"
> > > > },
> > > > "schema_version": {
> > > > "type": "string"
> > > > },
> > > > "aws_access_key": {
> > > > "type": "string"
> > > > },
> > > > "aws_secret_key": {
> > > > "type": "string"
> > > > }
> > > > },
> > > > "required": [
> > > > "name",
> > > > "schema_version",
> > > > "aws_access_key",
> > > > "aws_secret_key"
> > > > ]
> > > > }
> > > >
> > > > Fictional Connector config schema version:3.0
> > > > ---
> > > > {
> > > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > > "type": "object",
> > > > "properties": {
> > > > "name": {
> > > > "type": "string"
> > > > },
> > > > "schema_version": {
> > > > "type": "string"
> > > > },
> > > > "iam_role": {
> > > > "type": "string"
> > > > }
> > > > },
> > > > "required": [
> > > > "name",
> > > > "schema_version",
> > > > "iam_role"
> > > > ]
> > > > }
> > > >
> > > > The connector which supports Fictional config schema 2.0 will validate
> > > the
> > > > access key and secret key.
> > > > Whereas a connector which supports config with schema version 3.0 will
> > > only
> > > > validate the IAM role.
> > > >
> > > > This is the alternative which I wanted to suggest. Each plugin will
> > > > register the schema versions of connector config which it supports.
> > > >
> > > > The plugin paths may be optionally different i.e we don't have to
> > > > mandatorily add a new plugin path to support a new schema version.
> > > >
> > > > Thanks,
> > > > Ashwin
> > > >
> > > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <sn...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks for the input Ashwin.
> > > > >
> > > > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > > > > config is versioned and has a schema. Then a single plugin (whose
> > > > > > dependencies have not changed) can handle multiple config versions
> > > for
> > > > > the
> > > > > > same connector class.
> > > > >
> > > > > IIUC (please correct me if I am wrong here), what you highlighted
> > > above,
> > > > is
> > > > > a versioning scheme for a connector config for the same connector (and
> > > > not
> > > > > different versions of a connector plugin). That is a somewhat
> > > tangential
> > > > > problem. While it is definitely a useful feature to have, like a log to
> > > > > check what changes were made over time to the config which might make
> > > it
> > > > > easier to do rollbacks, it is not the focus here. Here by version we
> > > mean
> > > > > to say what underlying version of the plugin should the given
> > > > configuration
> > > > > of the connector use. Perhaps it is better to change the name of the
> > > > > parameter from connector.version to connector.plugin.version or
> > > > > plugin.version if it was confusing. wdyt?
> > > > >
> > > > > > 2. Any plans to support assisted migration e.g if a user invokes
> > > "POST
> > > > > > connector/config?migrate=latest", the latest version __attempts__ to
> > > > > > transform the existing config to the newer version. This would
> > > require
> > > > > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > > > > connector interface.
> > > > >
> > > > > This is an enhancement we can think of doing in future. Users can
> > > simply
> > > > do
> > > > > a PUT call with the updated config which has the updated version
> > > number.
> > > > > The assisted mode could be handy as the user does not need to know the
> > > > > config but beyond this it does not seem to justify its existence.
> > > > >
> > > > > Regards
> > > > > Snehashis
> > > > >
> > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin <ap...@confluent.io.invalid>
> > > > > wrote:
> > > > >
> > > > > > Hi Snehasis,
> > > > > >
> > > > > > This is a really useful feature and thanks for initiating this
> > > > > discussion.
> > > > > >
> > > > > > I had the following questions -
> > > > > >
> > > > > >
> > > > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > > > > config is versioned and has a schema. Then a single plugin (whose
> > > > > > dependencies have not changed) can handle multiple config versions
> > > for
> > > > > the
> > > > > > same connector class.
> > > > > >
> > > > > > 2. Any plans to support assisted migration e.g if a user invokes
> > > "POST
> > > > > > connector/config?migrate=latest", the latest version __attempts__ to
> > > > > > transform the existing config to the newer version. This would
> > > require
> > > > > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > > > > connector interface.
> > > > > >
> > > > > > Thanks,
> > > > > > Ashwin
> > > > > >
> > > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <sn...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'd like to start a discussion thread on KIP-891: Running multiple
> > > > > > versions
> > > > > > > of a connector.
> > > > > > >
> > > > > > > The KIP aims to add the ability for the connect runtime to run
> > > > multiple
> > > > > > > versions of a connector.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > >
> > > > > > > Please take a look and let me know what you think.
> > > > > > >
> > > > > > > Thank you
> > > > > > > Snehashis Pal
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >