You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Snehashis <sn...@gmail.com> on 2022/11/21 08:57:06 UTC
[DISCUSS] KIP-891: Running multiple versions of a connector.
Hi all,
I'd like to start a discussion thread on KIP-891: Running multiple versions
of a connector.
The KIP aims to add the ability for the connect runtime to run multiple
versions of a connector.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
Please take a look and let me know what you think.
Thank you
Snehashis Pal
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Posted by Snehashis <sn...@gmail.com>.
Hi Mickael. Thanks for your input. Addressing the point you mentioned
below.
> 1) Can you explain how this would work with the GET
> /{pluginName}/config endpoint? How do you specify a version for a
> connector?
This API returns the set of configurations for a given connector. Since
between versions
the configurations can change its allow a user given version to return the
correct
configs. The version is added as a query parameter, for example -
/S3SinkConnector/config?version=v1.1.1.
> 2) Some connectors come bundled with transformations (for example
> Debezium). How would multiple versions of a transformation be handled?
The version of transformations bundled with a particular connector version
will be used
when the connector is run with the corresponding version number. There will
be implicit
isolation between the two transformation as they are part of two separate
plugins and will
be loaded using different plugin classloaders during connector creation.
> 3) You mention the latest version will be picked by default if not
> specified. The version() method returns a string and currently
> enforces no semantics on the value it returns. Can you clarify the new
> expected semantics and explain how versions will be compared
> (alphabetical, semantics versioning, something else?)
The plugin loading mechanism already compares connector versions (new
connectors are created with only the latest version though).
The comparison between version will remain the same as it is currently
and is done using maven artefact versioning plugin. It is a generic
versioning scheme
that supports semantic, alphabetical and combinations with support
additional modifiers
(like alpha, beta, release and snapshot build versions). Please refer to
this javadoc for the
full comparison method.
https://maven.apache.org/ref/3.5.2/maven-artifact/apidocs/org/apache/maven/artifact/versioning/ComparableVersion.html
.
I do not think its necessary to enforce a new semantic with the version.
IMO the existing
versioning scheme is appropriate and flexible enough for all code
versioning methods.
On Mon, Nov 21, 2022 at 8:37 PM Mickael Maison <mi...@gmail.com>
wrote:
> Hi,
>
> Thanks for the KIP, this is something that could be really useful!
>
> 1) Can you explain how this would work with the GET
> /{pluginName}/config endpoint? How do you specify a version for a
> connector?
>
> 2) Some connectors come bundled with transformations (for example
> Debezium). How would multiple versions of a transformation be handled?
>
> 3) You mention the latest version will be picked by default if not
> specified. The version() method returns a string and currently
> enforces no semantics on the value it returns. Can you clarify the new
> expected semantics and explain how versions will be compared
> (alphabetical, semantics versioning, something else?)
>
> Thanks,
> Mickael
>
>
> On Mon, Nov 21, 2022 at 9:57 AM Snehashis <sn...@gmail.com>
> wrote:
> >
> > Hi all,
> >
> > I'd like to start a discussion thread on KIP-891: Running multiple
> versions
> > of a connector.
> >
> > The KIP aims to add the ability for the connect runtime to run multiple
> > versions of a connector.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> >
> > Please take a look and let me know what you think.
> >
> > Thank you
> > Snehashis Pal
>
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Posted by Mickael Maison <mi...@gmail.com>.
Hi,
Thanks for the KIP, this is something that could be really useful!
1) Can you explain how this would work with the GET
/{pluginName}/config endpoint? How do you specify a version for a
connector?
2) Some connectors come bundled with transformations (for example
Debezium). How would multiple versions of a transformation be handled?
3) You mention the latest version will be picked by default if not
specified. The version() method returns a string and currently
enforces no semantics on the value it returns. Can you clarify the new
expected semantics and explain how versions will be compared
(alphabetical, semantics versioning, something else?)
Thanks,
Mickael
On Mon, Nov 21, 2022 at 9:57 AM Snehashis <sn...@gmail.com> wrote:
>
> Hi all,
>
> I'd like to start a discussion thread on KIP-891: Running multiple versions
> of a connector.
>
> The KIP aims to add the ability for the connect runtime to run multiple
> versions of a connector.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
>
> Please take a look and let me know what you think.
>
> Thank you
> Snehashis Pal
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Posted by Snehashis <sn...@gmail.com>.
Thanks for the explanation Ashwin.
This is an interesting notion. This is something which many connectors
implicitly do anyway. There are several connectors which have different
methods of interpreting the configurations provided. Often the user has
some control over how provided configuration should be used, through
omission of configs, boolean flags that activate/deactivate certain
configs, etc. One could argue that this increases the verbosity of the
configurations and makes it monolithic, however the alternative proposal of
having multiple registered schemas only really seems worthwhile if that the
runtime has the ability to alter the functionality of a connector. There
needs to be some way of registering multiple functionalities, one for each
configuration type. Otherwise, if the runtime is simply passing on the
configuration to the connector, regardless of the which schema version it
belongs to, and delegating the responsibility of picking the functionality
to the connector itself, there is very little the runtime is adding by
registering schemas. Multiple connector versions implicitly define
different configs and functionalities and hence, the ability to run
different versions of the connector itself seems like a more elegant
solution to address this problem.
I also don't think multiple configurations are the only use case for
running different versions of a connector. There could be internal changes
to a connector that do not involve any config changes. A change that
targets a particular enhancement may be incompatible with the older
behaviour. Right now in order to make the changes backwards compatible we
would have to gate the changes behind a connector config (or a different
schema and functionality registration). Otherwise the user is forced to
keep using the older connector until they can upgrade. Problem is if there
are multiple such enhancements (and only one is a breaking change) then
they are missing out on all the other enhements. It is simpler for the user
to have the ability to run both versions of the connector.
On Tue, Nov 22, 2022 at 2:11 PM Ashwin <ap...@confluent.io.invalid> wrote:
> Hi Snehasis,
>
> > IIUC (please correct me if I am wrong here), what you highlighted above,
> is
> a versioning scheme for a connector config for the same connector (and not
> different versions of a connector plugin).
>
> Sorry for not being more precise in my wording - I meant registering
> versions of schema for connector config.
>
> Let's take the example of a fictional connector which uses a fictional AWS
> service.
>
> Fictional Connector Config schema version:2.0
> ---
> {
> "$schema": "http://json-schema.org/draft-04/schema#",
> "type": "object",
> "properties": {
> "name": {
> "type": "string"
> },
> "schema_version": {
> "type": "string"
> },
> "aws_access_key": {
> "type": "string"
> },
> "aws_secret_key": {
> "type": "string"
> }
> },
> "required": [
> "name",
> "schema_version",
> "aws_access_key",
> "aws_secret_key"
> ]
> }
>
> Fictional Connector config schema version:3.0
> ---
> {
> "$schema": "http://json-schema.org/draft-04/schema#",
> "type": "object",
> "properties": {
> "name": {
> "type": "string"
> },
> "schema_version": {
> "type": "string"
> },
> "iam_role": {
> "type": "string"
> }
> },
> "required": [
> "name",
> "schema_version",
> "iam_role"
> ]
> }
>
> The connector which supports Fictional config schema 2.0 will validate the
> access key and secret key.
> Whereas a connector which supports config with schema version 3.0 will only
> validate the IAM role.
>
> This is the alternative which I wanted to suggest. Each plugin will
> register the schema versions of connector config which it supports.
>
> The plugin paths may be optionally different i.e we don't have to
> mandatorily add a new plugin path to support a new schema version.
>
> Thanks,
> Ashwin
>
> On Tue, Nov 22, 2022 at 12:47 PM Snehashis <sn...@gmail.com>
> wrote:
>
> > Thanks for the input Ashwin.
> >
> > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > config is versioned and has a schema. Then a single plugin (whose
> > > dependencies have not changed) can handle multiple config versions for
> > the
> > > same connector class.
> >
> > IIUC (please correct me if I am wrong here), what you highlighted above,
> is
> > a versioning scheme for a connector config for the same connector (and
> not
> > different versions of a connector plugin). That is a somewhat tangential
> > problem. While it is definitely a useful feature to have, like a log to
> > check what changes were made over time to the config which might make it
> > easier to do rollbacks, it is not the focus here. Here by version we mean
> > to say what underlying version of the plugin should the given
> configuration
> > of the connector use. Perhaps it is better to change the name of the
> > parameter from connector.version to connector.plugin.version or
> > plugin.version if it was confusing. wdyt?
> >
> > > 2. Any plans to support assisted migration e.g if a user invokes "POST
> > > connector/config?migrate=latest", the latest version __attempts__ to
> > > transform the existing config to the newer version. This would require
> > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > connector interface.
> >
> > This is an enhancement we can think of doing in future. Users can simply
> do
> > a PUT call with the updated config which has the updated version number.
> > The assisted mode could be handy as the user does not need to know the
> > config but beyond this it does not seem to justify its existence.
> >
> > Regards
> > Snehashis
> >
> > On Tue, Nov 22, 2022 at 10:50 AM Ashwin <ap...@confluent.io.invalid>
> > wrote:
> >
> > > Hi Snehasis,
> > >
> > > This is a really useful feature and thanks for initiating this
> > discussion.
> > >
> > > I had the following questions -
> > >
> > >
> > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > config is versioned and has a schema. Then a single plugin (whose
> > > dependencies have not changed) can handle multiple config versions for
> > the
> > > same connector class.
> > >
> > > 2. Any plans to support assisted migration e.g if a user invokes "POST
> > > connector/config?migrate=latest", the latest version __attempts__ to
> > > transform the existing config to the newer version. This would require
> > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > connector interface.
> > >
> > > Thanks,
> > > Ashwin
> > >
> > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <sn...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start a discussion thread on KIP-891: Running multiple
> > > versions
> > > > of a connector.
> > > >
> > > > The KIP aims to add the ability for the connect runtime to run
> multiple
> > > > versions of a connector.
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > >
> > > > Please take a look and let me know what you think.
> > > >
> > > > Thank you
> > > > Snehashis Pal
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Posted by Greg Harris <gr...@aiven.io.INVALID>.
Hi Snehashis,
Thank you for the KIP! This is something I've wanted for a long time.
I know the discussion has gone cold, are you still interested in
pursuing this feature? I'll make time to review the KIP if you are
still accepting comments.
Thanks,
Greg
On Tue, Nov 22, 2022 at 12:29 PM Snehashis <sn...@gmail.com> wrote:
>
> Thanks for the points Sagar.
>
> > 1) Should we update the GET /connectors endpoint to include the version of
> > the plugin that is running? It could be useful to figure out the version
> of
> > the plugin or I am assuming it gets returned by the expand=info call?
>
> I think this is good to have and possible future enhancement. The version
> info will be present in the config of the connector if the user has
> specified the version. Otherwise it is the latest version which the user
> can find out from the connector-plugin endpoint. The information can be
> introduced to the response of the GET /connectors endpoint itself, however
> the most ideal way of doing this would be to get the currently running
> instance of the connector and get the version directly from there. This is
> slightly tricky as the connector could be running in a different node.
> One way to do this would be to persist the version information in the
> status backing store during instantiation of the connector. It requires
> some more thought and since the version is part of the configs if provided
> and evident otherwise, I have not included it in this KIP.
>
> > 2) I am not aware of this and hence asking, can 2 connectors with
> different
> > versions have the same name? Does the plugin isolation allow this? This
> > could have a bearing when using the lifecycle endpoints for connectors
> like
> > DELETE etc.
>
> All connectors in a cluster need to have uniquire connector names
> regardless of what version of the plugin the connector is running
> underneath. This is something enforced by the connect runtime itself. All
> connect CRUD operations are keyed on the connector name so there will not
> be an issue.
>
> Regards
> Snehashis
>
> On Tue, Nov 22, 2022 at 3:16 PM Sagar <sa...@gmail.com> wrote:
>
> > Hey Snehashsih,
> >
> > Thanks for the KIP. It looks like a very useful feature. Couple of
> > small-ish points, let me know what you think:
> >
> > 1) Should we update the GET /connectors endpoint to include the version of
> > the plugin that is running? It could be useful to figure out the version of
> > the plugin or I am assuming it gets returned by the expand=info call?
> > 2) I am not aware of this and hence asking, can 2 connectors with different
> > versions have the same name? Does the plugin isolation allow this? This
> > could have a bearing when using the lifecycle endpoints for connectors like
> > DELETE etc.
> >
> > Thanks!
> > Sagar.
> >
> >
> > On Tue, Nov 22, 2022 at 2:10 PM Ashwin <ap...@confluent.io.invalid>
> > wrote:
> >
> > > Hi Snehasis,
> > >
> > > > IIUC (please correct me if I am wrong here), what you highlighted
> > above,
> > > is
> > > a versioning scheme for a connector config for the same connector (and
> > not
> > > different versions of a connector plugin).
> > >
> > > Sorry for not being more precise in my wording - I meant registering
> > > versions of schema for connector config.
> > >
> > > Let's take the example of a fictional connector which uses a fictional
> > AWS
> > > service.
> > >
> > > Fictional Connector Config schema version:2.0
> > > ---
> > > {
> > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > "type": "object",
> > > "properties": {
> > > "name": {
> > > "type": "string"
> > > },
> > > "schema_version": {
> > > "type": "string"
> > > },
> > > "aws_access_key": {
> > > "type": "string"
> > > },
> > > "aws_secret_key": {
> > > "type": "string"
> > > }
> > > },
> > > "required": [
> > > "name",
> > > "schema_version",
> > > "aws_access_key",
> > > "aws_secret_key"
> > > ]
> > > }
> > >
> > > Fictional Connector config schema version:3.0
> > > ---
> > > {
> > > "$schema": "http://json-schema.org/draft-04/schema#",
> > > "type": "object",
> > > "properties": {
> > > "name": {
> > > "type": "string"
> > > },
> > > "schema_version": {
> > > "type": "string"
> > > },
> > > "iam_role": {
> > > "type": "string"
> > > }
> > > },
> > > "required": [
> > > "name",
> > > "schema_version",
> > > "iam_role"
> > > ]
> > > }
> > >
> > > The connector which supports Fictional config schema 2.0 will validate
> > the
> > > access key and secret key.
> > > Whereas a connector which supports config with schema version 3.0 will
> > only
> > > validate the IAM role.
> > >
> > > This is the alternative which I wanted to suggest. Each plugin will
> > > register the schema versions of connector config which it supports.
> > >
> > > The plugin paths may be optionally different i.e we don't have to
> > > mandatorily add a new plugin path to support a new schema version.
> > >
> > > Thanks,
> > > Ashwin
> > >
> > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <sn...@gmail.com>
> > > wrote:
> > >
> > > > Thanks for the input Ashwin.
> > > >
> > > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > > > config is versioned and has a schema. Then a single plugin (whose
> > > > > dependencies have not changed) can handle multiple config versions
> > for
> > > > the
> > > > > same connector class.
> > > >
> > > > IIUC (please correct me if I am wrong here), what you highlighted
> > above,
> > > is
> > > > a versioning scheme for a connector config for the same connector (and
> > > not
> > > > different versions of a connector plugin). That is a somewhat
> > tangential
> > > > problem. While it is definitely a useful feature to have, like a log to
> > > > check what changes were made over time to the config which might make
> > it
> > > > easier to do rollbacks, it is not the focus here. Here by version we
> > mean
> > > > to say what underlying version of the plugin should the given
> > > configuration
> > > > of the connector use. Perhaps it is better to change the name of the
> > > > parameter from connector.version to connector.plugin.version or
> > > > plugin.version if it was confusing. wdyt?
> > > >
> > > > > 2. Any plans to support assisted migration e.g if a user invokes
> > "POST
> > > > > connector/config?migrate=latest", the latest version __attempts__ to
> > > > > transform the existing config to the newer version. This would
> > require
> > > > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > > > connector interface.
> > > >
> > > > This is an enhancement we can think of doing in future. Users can
> > simply
> > > do
> > > > a PUT call with the updated config which has the updated version
> > number.
> > > > The assisted mode could be handy as the user does not need to know the
> > > > config but beyond this it does not seem to justify its existence.
> > > >
> > > > Regards
> > > > Snehashis
> > > >
> > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin <ap...@confluent.io.invalid>
> > > > wrote:
> > > >
> > > > > Hi Snehasis,
> > > > >
> > > > > This is a really useful feature and thanks for initiating this
> > > > discussion.
> > > > >
> > > > > I had the following questions -
> > > > >
> > > > >
> > > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > > > config is versioned and has a schema. Then a single plugin (whose
> > > > > dependencies have not changed) can handle multiple config versions
> > for
> > > > the
> > > > > same connector class.
> > > > >
> > > > > 2. Any plans to support assisted migration e.g if a user invokes
> > "POST
> > > > > connector/config?migrate=latest", the latest version __attempts__ to
> > > > > transform the existing config to the newer version. This would
> > require
> > > > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > > > connector interface.
> > > > >
> > > > > Thanks,
> > > > > Ashwin
> > > > >
> > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <sn...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to start a discussion thread on KIP-891: Running multiple
> > > > > versions
> > > > > > of a connector.
> > > > > >
> > > > > > The KIP aims to add the ability for the connect runtime to run
> > > multiple
> > > > > > versions of a connector.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > >
> > > > > > Please take a look and let me know what you think.
> > > > > >
> > > > > > Thank you
> > > > > > Snehashis Pal
> > > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Posted by Snehashis <sn...@gmail.com>.
Thanks for the points Sagar.
> 1) Should we update the GET /connectors endpoint to include the version of
> the plugin that is running? It could be useful to figure out the version
of
> the plugin or I am assuming it gets returned by the expand=info call?
I think this is good to have and possible future enhancement. The version
info will be present in the config of the connector if the user has
specified the version. Otherwise it is the latest version which the user
can find out from the connector-plugin endpoint. The information can be
introduced to the response of the GET /connectors endpoint itself, however
the most ideal way of doing this would be to get the currently running
instance of the connector and get the version directly from there. This is
slightly tricky as the connector could be running in a different node.
One way to do this would be to persist the version information in the
status backing store during instantiation of the connector. It requires
some more thought and since the version is part of the configs if provided
and evident otherwise, I have not included it in this KIP.
> 2) I am not aware of this and hence asking, can 2 connectors with
different
> versions have the same name? Does the plugin isolation allow this? This
> could have a bearing when using the lifecycle endpoints for connectors
like
> DELETE etc.
All connectors in a cluster need to have uniquire connector names
regardless of what version of the plugin the connector is running
underneath. This is something enforced by the connect runtime itself. All
connect CRUD operations are keyed on the connector name so there will not
be an issue.
Regards
Snehashis
On Tue, Nov 22, 2022 at 3:16 PM Sagar <sa...@gmail.com> wrote:
> Hey Snehashsih,
>
> Thanks for the KIP. It looks like a very useful feature. Couple of
> small-ish points, let me know what you think:
>
> 1) Should we update the GET /connectors endpoint to include the version of
> the plugin that is running? It could be useful to figure out the version of
> the plugin or I am assuming it gets returned by the expand=info call?
> 2) I am not aware of this and hence asking, can 2 connectors with different
> versions have the same name? Does the plugin isolation allow this? This
> could have a bearing when using the lifecycle endpoints for connectors like
> DELETE etc.
>
> Thanks!
> Sagar.
>
>
> On Tue, Nov 22, 2022 at 2:10 PM Ashwin <ap...@confluent.io.invalid>
> wrote:
>
> > Hi Snehasis,
> >
> > > IIUC (please correct me if I am wrong here), what you highlighted
> above,
> > is
> > a versioning scheme for a connector config for the same connector (and
> not
> > different versions of a connector plugin).
> >
> > Sorry for not being more precise in my wording - I meant registering
> > versions of schema for connector config.
> >
> > Let's take the example of a fictional connector which uses a fictional
> AWS
> > service.
> >
> > Fictional Connector Config schema version:2.0
> > ---
> > {
> > "$schema": "http://json-schema.org/draft-04/schema#",
> > "type": "object",
> > "properties": {
> > "name": {
> > "type": "string"
> > },
> > "schema_version": {
> > "type": "string"
> > },
> > "aws_access_key": {
> > "type": "string"
> > },
> > "aws_secret_key": {
> > "type": "string"
> > }
> > },
> > "required": [
> > "name",
> > "schema_version",
> > "aws_access_key",
> > "aws_secret_key"
> > ]
> > }
> >
> > Fictional Connector config schema version:3.0
> > ---
> > {
> > "$schema": "http://json-schema.org/draft-04/schema#",
> > "type": "object",
> > "properties": {
> > "name": {
> > "type": "string"
> > },
> > "schema_version": {
> > "type": "string"
> > },
> > "iam_role": {
> > "type": "string"
> > }
> > },
> > "required": [
> > "name",
> > "schema_version",
> > "iam_role"
> > ]
> > }
> >
> > The connector which supports Fictional config schema 2.0 will validate
> the
> > access key and secret key.
> > Whereas a connector which supports config with schema version 3.0 will
> only
> > validate the IAM role.
> >
> > This is the alternative which I wanted to suggest. Each plugin will
> > register the schema versions of connector config which it supports.
> >
> > The plugin paths may be optionally different i.e we don't have to
> > mandatorily add a new plugin path to support a new schema version.
> >
> > Thanks,
> > Ashwin
> >
> > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <sn...@gmail.com>
> > wrote:
> >
> > > Thanks for the input Ashwin.
> > >
> > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > > config is versioned and has a schema. Then a single plugin (whose
> > > > dependencies have not changed) can handle multiple config versions
> for
> > > the
> > > > same connector class.
> > >
> > > IIUC (please correct me if I am wrong here), what you highlighted
> above,
> > is
> > > a versioning scheme for a connector config for the same connector (and
> > not
> > > different versions of a connector plugin). That is a somewhat
> tangential
> > > problem. While it is definitely a useful feature to have, like a log to
> > > check what changes were made over time to the config which might make
> it
> > > easier to do rollbacks, it is not the focus here. Here by version we
> mean
> > > to say what underlying version of the plugin should the given
> > configuration
> > > of the connector use. Perhaps it is better to change the name of the
> > > parameter from connector.version to connector.plugin.version or
> > > plugin.version if it was confusing. wdyt?
> > >
> > > > 2. Any plans to support assisted migration e.g if a user invokes
> "POST
> > > > connector/config?migrate=latest", the latest version __attempts__ to
> > > > transform the existing config to the newer version. This would
> require
> > > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > > connector interface.
> > >
> > > This is an enhancement we can think of doing in future. Users can
> simply
> > do
> > > a PUT call with the updated config which has the updated version
> number.
> > > The assisted mode could be handy as the user does not need to know the
> > > config but beyond this it does not seem to justify its existence.
> > >
> > > Regards
> > > Snehashis
> > >
> > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin <ap...@confluent.io.invalid>
> > > wrote:
> > >
> > > > Hi Snehasis,
> > > >
> > > > This is a really useful feature and thanks for initiating this
> > > discussion.
> > > >
> > > > I had the following questions -
> > > >
> > > >
> > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > > config is versioned and has a schema. Then a single plugin (whose
> > > > dependencies have not changed) can handle multiple config versions
> for
> > > the
> > > > same connector class.
> > > >
> > > > 2. Any plans to support assisted migration e.g if a user invokes
> "POST
> > > > connector/config?migrate=latest", the latest version __attempts__ to
> > > > transform the existing config to the newer version. This would
> require
> > > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > > connector interface.
> > > >
> > > > Thanks,
> > > > Ashwin
> > > >
> > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <sn...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to start a discussion thread on KIP-891: Running multiple
> > > > versions
> > > > > of a connector.
> > > > >
> > > > > The KIP aims to add the ability for the connect runtime to run
> > multiple
> > > > > versions of a connector.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > >
> > > > > Please take a look and let me know what you think.
> > > > >
> > > > > Thank you
> > > > > Snehashis Pal
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Posted by Sagar <sa...@gmail.com>.
Hey Snehashsih,
Thanks for the KIP. It looks like a very useful feature. Couple of
small-ish points, let me know what you think:
1) Should we update the GET /connectors endpoint to include the version of
the plugin that is running? It could be useful to figure out the version of
the plugin or I am assuming it gets returned by the expand=info call?
2) I am not aware of this and hence asking, can 2 connectors with different
versions have the same name? Does the plugin isolation allow this? This
could have a bearing when using the lifecycle endpoints for connectors like
DELETE etc.
Thanks!
Sagar.
On Tue, Nov 22, 2022 at 2:10 PM Ashwin <ap...@confluent.io.invalid> wrote:
> Hi Snehasis,
>
> > IIUC (please correct me if I am wrong here), what you highlighted above,
> is
> a versioning scheme for a connector config for the same connector (and not
> different versions of a connector plugin).
>
> Sorry for not being more precise in my wording - I meant registering
> versions of schema for connector config.
>
> Let's take the example of a fictional connector which uses a fictional AWS
> service.
>
> Fictional Connector Config schema version:2.0
> ---
> {
> "$schema": "http://json-schema.org/draft-04/schema#",
> "type": "object",
> "properties": {
> "name": {
> "type": "string"
> },
> "schema_version": {
> "type": "string"
> },
> "aws_access_key": {
> "type": "string"
> },
> "aws_secret_key": {
> "type": "string"
> }
> },
> "required": [
> "name",
> "schema_version",
> "aws_access_key",
> "aws_secret_key"
> ]
> }
>
> Fictional Connector config schema version:3.0
> ---
> {
> "$schema": "http://json-schema.org/draft-04/schema#",
> "type": "object",
> "properties": {
> "name": {
> "type": "string"
> },
> "schema_version": {
> "type": "string"
> },
> "iam_role": {
> "type": "string"
> }
> },
> "required": [
> "name",
> "schema_version",
> "iam_role"
> ]
> }
>
> The connector which supports Fictional config schema 2.0 will validate the
> access key and secret key.
> Whereas a connector which supports config with schema version 3.0 will only
> validate the IAM role.
>
> This is the alternative which I wanted to suggest. Each plugin will
> register the schema versions of connector config which it supports.
>
> The plugin paths may be optionally different i.e we don't have to
> mandatorily add a new plugin path to support a new schema version.
>
> Thanks,
> Ashwin
>
> On Tue, Nov 22, 2022 at 12:47 PM Snehashis <sn...@gmail.com>
> wrote:
>
> > Thanks for the input Ashwin.
> >
> > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > config is versioned and has a schema. Then a single plugin (whose
> > > dependencies have not changed) can handle multiple config versions for
> > the
> > > same connector class.
> >
> > IIUC (please correct me if I am wrong here), what you highlighted above,
> is
> > a versioning scheme for a connector config for the same connector (and
> not
> > different versions of a connector plugin). That is a somewhat tangential
> > problem. While it is definitely a useful feature to have, like a log to
> > check what changes were made over time to the config which might make it
> > easier to do rollbacks, it is not the focus here. Here by version we mean
> > to say what underlying version of the plugin should the given
> configuration
> > of the connector use. Perhaps it is better to change the name of the
> > parameter from connector.version to connector.plugin.version or
> > plugin.version if it was confusing. wdyt?
> >
> > > 2. Any plans to support assisted migration e.g if a user invokes "POST
> > > connector/config?migrate=latest", the latest version __attempts__ to
> > > transform the existing config to the newer version. This would require
> > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > connector interface.
> >
> > This is an enhancement we can think of doing in future. Users can simply
> do
> > a PUT call with the updated config which has the updated version number.
> > The assisted mode could be handy as the user does not need to know the
> > config but beyond this it does not seem to justify its existence.
> >
> > Regards
> > Snehashis
> >
> > On Tue, Nov 22, 2022 at 10:50 AM Ashwin <ap...@confluent.io.invalid>
> > wrote:
> >
> > > Hi Snehasis,
> > >
> > > This is a really useful feature and thanks for initiating this
> > discussion.
> > >
> > > I had the following questions -
> > >
> > >
> > > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > > config is versioned and has a schema. Then a single plugin (whose
> > > dependencies have not changed) can handle multiple config versions for
> > the
> > > same connector class.
> > >
> > > 2. Any plans to support assisted migration e.g if a user invokes "POST
> > > connector/config?migrate=latest", the latest version __attempts__ to
> > > transform the existing config to the newer version. This would require
> > > adding a method like "boolean migrate(Version fromVersion)" to the
> > > connector interface.
> > >
> > > Thanks,
> > > Ashwin
> > >
> > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <sn...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start a discussion thread on KIP-891: Running multiple
> > > versions
> > > > of a connector.
> > > >
> > > > The KIP aims to add the ability for the connect runtime to run
> multiple
> > > > versions of a connector.
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > >
> > > > Please take a look and let me know what you think.
> > > >
> > > > Thank you
> > > > Snehashis Pal
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Posted by Ashwin <ap...@confluent.io.INVALID>.
Hi Snehasis,
> IIUC (please correct me if I am wrong here), what you highlighted above,
is
a versioning scheme for a connector config for the same connector (and not
different versions of a connector plugin).
Sorry for not being more precise in my wording - I meant registering
versions of schema for connector config.
Let's take the example of a fictional connector which uses a fictional AWS
service.
Fictional Connector Config schema version:2.0
---
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"properties": {
"name": {
"type": "string"
},
"schema_version": {
"type": "string"
},
"aws_access_key": {
"type": "string"
},
"aws_secret_key": {
"type": "string"
}
},
"required": [
"name",
"schema_version",
"aws_access_key",
"aws_secret_key"
]
}
Fictional Connector config schema version:3.0
---
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"properties": {
"name": {
"type": "string"
},
"schema_version": {
"type": "string"
},
"iam_role": {
"type": "string"
}
},
"required": [
"name",
"schema_version",
"iam_role"
]
}
The connector which supports Fictional config schema 2.0 will validate the
access key and secret key.
Whereas a connector which supports config with schema version 3.0 will only
validate the IAM role.
This is the alternative which I wanted to suggest. Each plugin will
register the schema versions of connector config which it supports.
The plugin paths may be optionally different i.e we don't have to
mandatorily add a new plugin path to support a new schema version.
Thanks,
Ashwin
On Tue, Nov 22, 2022 at 12:47 PM Snehashis <sn...@gmail.com> wrote:
> Thanks for the input Ashwin.
>
> > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > config is versioned and has a schema. Then a single plugin (whose
> > dependencies have not changed) can handle multiple config versions for
> the
> > same connector class.
>
> IIUC (please correct me if I am wrong here), what you highlighted above, is
> a versioning scheme for a connector config for the same connector (and not
> different versions of a connector plugin). That is a somewhat tangential
> problem. While it is definitely a useful feature to have, like a log to
> check what changes were made over time to the config which might make it
> easier to do rollbacks, it is not the focus here. Here by version we mean
> to say what underlying version of the plugin should the given configuration
> of the connector use. Perhaps it is better to change the name of the
> parameter from connector.version to connector.plugin.version or
> plugin.version if it was confusing. wdyt?
>
> > 2. Any plans to support assisted migration e.g if a user invokes "POST
> > connector/config?migrate=latest", the latest version __attempts__ to
> > transform the existing config to the newer version. This would require
> > adding a method like "boolean migrate(Version fromVersion)" to the
> > connector interface.
>
> This is an enhancement we can think of doing in future. Users can simply do
> a PUT call with the updated config which has the updated version number.
> The assisted mode could be handy as the user does not need to know the
> config but beyond this it does not seem to justify its existence.
>
> Regards
> Snehashis
>
> On Tue, Nov 22, 2022 at 10:50 AM Ashwin <ap...@confluent.io.invalid>
> wrote:
>
> > Hi Snehasis,
> >
> > This is a really useful feature and thanks for initiating this
> discussion.
> >
> > I had the following questions -
> >
> >
> > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > config is versioned and has a schema. Then a single plugin (whose
> > dependencies have not changed) can handle multiple config versions for
> the
> > same connector class.
> >
> > 2. Any plans to support assisted migration e.g if a user invokes "POST
> > connector/config?migrate=latest", the latest version __attempts__ to
> > transform the existing config to the newer version. This would require
> > adding a method like "boolean migrate(Version fromVersion)" to the
> > connector interface.
> >
> > Thanks,
> > Ashwin
> >
> > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <sn...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to start a discussion thread on KIP-891: Running multiple
> > versions
> > > of a connector.
> > >
> > > The KIP aims to add the ability for the connect runtime to run multiple
> > > versions of a connector.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > >
> > > Please take a look and let me know what you think.
> > >
> > > Thank you
> > > Snehashis Pal
> > >
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Posted by Snehashis <sn...@gmail.com>.
Thanks for the input Ashwin.
> 1. Can you elaborate on the rejected alternatives ? Suppose connector
> config is versioned and has a schema. Then a single plugin (whose
> dependencies have not changed) can handle multiple config versions for the
> same connector class.
IIUC (please correct me if I am wrong here), what you highlighted above, is
a versioning scheme for a connector config for the same connector (and not
different versions of a connector plugin). That is a somewhat tangential
problem. While it is definitely a useful feature to have, like a log to
check what changes were made over time to the config which might make it
easier to do rollbacks, it is not the focus here. Here by version we mean
to say what underlying version of the plugin should the given configuration
of the connector use. Perhaps it is better to change the name of the
parameter from connector.version to connector.plugin.version or
plugin.version if it was confusing. wdyt?
> 2. Any plans to support assisted migration e.g if a user invokes "POST
> connector/config?migrate=latest", the latest version __attempts__ to
> transform the existing config to the newer version. This would require
> adding a method like "boolean migrate(Version fromVersion)" to the
> connector interface.
This is an enhancement we can think of doing in future. Users can simply do
a PUT call with the updated config which has the updated version number.
The assisted mode could be handy as the user does not need to know the
config but beyond this it does not seem to justify its existence.
Regards
Snehashis
On Tue, Nov 22, 2022 at 10:50 AM Ashwin <ap...@confluent.io.invalid>
wrote:
> Hi Snehasis,
>
> This is a really useful feature and thanks for initiating this discussion.
>
> I had the following questions -
>
>
> 1. Can you elaborate on the rejected alternatives ? Suppose connector
> config is versioned and has a schema. Then a single plugin (whose
> dependencies have not changed) can handle multiple config versions for the
> same connector class.
>
> 2. Any plans to support assisted migration e.g if a user invokes "POST
> connector/config?migrate=latest", the latest version __attempts__ to
> transform the existing config to the newer version. This would require
> adding a method like "boolean migrate(Version fromVersion)" to the
> connector interface.
>
> Thanks,
> Ashwin
>
> On Mon, Nov 21, 2022 at 2:27 PM Snehashis <sn...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I'd like to start a discussion thread on KIP-891: Running multiple
> versions
> > of a connector.
> >
> > The KIP aims to add the ability for the connect runtime to run multiple
> > versions of a connector.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> >
> > Please take a look and let me know what you think.
> >
> > Thank you
> > Snehashis Pal
> >
>
Re: [DISCUSS] KIP-891: Running multiple versions of a connector.
Posted by Ashwin <ap...@confluent.io.INVALID>.
Hi Snehasis,
This is a really useful feature and thanks for initiating this discussion.
I had the following questions -
1. Can you elaborate on the rejected alternatives ? Suppose connector
config is versioned and has a schema. Then a single plugin (whose
dependencies have not changed) can handle multiple config versions for the
same connector class.
2. Any plans to support assisted migration e.g if a user invokes "POST
connector/config?migrate=latest", the latest version __attempts__ to
transform the existing config to the newer version. This would require
adding a method like "boolean migrate(Version fromVersion)" to the
connector interface.
Thanks,
Ashwin
On Mon, Nov 21, 2022 at 2:27 PM Snehashis <sn...@gmail.com> wrote:
> Hi all,
>
> I'd like to start a discussion thread on KIP-891: Running multiple versions
> of a connector.
>
> The KIP aims to add the ability for the connect runtime to run multiple
> versions of a connector.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
>
> Please take a look and let me know what you think.
>
> Thank you
> Snehashis Pal
>